Multi-agent reinforcement learning-based multi-traffic signal lamp control method and system

Through the method of reinforcement learning of multiple agents, the mechanism of agent and state communication is constructed, and the problem of dynamic traffic feature capture and influence relationships in multi-intersection signal light control is solved, the coordination control of multi-intersection signal lights is realized, and the traffic system is improved pass efficiency and overall performance.

CN118968790APending Publication Date: 2024-11-15HUAQIAO UNIVERSITY +1
View PDF 0 Cites 9 Cited by

Patent Information

Application Number
CN202411036565.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing multi-intersection signal light control algorithm cannot capture the dynamic traffic characteristics and impact relationships between adjacent intersections, resulting in poor coordination control effects.

Method used

Using a method based on reinforcement learning of multiple agents, the agents at each intersection are constructed, and the state communication mechanism between agents is established through state space, action space and fairness reward function, and the target reward function is constructed in combination with continuous congestion penalty items. , train the estimation network and the target network to realize signal light phase coordination control.

Benefits of technology

Effectively capture the dynamic traffic characteristics of adjacent intersections, optimize the coordinated control of signal lights at multiple intersections, improve traffic efficiency, avoid wrong decisions, and improve the overall performance of the transportation system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968790A_ABST
    Figure CN118968790A_ABST
Patent Text Reader

Abstract

The invention provides a multi-agent reinforcement learning-based multi-traffic signal lamp control method and system, and relates to the technical field of signal lamp phase control. Road network structures of a plurality of intersections and traffic data of a preset time period are obtained, and agents of the intersections are constructed; constructing a state communication mechanism of multiple intersections by taking the traffic flow states of the exits of the upstream road sections of the adjacent intersections as communication contents; constructing a target reward function in combination with a continuous congestion penalty term; two networks with the same structure and different parameters are constructed for each intersection, and signal lamp coordination control of corresponding intelligent agents is executed for all intersections; and finally, network parameters are updated by adopting a loss function and a gradient descent method until the maximum simulation frequency is reached, and a signal lamp phase control strategy of multiple intersections is obtained. According to the invention, a state communication mechanism between intelligent agents is established, dynamic traffic flow characteristics between adjacent intersections are captured, and the traffic efficiency of a multi-intersection area can be effectively improved under real-time traffic data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intersection signal light phase control, and in particular to a multi-traffic signal light control method and system based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of cities and the growing demand for transportation, the contradiction between transportation supply and demand has become more prominent, and traffic congestion has become one of the current research hotspots. Traffic congestion not only affects the efficiency and quality of urban life, but also brings about a series of economic and social problems such as environmental pollution and energy shortages. Therefore, managing traffic congestion and optimizing the organization and control of modern urban road networks have positive significance for promoting economic and social development.

[0003] Traffic light control at multiple intersections is complex and requires attention to coordinated control between multiple intersections. In traffic light control at multiple intersections, completely isolated multi-agent control faces the problem of moving target learning and unique equilibrium countermeasures. The system cannot converge to an equilibrium joint strategy, so the coordinated control strategy between agents is very important. The collaborative mode of agents is mainly reflected in the communication mechanism of agents and the learning objective function of agents. At present, the traffic light control algorithm for multiple intersections cannot capture the dynamic traffic flow characteristics between adjacent intersections in terms of communication mechanism, and the influence relationship between adjacent traffic lights cannot be accurately calculated, which will eventually lead to poor coordinated control effect of each traffic light.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a multi-traffic signal light control method and system based on multi-agent reinforcement learning, so as to solve the problem that the existing multi-intersection signal light control algorithm cannot capture the dynamic traffic flow characteristics between adjacent intersections in terms of communication mechanism, and the influence relationship between adjacent signal lights cannot be accurately calculated, resulting in poor coordinated control effect of each signal light.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A multi-traffic signal light control method based on multi-agent reinforcement learning, comprising:

[0008] S1, obtaining a road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in a selected area; and the traffic data includes the volume of vehicles entering during the preset time period;

[0009] S2, constructing an intelligent agent corresponding to each intersection according to the road network structure and the traffic data, wherein the intelligent agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction;

[0010] S3, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection, two networks with the same structure and different parameters are constructed for each intersection, namely, an estimation network and a target network; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed;

[0011] S4, taking the traffic flow state of the upstream road section exit of the adjacent intersection as the communication content, constructing a state communication mechanism for multiple intersections, selecting a road section at a preset distance from the upstream exit of the intersection as a monitoring target, judging whether continuous congestion occurs by monitoring the vehicle queue situation of the upstream exit section, constructing a continuous congestion penalty item, and combining the fairness reward function with the continuous congestion penalty item, thereby obtaining a target reward function;

[0012] S5, calculate the estimated Q value corresponding to each action under the intersection state at the current moment according to the estimation network of each intersection, after the intelligent agent at the current intersection selects the action using the ε-greedy strategy, the selected phase executes the action, and calculates the reward value through the target reward function to obtain the traffic state at the next moment after the action is executed; store the data of the intersection state, action, reward value, and traffic state at the next moment in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the traffic lights of the corresponding intelligent agents at all intersections;

[0013] S6, taking the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby obtaining the signal light phase control model of each intersection;

[0014] S7, obtaining real-time traffic flow data of the external entrances of multiple intersections, and inputting the signal light phase control model, and outputting the signal light phase control scheme of each intersection under the real-time traffic flow data.

[0015] The present invention also provides a multi-traffic signal light control system based on multi-agent reinforcement learning, comprising:

[0016] A multi-intersection data acquisition unit, used to acquire the road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in the selected area; the traffic data includes the volume of vehicles entering during the preset time period;

[0017] A multi-agent construction unit is used to construct an agent, and construct an agent corresponding to each intersection according to the road network structure and the traffic data, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction;

[0018] An algorithm network construction unit is used to construct two networks with the same structure and different parameters for each intersection, namely, an estimation network and a target network, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed;

[0019] A target reward function construction unit is used to use the traffic flow state of the upstream road section exit of the adjacent intersection as the communication content, construct a state communication mechanism for multiple intersections, select a road section at a preset distance from the upstream exit of the intersection as a monitoring target, determine whether continuous congestion occurs by monitoring the vehicle queue situation of the upstream exit section, construct a continuous congestion penalty item, and combine the continuous congestion penalty item according to the fairness reward function to obtain a target reward function;

[0020] The multi-intersection coordination control unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network of each intersection. The intelligent agent at the current intersection adopts After the strategy selects an action, the selected phase executes the action, and the reward value is calculated through the target reward function to obtain the traffic state at the next moment after the action is executed; and the data of the intersection state, action, reward value, and traffic state at the next moment are stored in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the signal lights of the corresponding agents at all intersections;

[0021] A model training and updating unit is used to take the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, use the loss function MSE to calculate the error between the target value and the estimated Q value, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to obtain the signal light phase control model of each intersection;

[0022] The output unit is used to obtain real-time traffic flow data of the external entrances of multiple intersections, input the signal light phase control model, and output the signal light phase control scheme of each intersection under the real-time traffic flow data.

[0023] The present invention also provides a multi-traffic light control device based on multi-agent reinforcement learning, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a multi-traffic light control method based on multi-agent reinforcement learning as described above.

[0024] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a multi-traffic light control method based on multi-agent reinforcement learning is implemented as described above.

[0025] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention extracts the factors affecting traffic flow based on the relationship between multiple traffic lights, constructs corresponding intelligent body structures for multiple intersections, establishes a state communication mechanism between intelligent bodies, and captures the dynamic traffic flow characteristics between adjacent intersections. Then, the local traffic efficiency of each intersection and the overall traffic efficiency of the region are combined, and the target reward function of the coordinated control of multiple traffic lights is constructed based on the fairness reward mechanism of the coordinated control of traffic lights at each intersection, and a coordinated control model of traffic lights at multiple intersections is obtained. The trained model can be used for real-time traffic data under working conditions, which has positive significance for managing traffic congestion, optimizing the organization and control of modern urban road networks, and promoting economic and social development.

[0027] In addition, on the basis of the fairness reward function, the present invention additionally considers the number of vehicles passing through the area and the factors affecting continuous congestion, and constructs a target reward function to avoid being affected by the control of adjacent signal lights and making wrong decisions. For example, when the downstream signal light cannot release the traffic flow from the upstream in time, it will cause the upstream exit section to accumulate vehicles, thus causing continuous congestion at the upstream and downstream intersections. The present invention gives a penalty term to avoid the next wrong decision of the intelligent agent and realize the coordinated control of multi-intersection signal lights. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of a multi-traffic signal light control method based on multi-agent reinforcement learning provided in Example 1;

[0030] Figure 2 A flow chart of a multi-signal light coordination control algorithm provided in Example 1;

[0031] Figure 3 A schematic diagram of the adjacent intersection structure provided in Example 1;

[0032] Figure 4 A schematic diagram of a model for the multi-intersection signal light control problem provided in Example 1;

[0033] Figure 5 A schematic diagram of the structure of multiple intersections provided in Example 1;

[0034] Figure 6 A comparison chart of vehicle traffic conditions during peak hours provided in Example 1;

[0035] Figure 7 A comparison chart of average travel time at each intersection during peak hours provided in Example 1;

[0036] Figure 8 A comparison chart of average queue lengths at each intersection during peak hours provided in Example 1;

[0037] Fig. 9 A schematic diagram of a multi-traffic light control system based on multi-agent reinforcement learning provided in Example 2.

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] Embodiment 1

[0041] Embodiment 1 of the present invention provides a multi-traffic light control method based on multi-agent reinforcement learning, which can be implemented by a multi-traffic light control device based on multi-agent reinforcement learning (hereinafter referred to as the traffic light control device), and in particular, executed by one or more processors in the traffic light control device.

[0042] In this embodiment, the traffic light control device may be an electronic device equipped with a processor, which carries a computer program of the multi-traffic light control method based on multi-agent reinforcement learning and can be executed, such as a computer, a smart phone, a smart tablet, a workstation, a server and other computing devices with data calculation and processing capabilities, which are not specifically limited in the present invention.

[0043] like Figure 1-Figure 2 As shown, a multi-traffic light control method based on multi-agent reinforcement learning includes steps S1 to S7.

[0044] S1, obtaining the road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in the selected area; and the traffic data includes the vehicle flow entering during the preset time period.

[0045] In this embodiment, the historical traffic data set of the intersection can be obtained from various traffic platforms. The road network structure includes n1 intersections, n2 external entrances, and n3 entrance lanes in the selected area. The traffic data is the traffic flow q of vehicles entering during the period [t, t+Δt]. j (t,t+Δt), where j represents the entrance lane number.

[0046] S2, constructing an intelligent agent corresponding to each intersection according to the road network structure and the traffic data, the intelligent agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items in each direction.

[0047] In this embodiment, according to the n1 intersections in the selected area, n1 basic structures of intelligent agents corresponding to the intersections are constructed, and the basic structure of each intelligent agent includes a state space s, an action space a and a reward function r.

[0048] Each agent selects the vehicle position distribution and speed on the entrance lane to describe the traffic flow state space. In this embodiment, all non-conflicting phases of each intersection are selected as action spaces, and the right turn direction is defined as not controlled by traffic lights. Finally, 12 phase action spaces are defined: "straight from north to south + left turn", "straight from east to west + left turn", "straight from south to north + left turn", "straight from west to east + left turn", "straight from north to south + left turn from east to west", "straight from east to west + left turn from south to north", "straight from south to north + left turn from west to east", "straight from west to east + left turn from north to south", "straight from south to north + straight from north to south", "straight from east to west + straight from west to east", "left turn from south to north + left turn from north to south", "left turn from east to west + left turn from west to east";

[0049] The number of types of phase action spaces may also be determined according to actual conditions, such as 8 types, 4 types, etc., which is not limited here.

[0050] When constructing the fairness reward function, we consider the traffic efficiency and fairness separately; in terms of traffic efficiency, we consider the impact of each decision period and a traffic light cycle; in terms of decision period, we select the change of queue length and the utilization rate of green light time of each phase to reflect the short-term impact of the current decision on the traffic efficiency per unit time; in a traffic light cycle, we select the average travel time to evaluate the impact of the selected action set on the traffic efficiency within a cycle; in terms of fairness, we establish constraints and set penalty functions according to the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle; among them,

[0051] The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points; if the difference between the average queue lengths is a positive number, it means that the action reduces the average queue length of the intersection; if it is a negative number, it means that the action increases the average queue length of the intersection;

[0052] The green light time utilization rate is obtained by calculating the green light time utilization rates of the two traffic directions in the phase; if the green light time utilization rate is larger, it means that the number of vehicles passing through the traffic direction corresponding to the phase in a unit time period is larger, and the contribution to improving the traffic efficiency of the intersection is greater; otherwise, the contribution is smaller. The formula of the green light time utilization rate is:

[0053]

[0054] in, represents the green light time utilization rate of the traffic direction di at intersection i, where di represents a certain traffic direction at the intersection. represents the traffic flow through the intersection in the di direction of intersection i, Represents the road capacity in direction di of intersection i.

[0055] In this embodiment, di is defined as {go straight from north to south, turn left from north to south, go straight from east to west, turn left from east to west, go straight from south to north, turn left from south to north, go straight from west to east, turn left from west to east}.

[0056] The green light duration penalty term e1 is the minimum green light duration g min and maximum green light duration g max A penalty term is established. If the duration of the green light is within the green light range, e1 is 0. If it is not, the closer it is to the green light range, the smaller e1 is, and vice versa.

[0057] The red light waiting time penalty term e2 is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e2 is φ; if it does not exist, then the penalty term e2 is 0;

[0058] The traffic condition reward and penalty items e3 in each direction are determined by judging each maximum signal light cycle cy max The traffic conditions of all traffic directions within the period cy are used to establish reward and penalty items; if the period cy max If all traffic directions in the vehicle are clear, e3 is the reward φ; if not, it is the penalty φ, i.e., a negative number φ;

[0059] Then, the penalty function is expressed as follows:

[0060]

[0061] Among them, φ represents the reward and penalty scale, g represents the duration of the green light in the current phase, and rw represents the duration of the phase waiting for the red light;

[0062] The formula of the fairness reward function of intersection i after weighting is:

[0063]

[0064] Among them, r i is the fairness reward function of intersection i; is the average queue length at intersection i at the current time t, is the average queue length of intersection i at the previous moment; k1, k2, k3, k4, k5, k6 are weight coefficients respectively; They represent the green light time utilization rates of the traffic directions di1 and di2 corresponding to the current green light phase of intersection i; travel_time i is the average travel time at intersection i, if travel_time i If it is not 0, travel_time i The smaller it is, the faster the vehicles pass through the intersection and the better the traffic condition. i The larger the value, the worse the operation condition of the intersection; if travel_time i If it is 0, it is necessary to judge whether there is no vehicle entering the corresponding road section or there is traffic congestion that prevents vehicles from passing through the corresponding road section based on the queue length. If there are no vehicles in the queue, there is no vehicle entering the corresponding road section. If there are vehicles in the queue, there is traffic congestion.

[0065] S3, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection, two networks with the same structure and different parameters are constructed for each intersection, namely, an estimation network and a target network; the estimation network calculates the estimated Q value corresponding to each action under the current intersection state; the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed.

[0066] In this embodiment, for each agent corresponding to an intersection, two networks with the same structure but different parameters are constructed to update the Q value. The estimation network of the agent at intersection i is responsible for calculating the current state The Q value corresponding to each action is The target network is responsible for calculating the intersection state after executing the action The cumulative value to the final state, i.e. the target value Calculate target value Medium status When the corresponding maximum estimated Q value is reached, the state is first calculated by estimating the network The action corresponding to the maximum estimated Q value Reuse the target network to calculate the state action The corresponding target Q value.

[0067] Furthermore, the target value is calculated as follows:

[0068]

[0069] in, is the target value of intersection i at time t; Respectively represent the execution actions Traffic flow status before and after; represents the action chosen by the agent at intersection i at time t; Indicates execution of an action After the reward; Status Next action The corresponding estimated Q value; Indicates in status Select the action corresponding to the maximum Q value in the estimated network γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

[0070] S4, taking the traffic status of the upstream road section exit of the adjacent intersection as the communication content, constructing a state communication mechanism for multiple intersections, selecting the road section at a preset distance from the upstream exit of the intersection as the monitoring target, judging whether continuous congestion occurs by monitoring the vehicle queuing situation of the upstream exit section, constructing a continuous congestion penalty item, and combining the fairness reward function with the continuous congestion penalty item to obtain the target reward function.

[0071] In this embodiment, Figure 3 As shown in the figure, for the downstream intersection, the arriving traffic flow is composed of the outgoing traffic flow of the upstream intersection and the traffic flow originating from this section. Therefore, the traffic lights at the upstream intersection control the incoming traffic flow of the downstream intersection, which in turn indirectly affects the traffic light control of the downstream intersection. For the upstream intersection, the premise for effectively controlling vehicle traffic is that the downstream intersection is not blocked and the outgoing vehicles of the upstream section can successfully exit the downstream section. Therefore, the traffic light control of the downstream intersection also indirectly affects the traffic light control of the upstream intersection.

[0072] When optimizing the traffic light control strategy, each intersection needs to obtain more traffic dynamics of adjacent intersections to provide auxiliary decision-making information for the coordination between the traffic lights. The traffic status of the upstream road section exits of the adjacent intersections is used as the communication content to build a state communication mechanism for multiple intersections.

[0073] In this embodiment, the L-meter lane of the upstream exit section is divided into cells of length c meters. If there is a vehicle in the cell, the value of the position pos is 1, otherwise it is 0. When there is a vehicle in the cell, the vehicle speed is compared with the maximum speed v of the road limit. max Normalize it and record it as the value of speed v;

[0074] The vehicle position matrix of the upstream exit section of intersection i is expressed as The velocity matrix is ​​expressed as The vehicle position matrix of the entrance lane of intersection i is expressed as The velocity matrix is ​​expressed as

[0075] The traffic flow state of the entrance lane of intersection i and the traffic status of the upstream exit section Combined into the state variables of intersection i

[0076] Where i represents the intersection number; ent i represents the entrance lane of intersection i, adj i represents the upstream exit lane of intersection i; n6 represents the number of cells for lane division, m represents the cell number, and T represents the transposed matrix.

[0077] In this embodiment, due to the influence of the control of adjacent traffic lights, when the downstream traffic lights cannot release the traffic flow from the upstream in time, vehicles will be piled up in the upstream exit section, which will cause continuous congestion at the upstream and downstream intersections. In order to avoid the occurrence of such extreme situations, a penalty function cg is established, and the section of L meters upstream of the intersection exit is selected as the monitoring target. By monitoring the vehicle queuing situation in the section, it is determined whether continuous congestion occurs. If there are queued vehicles, it means that continuous congestion has occurred, and a penalty is given to avoid the next intelligent agent's wrong decision; if there are no queued vehicles, no penalty is given.

[0078] Preferably, the formula of the continuous congestion penalty term is:

[0079]

[0080] Among them, cg is the continuous congestion penalty item.

[0081] Then construct the reward function of each agent, that is, the target reward function.

[0082] The target reward function is a weighted sum of the fairness reward function, the number of vehicles passing through the area, and the continuous congestion penalty term, as the reward function of each agent, and its formula is:

[0083]

[0084] Among them, R i is the target reward function of intersection i, k7, k8, k9 are weight coefficients, tr is the number of vehicles passing through the area, is the upstream exit lane adj of intersection i i The continuous congestion penalty term; n6 represents the number of exit lanes, q' o represents the outgoing traffic flow of lane o.

[0085] S5, calculate the estimated Q value corresponding to each action under the intersection state at the current moment according to the estimation network of each intersection, after the intelligent agent at the current intersection selects the action using the ε-greedy strategy, the selected phase executes the action, and calculates the reward value through the target reward function to obtain the traffic state at the next moment after the action is executed; the data of the intersection state, action, reward value, and traffic state at the next moment are stored in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the traffic lights of the corresponding intelligent agents at all intersections.

[0086] In this embodiment, the estimated network in the agent network structure of the constructed intersection i is used to calculate the intersection i in state The estimated Q value corresponding to each action is given, and the agent adopts the ε-greedy strategy when selecting actions;

[0087] When selecting actions, the agent adopts the ε-greedy strategy to select actions, and realizes exploration and utilization in reinforcement learning by adjusting the exploration rate ε; where ε represents the probability of randomly selecting an action, 1-ε represents the probability of selecting the action corresponding to the maximum estimated Q value, and ε decreases with the increase of the number of iterative training, realizing the process of transitioning from exploration to utilization.

[0088] Select an action based on the current selection strategy, and execute the action in the selected phase, such as turning the light green. Calculate the target reward function R based on the output simulation data i , and obtain the traffic status after executing the action Store the sample data into the experience replay set.

[0089] S6, taking the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby obtaining the signal light phase control model of each intersection.

[0090] In this embodiment, a multi-signal light coordinated control algorithm process is executed to train a multi-intersection signal light coordinated control model.

[0091] During the training process, the historical traffic flow of the external entrance of the area is As the input of the multi-intersection signal control model, the signal control scheme of each intersection as the output of the model.

[0092] Among them, h represents the historical traffic data sign, j∈[1,n3] represents the entrance lane number, and j′∈[1,n2] represents the external entrance lane number.

[0093] In this embodiment, when the parameters of the algorithm network are updated by the gradient descent method, a CNN network is used to fit the Q value function corresponding to the state action to solve the problem that the Q value function corresponding to each state cannot be listed when continuous large-scale states are generated during the operation of traffic flow. The CNN network is divided into 4 layers, the first layer consists of 32 convolution kernels; the second layer consists of 64 convolution kernels; the third layer is a fully connected layer consisting of 128 neurons; and the fourth layer is a fully connected layer consisting of 64 neurons.

[0094] In this embodiment, the loss function MSE is used to calculate the error between the target value and the estimated Q value. The formula of the loss function MSE is:

[0095]

[0096] in, is the current state of intersection i Next action The corresponding estimated Q value.

[0097] The update formula of the estimation network is:

[0098]

[0099] Among them, α represents the learning rate, and the estimated network and target network are updated at frequencies f1 and f2 respectively.

[0100] After the network model is built, set the maximum number of simulations ST max Each simulation process of ST is set with a simulation time T, and the calculation is iterated within T. If the current time t exceeds T, the number of simulations ST increases. If the number of simulations ST exceeds ST max Finally, the signal light phase control strategy of the intersection is output.

[0101] S7, obtaining real-time traffic flow data of the external entrances of multiple intersections, and inputting the signal light phase control model, and outputting the signal light phase control scheme of each intersection under the real-time traffic flow data.

[0102] In this embodiment, Figure 4As shown, according to the trained multi-intersection signal light coordinated control model, it is applied to the real-time traffic flow data of the working condition for verification, and the real-time traffic flow of the external entrances of the multiple intersections is converted into As the input of the model, the model outputs the signal light phase control scheme for each intersection under real-time traffic flow data:

[0103]

[0104] Wherein, w represents the traffic data sign under working conditions.

[0105] In order to verify the effectiveness of the model and model solution proposed by the present invention, the present invention provides another preferred embodiment to illustrate the application of the present invention.

[0106] Among them, the intersections of Diezhang Lingxi Road-Diezhang Zhuangyuan Road-Aofeng Lingxi Road-Aofeng Zhuangyuan Road in Xuancheng City, Anhui Province were selected as experimental objects. The structure of multiple intersections is as follows: Figure 5 The data set is taken from the open data set of Xuancheng City, Anhui Province, including GIS data, intersection lane traffic volume data and intersection lane information on December 15, 2016. This embodiment selects the traffic volume data during the peak period of 7:30-8:30 and the off-peak period of 10:00-11:00.

[0107] This embodiment uses traffic flow data during peak hours for training, and then applies the trained signal light coordination control model to peak and off-peak hours for verification to evaluate the coordination control effect of the signal lights.

[0108] The average travel time Avg_T, the average queue length Avg_Q, the maximum average travel time Max_T, the maximum average queue length Max_Q, and the number of vehicles tr passing through the selected area at each intersection are selected as evaluation indicators. The shorter the average travel time Avg_T, the faster the vehicles can pass through the intersection and the better the traffic conditions at the intersection; the longer the average travel time, the worse the traffic conditions at the intersection. The larger the average queue length Avg_Q, the more serious the congestion at the intersection; the smaller the average queue length, the lighter the congestion at the intersection or the non-existent congestion. Under the same input traffic volume, the more vehicles tr passing through the selected area, the smoother the traffic conditions in the area and the better the control effect of the traffic lights; the fewer vehicles passing through, the more congested the traffic conditions in the area and the worse the control effect of the traffic lights. The specific calculation method is as follows:

[0109]

[0110] Among them, travel_time j represents the average travel time of vehicles on lane j, queue jrepresents the average queue length of lane j, travel_time i represents the average travel time of vehicles at intersection i, queue i represents the average queue length at intersection i.

[0111] In this embodiment, the length of the upstream exit lane is selected as L = 80 meters, the cell size is c = 5 meters, and the maximum speed limit of the road is v max =60km / h. For each intersection, the traffic efficiency of the intersection is as important as the overall traffic efficiency of the region. Continuous congestion is an extreme congestion, which will affect the traffic flow of each intersection in the region. Therefore, this paper sets the weight coefficient of each indicator according to its importance. The parameter settings are shown in Table 1.

[0112] Table 1. Parameter settings of the objective function of traffic light coordination control

[0113]

[0114]

[0115] This embodiment uses the traffic flow data during peak hours to train the multi-signal light coordination control model, and performs 10 verifications with different random seed numbers during peak hours and off-peak hours.

[0116] Tables 2 and 3 are the average control results of 10 simulations during peak and off-peak hours, comparing the control effects of various methods on the average traffic efficiency of the region and the traffic efficiency of the worst intersection.

[0117] Table 2. Simulation control results during peak hours

[0118]

[0119] Table 3. Simulation control results during off-peak hours

[0120]

[0121] Figure 6 It is a statistic of vehicle traffic conditions. Under the condition of the same input traffic flow, the control effect of multiple signal lights can be intuitively reflected by counting the number of vehicles leaving the area. As can be seen from the figure, the number of vehicles passing through the area by the method of the present invention is the largest, and the control effect in 10 verifications is relatively stable, indicating that the signal light control effect of the method of the present invention is better than that of other comparative methods. The results during off-peak hours are similar to those during peak hours, indicating that the multi-signal light coordination control model trained by the method of the present invention can be applied to signal light control under different traffic flows, and has good generalization ability.

[0122] Depend on Figure 7 and Figure 8It can be seen that during peak hours, the average travel time and average queue length at each intersection of the method of the present invention are both small, while the results of the average travel time and average queue length at each intersection of other comparison algorithms are quite different, which may cause congestion at some intersections. The traffic efficiency of each intersection during off-peak hours is similar to the results during peak hours.

[0123] Tables 2 and 3 are comparisons of the regional control results of various methods. The average value represents the average control effect of each intersection, and the maximum value represents the worst control effect in the region. Compared with the average value, the average travel time of the method of the present invention is 22.89 seconds, and the average queue length is 3.64 meters; compared with the maximum value, the average travel time of the method of the present invention is 36.22 seconds, and the average queue length is 7.56 meters; all are better than the uncoordinated control method NC-FR, the coordinated control method MFT-FR, and the multi-signal coordinated control method QT-FR based on fairness reward mechanism and Q value migration. Comparing NC-FR with MFT-FR and QT-FR, the control result of the coordinated control method MFT-FR at the maximum value is better than that of the uncoordinated control method NC-FR. The experimental results show that the method of the present invention can effectively improve the traffic efficiency in multi-intersection areas and has practical application value.

[0124] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0125] The present invention provides a multi-traffic light control method based on multi-agent reinforcement learning, and constructs a traffic light phase control model based on reinforcement learning and a fairness reward mechanism, taking into account the traffic efficiency and fairness respectively. By constructing corresponding multiple agents and network structures for each intersection, a state communication mechanism between the intersections and a multi-signal light coordination control objective function are constructed to achieve the purpose of controlling the traffic lights at multiple intersections. The trained multi-intersection signal light coordination control model is applied to real-time traffic data under working conditions. The results show that the method of the present invention can effectively improve the traffic efficiency of multi-intersection areas, and provide a reference for the design of actual traffic light phase control.

[0126] Embodiment 2

[0127] like Fig. 9 As shown, the second embodiment of the present invention further provides a multi-traffic signal light control device based on multi-agent reinforcement learning, comprising:

[0128] A multi-intersection data acquisition unit, used to acquire the road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in the selected area; the traffic data includes the volume of vehicles entering during the preset time period;

[0129] A multi-agent construction unit is used to construct an agent, and construct an agent corresponding to each intersection according to the road network structure and the traffic data, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction;

[0130] An algorithm network construction unit is used to construct two networks with the same structure and different parameters for each intersection, namely, an estimation network and a target network, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed;

[0131] A target reward function construction unit is used to use the traffic flow state of the upstream road section exit of the adjacent intersection as the communication content, construct a state communication mechanism for multiple intersections, select a road section at a preset distance from the upstream exit of the intersection as a monitoring target, determine whether continuous congestion occurs by monitoring the vehicle queue situation of the upstream exit section, construct a continuous congestion penalty item, and combine the continuous congestion penalty item according to the fairness reward function to obtain a target reward function;

[0132] The multi-intersection coordination control unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network of each intersection. The intelligent agent at the current intersection adopts After the strategy selects an action, the selected phase executes the action, and the reward value is calculated through the target reward function to obtain the traffic state at the next moment after the action is executed; and the data of the intersection state, action, reward value, and traffic state at the next moment are stored in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the signal lights of the corresponding agents at all intersections;

[0133] A model training and updating unit is used to take the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, use the loss function MSE to calculate the error between the target value and the estimated Q value, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to obtain the signal light phase control model of each intersection;

[0134] The output unit is used to obtain real-time traffic flow data of the external entrances of multiple intersections, input the signal light phase control model, and output the signal light phase control scheme of each intersection under the real-time traffic flow data.

[0135] Embodiment 3

[0136] The third embodiment of the present invention also provides a multi-traffic light control device based on multi-agent reinforcement learning, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the multi-traffic light control method based on multi-agent reinforcement learning as described above.

[0137] Embodiment 4

[0138] The fourth embodiment of the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the multi-traffic light control method based on multi-agent reinforcement learning as described above is implemented.

[0139] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0141] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0142] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0143] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0144] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0145] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-traffic signal control method based on multi-agent reinforcement learning, It is characterized in that include: S1, obtaining a road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in a selected area; and the traffic data includes the volume of vehicles entering during the preset time period; S2, constructing an intelligent agent corresponding to each intersection according to the road network structure and the traffic data, wherein the intelligent agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction; S3, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection, two networks with the same structure and different parameters are constructed for each intersection, namely, an estimation network and a target network; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed; S4, taking the traffic flow state of the upstream road section exit of the adjacent intersection as the communication content, constructing a state communication mechanism for multiple intersections, selecting a road section at a preset distance from the upstream exit of the intersection as a monitoring target, judging whether continuous congestion occurs by monitoring the vehicle queue situation of the upstream exit section, constructing a continuous congestion penalty item, and combining the fairness reward function with the continuous congestion penalty item, thereby obtaining a target reward function; S5, calculate the estimated Q value corresponding to each action under the intersection state at the current moment according to the estimation network of each intersection, after the intelligent agent at the current intersection selects the action using the ε-greedy strategy, the selected phase executes the action, and calculates the reward value through the target reward function to obtain the traffic state at the next moment after the action is executed; store the data of the intersection state, action, reward value, and traffic state at the next moment in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the traffic lights of the corresponding intelligent agents at all intersections; S6, taking the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby obtaining the signal light phase control model of each intersection; S7, obtaining real-time traffic flow data of the external entrances of multiple intersections, and inputting the signal light phase control model, and outputting the signal light phase control scheme of each intersection under the real-time traffic flow data.

2. According to claim 1, a multi-traffic signal light control method based on multi-agent reinforcement learning, It is characterized in that When constructing the fairness reward function, we consider traffic efficiency and fairness separately. In terms of traffic efficiency, we consider the impact of each decision period and a traffic light cycle. In the decision-making period, the change of queue length and the green light time utilization rate of each phase are selected to reflect the short-term impact of the current decision on the traffic efficiency within a unit time; in a traffic light cycle, the average travel time is selected to evaluate the impact of the selected action set on the traffic efficiency within a cycle; in terms of fairness, constraints are established and penalty functions are set according to the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle; among them, The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points; if the difference between the average queue lengths is a positive number, it means that the action reduces the average queue length of the intersection; if it is a negative number, it means that the action increases the average queue length of the intersection; The green light time utilization rate is obtained by calculating the green light time utilization rates of the two traffic directions in the phase; if the green light time utilization rate is larger, it means that the number of vehicles passing through the traffic direction corresponding to the phase in a unit time period is larger, and the contribution to improving the traffic efficiency of the intersection is greater; otherwise, the contribution is smaller; then, the formula of the green light time utilization rate is: in, represents the green light time utilization rate of the traffic direction di at intersection i, where di represents a certain traffic direction at the intersection. represents the traffic flow through the intersection in the direction di of intersection i, It represents the road capacity in the direction di of intersection i; The green light duration penalty term e 1 The minimum green light duration is g min and maximum green light duration g max Establish a penalty item. If the green light duration is within the green light range, then e 1 If it does not meet the requirements, the closer to the green light range, the 1 The smaller it is, the bigger it is; The red light waiting time penalty term e 2 It is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e 2 is φ; if it does not exist, the penalty term e 2 is 0; The traffic conditions in each direction are rewarded or penalized 3 By determining each maximum signal light cycle cy max The traffic conditions of all traffic directions within the period cy are used to establish reward and penalty items; if the period cy max If all traffic directions in the area are clear, then e 3 is the reward φ; if it is not satisfied, it is the penalty φ, that is, a negative number φ; The penalty function is expressed as follows: Among them, φ represents the reward and penalty scale, g represents the duration of the green light in the current phase, and rw represents the duration of the phase waiting for the red light; The formula for the fairness reward function of intersection i is obtained as follows: Among them, r i is the fairness reward function of intersection i; is the average queue length at intersection i at the current time t, is the average queue length at intersection i at the previous moment; k 1 , k 2 , k 3 , k 4 , k 5 , k 6 are weight coefficients respectively; They represent the traffic direction di corresponding to the current green light phase of intersection i 1 、di 2 Green light time utilization rate; travel_time i is the average travel time at intersection i.

3. According to claim 2, a method for controlling multiple traffic lights based on multi-agent reinforcement learning, Features ,When constructing the multi-intersection state communication mechanism, the traffic state of the entrance lane of ,intersection i and the traffic state of the upstream exit section are combined as the ,state variables of intersection i; The traffic state of the entrance lane is defined as The traffic state of the upstream exit section is Then the state variable of intersection i is in, represents the vehicle position matrix of the upstream exit section of intersection i; represents the velocity matrix; represents the vehicle position matrix of the entrance lane of intersection i; represents the velocity matrix; Where i represents the intersection number; ent i represents the entrance lane of intersection i, adj i represents the upstream exit lane of intersection i; n 6 represents the number of cells for lane division, m represents the cell number, and T represents the transposed matrix.

4. According to claim 3, a method for controlling multiple traffic lights based on multi-agent reinforcement learning, Features , the calculation formula of the target value is: in, is the target value of intersection i at time t; Respectively represent the execution actions Traffic flow status before and after; represents the action chosen by the agent at intersection i at time t; Indicates execution of an action After the reward; Status Next action The corresponding estimated Q value; Indicates in status Select the action corresponding to the maximum Q value in the estimated network γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

5. According to claim 4, a method for controlling multiple traffic lights based on multi-agent reinforcement learning, Features ,The formula of the continuous congestion penalty term is: Among them, cg is the continuous congestion penalty item.

6. According to claim 5, a method for controlling multiple traffic lights based on multi-agent reinforcement learning, Features ,The target reward function is the weighted sum of the fairness reward function, the number of vehicles passing through the area, and the continuous congestion penalty term, as the reward function of each agent, and its formula is: Among them, R i is the target reward function for intersection i, k 7 , k 8 , k 9 is the weight coefficient, tr is the number of vehicles passing through the area, is the upstream exit lane adj of intersection i i Continuous congestion penalty term; n 6 represents the number of exit lanes, q′ o represents the outgoing traffic flow of lane o.

7. A method for controlling multiple traffic lights based on multi-agent reinforcement learning according to claim 6, Features , the formula of the loss function MSE is: in, is the current state of intersection i Next action The corresponding estimated Q value.

8. The method for controlling multiple traffic lights based on multi-agent reinforcement learning according to claim 7, Features ,The update formula of the estimation network is: Among them, α represents the learning rate.

9. The method for controlling multiple traffic lights based on multi-agent reinforcement learning according to claim 1, Features When the ε-greedy strategy is used to select actions, the exploration and utilization in reinforcement learning are realized by adjusting the exploration rate ε; where ε represents the probability of randomly selecting an action, 1-ε represents the probability of the corresponding action when the maximum estimated Q value is selected, and ε decreases as the number of iterative training increases.

10. A multi-traffic light control system based on multi-agent reinforcement learning, It is characterized in that include: A multi-intersection data acquisition unit, used to acquire the road network structure of multiple intersections and traffic data of a preset time period; wherein the road network structure includes: multiple intersections, multiple external entrances and multiple entrance lanes in the selected area; the traffic data includes the volume of vehicles entering during the preset time period; A multi-agent construction unit is used to construct an agent, and construct an agent corresponding to each intersection according to the road network structure and the traffic data, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the corresponding entrance lane; the action space is all non-conflicting phases of each intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction; An algorithm network construction unit is used to construct two networks with the same structure and different parameters for each intersection, namely, an estimation network and a target network, according to the road network structure, the traffic data and the intelligent agents corresponding to each intersection; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next state to the terminal state of the intersection after the action is executed; A target reward function construction unit is used to use the traffic flow state of the upstream road section exit of the adjacent intersection as the communication content, construct a state communication mechanism for multiple intersections, select a road section at a preset distance from the upstream exit of the intersection as a monitoring target, determine whether continuous congestion occurs by monitoring the vehicle queue situation of the upstream exit section, construct a continuous congestion penalty item, and combine the continuous congestion penalty item according to the fairness reward function to obtain a target reward function; The multi-intersection coordination control unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network of each intersection. The intelligent agent at the current intersection adopts After the strategy selects an action, the selected phase executes the action, and the reward value is calculated through the target reward function to obtain the traffic state at the next moment after the action is executed; and the data of the intersection state, action, reward value, and traffic state at the next moment are stored in the experience playback set until the maximum number of simulations for each intersection is calculated, so as to realize the coordinated control of the signal lights of the corresponding agents at all intersections; A model training and updating unit is used to take the historical traffic flow data of the external entrance of the selected area of ​​each intersection as input, use the loss function MSE to calculate the error between the target value and the estimated Q value, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to obtain the signal light phase control model of each intersection; The output unit is used to obtain real-time traffic flow data of the external entrances of multiple intersections, input the signal light phase control model, and output the signal light phase control scheme of each intersection under the real-time traffic flow data.

Citation Information

Cited By

  • Multi-RIS-assisted wireless network coverage optimization method and device

    CN119155691A

  • Route linkage type traffic signal control method and system based on reinforcement learning

    CN119851491A

  • A route-linked traffic signal control method and system based on reinforcement learning

    CN119851491B

  • Multi-intersection traffic signal lamp cooperative control system and method

    CN119942817A

  • Multi-intersection traffic signal light collaborative control system and method

    CN119942817B