Distributed self-adaptive signal control method considering downstream traffic pressure conduction influence

Abstract

The invention relates to a distributed self-adaptive signal control method considering downstream traffic pressure conduction influence, which comprises the following steps of: estimating the number of queuing vehicles at an entrance lane of a downstream intersection to obtain real-time information of a downstream traffic state; constructing a signal control agent considering the body and traffic states of a plurality of downstream intersection entrance lanes, bringing the number of queuing vehicles into an input state and a reward function of the agent at the same time, and covering all possible signal phase combinations in an action set; a correction coefficient is introduced to adjust a control scheme selection strategy of the intelligent agents, a deep reinforcement learning framework based on a near-end strategy optimization algorithm is constructed, and the intelligent agents continuously optimize the control strategy through a distributed training method; and outputting an optimal control scheme by using the trained intelligent body, and adaptively controlling the intersection signal lamp. Compared with the prior art, the method can achieve the coordination control optimization of the traffic of the whole road network under the condition that only a small number of agents are deployed in the road network.

Description

Technical Field

[0001] The present invention relates to the field of traffic signal control, and in particular to a decentralized adaptive signal control method that takes into account the influence of downstream traffic pressure conduction. Background Art

[0002] Signal control systems in traffic road networks play a vital role in coordinating vehicle flows within urban areas. Traditional signal control networks face several challenges. First, existing traffic models often focus on specific aspects and usually only simulate some typical network characteristics. For example, traffic flow theory, which ignores individual vehicle driving behavior, and queuing theory, fail to fully address the spatiotemporal dynamics in road networks. Second, as the number of network nodes increases, the optimization complexity based on traditional signal control models grows exponentially. Third, the increasing complexity of these problems poses significant challenges to most analytical optimization algorithms, including issues such as non-convexity, high dimensionality, and nonlinearity.

[0003] The continuous evolution of innovative methods has enhanced the comprehensiveness and accuracy of traffic management and monitoring capabilities. Technologies such as holographic intersections and integrated perception systems use advanced sensor technology and real-time data processing to capture the real-time traffic conditions at intersections. The emergence of new intersection control technologies has brought new prospects for traffic management, such as multimodal traffic management and regional traffic optimization. In addition, advances in perception technology and improvements in big data processing capabilities have promoted the integration of artificial intelligence technology with traffic signal control systems.

[0004] Due to the differences in the capabilities of traffic perception and control equipment at different intersections, some intersections will inevitably be upgraded to intelligent systems in advance, while other intersections will continue to use traditional control modes. This means that there may be intersections with different levels of intelligence in the road network, for example, the integration of traditional rule-based traffic lights with traffic lights driven by deep reinforcement learning algorithms. Therefore, it becomes crucial to achieve compatibility between intelligent and non-intelligent intersections within a limited road network. Summary of the invention

[0005] The purpose of the present invention is to provide a decentralized adaptive signal control method that takes into account the influence of downstream traffic pressure transmission. The method is based on a deep reinforcement learning algorithm based on the proximal policy optimization algorithm (PPO). Under the decentralized control framework, based on the "one (entity) to many (downstream intersections)" topology mode, a small number of intelligent agents are deployed in the road network to successfully achieve coordinated control optimization of the entire road network traffic.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] The Autonomous Traffic System project is dedicated to optimizing urban traffic flow and traffic signal control in an intelligent and adaptive manner to address the limitations of traditional traffic signal control systems in dynamic traffic environments. The core goal of the project is to achieve autonomous optimization of traffic signal control, thereby improving the overall efficiency of the traffic system. Based on the Autonomous Traffic System project, the present invention relates to a decentralized adaptive signal control method that considers the influence of downstream traffic pressure transmission, referred to as Downstream Traffic Aware Intersection Controller (DTAIC), which includes the following steps:

[0008] S1, traffic status acquisition: by calculating the number of vehicles flowing out of the main intersection and the downstream intersection, estimating the number of queued vehicles at the entrance of the downstream intersection, and obtaining real-time information on the downstream traffic status;

[0009] S2, agent construction: construct a signal control agent that considers the traffic status of the main body and multiple downstream intersection entrances, incorporates the number of queued vehicles into the input state and reward function of the agent, and covers all possible signal phase combinations in the action set to achieve more accurate control decisions;

[0010] S3, agent training: introduce correction coefficients to adjust the agent's control scheme selection strategy, dynamically adjust the weight of the main body and downstream traffic pressure, and build a deep reinforcement learning framework based on the proximal strategy optimization algorithm. Through the distributed training method, each agent continuously optimizes the control strategy to achieve multi-agent convergence and optimization;

[0011] S4, uses the trained agent to output the optimal control plan and adaptively control the intersection traffic lights.

[0012] Each intersection consists of four entrances, each of which includes two roads: an entry road and an exit road. Each road consists of one or more one-way lanes. The entry lane of each intersection that forms an entry road is denoted as m. k , forming the leaving lane of the road, denoted as n k , where m k and n k Maintain a one-to-one correspondence between them.

[0013] The agent takes lane occupancy, queue length measured by the number of queued vehicles, and the current phase state of the lane entering the main body intersection as input for the initial state.

[0014] The input volume for each lane is considered as a vector containing three values ​​as follows:

[0015]

[0016] Among them, at a certain time step t, occ t and q t They represent the occupancy and queue length of lane i, respectively, and g t is the phase state, which is a binary value indicating whether lane i is in the released state at the current time step t.

[0017] The agent's field of view is expanded so that it can access the data of the R lanes entering the main body and the R lanes entering the four downstream roads. The state of the agent is expressed as:

[0018]

[0019] in, represents the three-value input vector of the lane entering the main intersection at time t, A three-valued input vector representing the incoming lanes of the downstream intersection at time t, where the number of lanes R is obtained by multiplying the number of lanes of each road by 4.

[0020] The actions of the agent consist of all possible combinations of traffic flows in different directions. Any two non-conflicting traffic flows constitute a phase. A set of feasible actions is constructed based on all phases. Each time a decision is made, the agent selects a phase from the set of feasible actions.

[0021] If the next phase selected by the agent is the same as the previous phase, the green light of the phase is directly executed for the first preset duration;

[0022] If the next phase selected by the agent is different from the previous phase, it means that the phase needs to be changed. First, the yellow light of the previous phase is executed for the second preset time. After the yellow light is executed, the green light of the next phase is executed for the third preset time. The sum of the second preset time and the third preset time is equal to the first preset time.

[0023] The reward of the agent includes the queue length of the main body intersection and the queue length of the downstream lane corrected by the correction coefficient.

[0024] The correction factor is:

[0025]

[0026] in, is the correction coefficient of the exit road j of the main intersection. The exit road j has multiple exit lanes, represented by {n1,...n k ,...} indicates that the corresponding entry lane is {m1,...,m k ,...},K jis a set of exit lane indices belonging to exit road j, and represent lane n in the most recent simulation step k and m k The number of vehicles on.

[0027] The reward of agent i is expressed as:

[0028]

[0029] in, The time step t is the time when the main intersection enters lane m k The queue length, Enter lane n at the downstream intersection at time step t k The queue length.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) Technological innovation: At present, the cooperation mechanism of multi-agent collaboration is mostly based on the environment of the same type of agents. The one-to-many intelligent control mode proposed in this invention can effectively coordinate the cooperation between low-level intersections and high-level intersections in the road network. Simulation experiments have verified the effectiveness of the proposed algorithm.

[0032] (2) Efficient optimization: Adaptive signal control based on deep reinforcement learning can take advantage of real-time big data processing capabilities and high accuracy. In a decentralized control framework, based on the "one (entity) to many (downstream intersections)" topology model, with only a small number of agents deployed in the road network, the coordinated control optimization of the entire road network traffic can be achieved.

[0033] (3) Strong scalability: The adaptive signal control system based on deep reinforcement learning can be flexibly deployed in the road network, and intersections equipped with corresponding hardware facilities can perform signal control based on the control mode proposed by the present invention. Each trained DTAIC intersection obtains data based on the integrated radar and visual environment, and can make a comprehensive judgment on the congestion status of the main body and the downstream pressure, and quickly provide a signal control plan.

[0034] (4) Wide applicability: There are no limitations on the installation and deployment of DTAIC intersections, and there are no requirements for the traffic conditions of the intersection itself. DTAIC intersections select phases based on the queue length transmitted by sensors, and are suitable for intersection signal control simulation scenarios with different traffic flows. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the control process of the present invention;

[0036] Figure 2is a schematic diagram of elements of a typical intersection in an embodiment;

[0037] Figure 3 A schematic diagram of an action set of an intelligent agent in one embodiment;

[0038] Figure 4 A traffic network designed for a simulation experiment in one embodiment and the number of each intersection;

[0039] Figure 5 The changes in the queue lengths of all entrances to the road network when different numbers of DTAIC intersections are deployed in the road network;

[0040] Figure 6 A graph showing the change in queue length (unit: meter) over time at a fixed-timing intersection and a DTAIC intersection in a simulation verification in an embodiment;

[0041] Figure 7 A graph showing the variation of queue length (unit: meter) over time at a fixed-timing intersection and an intersection with three DTAICs in a simulation verification in an embodiment. DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0043] This embodiment provides a decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission, referred to as Downstream Traffic Aware Intersection Controller (DTAIC). Figure 1 As shown, the method comprises the following steps:

[0044] S1, traffic status acquisition: by calculating the number of vehicles flowing out of the main intersection and the downstream intersection, estimating the number of queued vehicles at the entrance of the downstream intersection, and obtaining real-time information on the downstream traffic status.

[0045] The typical intersection used in this embodiment is as follows Figure 2 As shown, each intersection consists of four entrances, each entrance includes two roads: an entry road and an exit road, each road consists of three one-way lanes, and the entry lane of each intersection forming the entry road is represented by m k , forming the leaving lane of the road, denoted as n k , where m k and n k Maintain a one-to-one correspondence between them.

[0046] S2, agent construction: construct a signal control agent that considers the traffic status of the main body and multiple downstream intersection entrances, incorporates the number of queued vehicles into the input state and reward function of the agent at the same time, and covers all possible signal phase combinations in the action set to achieve more accurate control decisions.

[0047] S21, the state of the agent

[0048] The agent takes the lane occupancy, queue length measured by the number of queued vehicles, and the current phase state of the entry lane of the main body intersection as input for the initial state.

[0049] The input volume for each lane is considered as a vector containing three values ​​as follows:

[0050]

[0051] Among them, at a certain time step t, occ t and q t They represent the occupancy and queue length of lane i, respectively, and g t is the phase state, which is a binary value indicating whether lane i is in the released state at the current time step t.

[0052] The agent's field of view is expanded so that it can access the data of the 12 entry lanes of the main body and the 12 entry lanes corresponding to the four downstream roads. The state of the agent is expressed as:

[0053]

[0054] in, represents the three-value input vector of the lane entering the main intersection at time t, A three-valued input vector representing the incoming lanes of the downstream intersection at time t.

[0055] S22, Agent Actions

[0056] For each DTAIC agent, the agent’s actions consist of all possible combinations of different directional traffic flows. Figure 3 As shown in the figure, any two non-conflicting traffic flows constitute a phase. Among the eight flows at an intersection, there are eight possible action choices, and the action set can be constructed based on all phases. Specifically, all the action sets for each intersection are predefined, and each time the agent makes a decision, it selects a phase from the action set.

[0057] If the next phase chosen by the agent is the same as the previous phase, the green light for that phase is directly executed for Δt seconds.

[0058] If the next phase selected by the agent is different from the previous phase, it means that the phase needs to be changed. First, the yellow light of the previous phase lasts for Δt. y seconds, after the yellow light is executed, the green light of the next phase will last for Δt g seconds, and Δt y +Δt g =Δt.

[0059] In this embodiment, the attached Figure 3 The phase combination scheme presented in is used as an action set to further explore the potential of reinforcement learning agents to control and manage intersections.

[0060] S23, Agent Reward

[0061] A well-designed reward can help the agent perceive the surrounding traffic conditions and guide it to choose a suitable downstream intersection to release the traffic flow. The present invention designs a method based on lane connection relationship to correct the reward, so that the agent can focus on the downstream pressure and compare the congestion level, thereby helping it to choose the appropriate phase.

[0062] The original reward of agent i is:

[0063]

[0064] in, The time step t is the time when the main intersection enters lane m k The length of the queue (measured in number of vehicles).

[0065] To coordinate the cooperation between agents, the queue length of the downstream lane is included in the reward and adjusted by the coefficient. The formula for the correction coefficient is:

[0066]

[0067] in, is the correction coefficient of the exit road j of the main intersection. The exit road j has multiple exit lanes, represented by {n1,...n k ,...} indicates that the corresponding entry lane is {m1,...,m k ,...},K j is a set of exit lane indices belonging to exit road j, and represent lane n in the most recent simulation step k and m k For example, in Figure 2In the figure, the exit road marked with a dotted box is exit road 2, and its corresponding exit lanes are n4, n5 and n6, so the index set is K2 = {4,5,6}. The entry lanes m4, m5 and m6 are associated with these three exit lanes respectively. Therefore, the correction coefficient expression of exit road 2 is

[0068]

[0069] The modified reward for DTAIC agent i is obtained by multiplying the queue length of each downstream lane by its corresponding correction factor and then adding the largest resulting value to the reward:

[0070]

[0071] in, The time step t is the time when the main intersection enters lane m k The queue length, Enter lane n at the downstream intersection at time step t k The queue length.

[0072] S3, agent training: introduce correction coefficients to adjust the control scheme selection strategy of the agent, dynamically adjust the influence weights of the agent and downstream traffic pressure, and build a deep reinforcement learning framework based on the proximal strategy optimization algorithm. Through distributed training methods, each agent continuously optimizes the control strategy to achieve convergence and optimization of multiple agents.

[0073] S4, uses the trained agent to output the optimal control plan and adaptively control the intersection traffic lights.

[0074] The topological pattern of "one (entity) to many (downstream intersections)". The Downstream Traffic Aware Intersection Control (DTAIC) mechanism is a fully decentralized multi-agent control method based on the PPO algorithm framework. The DTAIC agent observes the upstream characteristics and combines them with the downstream characteristics, and uses the connection relationship between the lanes of the intersection to judge the difference in the actual traffic flow in and out of the lanes, thereby adjusting the reward of the DTAIC intersection. This correction guides the DTAIC intersection to evaluate the downstream pressure, so as to select a downstream intersection with less congestion to release the traffic flow, so as to balance the traffic flow in the upstream and downstream directions, aiming to reduce the queue length of the entity intersection and the surrounding intersections. In a decentralized control framework, based on the "one to many" topological pattern, the coordinated control optimization of the entire road network traffic is achieved under the condition of deploying only a small number of agents in the road network.

[0075] Figure 4 The 3×3 traffic network and intersection numbering designed for the simulation experiment of this embodiment are shown.

[0076] Figure 5The total queue length (number of vehicles) of all intersection entrances in the entire road network is shown as the number of DTAIC agents increases. As the number of DTAIC agents increases, the total queue length is reduced by 22.4%, 25.9% and 30.8% in networks with one, two and three DTAIC agents, respectively, compared with the baseline method. This confirms the effectiveness of the present invention in improving the efficiency of the entire network.

[0077] Figure 6 The figure shows the changes in the queue lengths (meters) at all intersections when a DTAIC agent is only located at intersection E. As can be seen from the figure, the DTAIC agent significantly reduces the queue length at intersection E and has an indirect impact on other intersections. Specifically, compared with the fixed timing (FT) method, the average queue length at intersection E is reduced by about 55.9%, while the queue lengths at the surrounding fixed timing intersections are also reduced by an average of 21.4%. This further highlights the effectiveness of the designed cooperation mechanism between the DTAIC agent and the FT agent. In addition, Figure 7 As shown, the average queue length at all intersections is reduced by about 33.8% when DTAIC is located at intersections B, D, and I. It is worth noting that among all FT-controlled intersections, intersection E shows the most significant performance improvement, with a reduction of 33.2%.

[0078] The present invention relates to a decentralized adaptive signal control method that considers the impact of downstream traffic pressure transmission, called the Downstream Traffic Aware Intersection Controller (DTAIC) collaboration mechanism. The mechanism calculates the traffic flow difference between the inbound and outbound lanes based on the lane-to-lane connection to adjust the reward of the DTAIC intersection. This adjusted reward helps the DTAIC intersection to assess downstream pressure and promote strategic phase selection to balance traffic flow in the upstream and downstream directions. Simulation results highlight the effectiveness of the DTAIC method.

[0079] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission, characterized in that: The following steps are involved: S1, traffic status acquisition: by calculating the number of vehicles flowing out of the main intersection and the downstream intersection, estimating the number of queued vehicles at the entrance of the downstream intersection, and obtaining real-time information on the downstream traffic status; S2, agent construction: construct a signal control agent that considers the traffic status of the main body and multiple downstream intersection entrances, incorporates the number of queued vehicles into the input state and reward function of the agent, and covers all possible signal phase combinations in the action set to achieve more accurate control decisions; S3, agent training: introduce correction coefficients to adjust the agent's control scheme selection strategy, dynamically adjust the weight of the main body and downstream traffic pressure, and build a deep reinforcement learning framework based on the proximal strategy optimization algorithm. Through the distributed training method, each agent continuously optimizes the control strategy to achieve multi-agent convergence and optimization; S4, uses the trained agent to output the optimal control plan and adaptively control the intersection traffic lights.

2. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 1, characterized in that: Each intersection consists of four entrances, each of which includes two roads: an entry road and an exit road. Each road consists of one or more one-way lanes. The entry lane of each intersection that forms an entry road is denoted as m. k , forming the leaving lane of the road, denoted as n k , where m k and n k Maintain a one-to-one correspondence between them.

3. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 2, characterized in that: The agent takes lane occupancy, queue length measured by the number of queued vehicles, and the current phase state of the lane entering the main body intersection as input for the initial state.

4. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 3, characterized in that: The input volume for each lane is considered as a vector containing three values ​​as follows: Among them, at a certain time step t, occ t and q t They represent the occupancy and queue length of lane i, respectively, and g t is the phase state, which is a binary value indicating whether lane i is in the released state at the current time step t.

5. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 4, characterized in that: The agent's field of view is expanded so that it can access the data of the R lanes entering the main body and the R lanes entering the four downstream roads. The state of the agent is expressed as: in, represents the three-value input vector of the lane entering the main intersection at time t, A three-valued input vector representing the incoming lanes of the downstream intersection at time t, where the number of lanes R is obtained by multiplying the number of lanes of each road by 4.

6. The decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 1 is characterized in that: The actions of the agent consist of all possible combinations of traffic flows in different directions. Any two non-conflicting traffic flows constitute a phase. A set of feasible actions is constructed based on all phases. Each time a decision is made, the agent selects a phase from the set of feasible actions.

7. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 6, characterized in that: If the next phase selected by the agent is the same as the previous phase, the green light of the phase is directly executed for the first preset duration; If the next phase selected by the agent is different from the previous phase, it means that the phase needs to be changed. First, the yellow light of the previous phase is executed for the second preset time. After the yellow light is executed, the green light of the next phase is executed for the third preset time. The sum of the second preset time and the third preset time is equal to the first preset time.

8. The decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 1 is characterized in that: The reward of the agent includes the queue length of the main body intersection and the queue length of the downstream lane corrected by the correction coefficient.

9. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 8, characterized in that: The correction factor is: in, is the correction coefficient of the exit road j of the main intersection. The exit road j has multiple exit lanes, represented by {n1,...n k ,...} indicates that the corresponding entry lane is {m1,...,m k ,...},K j is a set of exit lane indices belonging to exit road j, and represent lane n in the most recent simulation step k and m k The number of vehicles on.

10. A decentralized adaptive signal control method considering the influence of downstream traffic pressure transmission according to claim 9, characterized in that: The reward of agent i is expressed as: in, The time step t is the time when the main intersection enters lane m k The queue length, Enter lane n at the downstream intersection at time step t k The queue length.

Images