A long-distance single-lane two-way traffic multi-point signal adaptive cooperative optimization method and system based on multi-agent reinforcement learning

Abstract

The application provides a kind of long-distance single lane two-way traffic multi-point signal adaptive cooperative optimization method and system based on multi-agent reinforcement learning.The method sets meeting point and signal light at both ends and inside of road section, defines each signal light as agent, and constructs multi-agent reinforcement learning environment.The agent takes maximizing long-term cumulative reward as target, adjusts signal timing strategy in coordination with local traffic state and neighbor information.The system includes signal light, sensing device, computing unit and control unit, supports distributed cooperative learning and real-time control.The application solves the problems of insufficient adaptability and high computing requirement of traditional method, improves traffic efficiency, reduces delay and ensures safety, and has strong adaptability, high cooperative efficiency and robustness.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic control, specifically relating to an adaptive collaborative optimization method and system for multi-point signals of long-distance single-lane bidirectional traffic based on multi-agent reinforcement learning. Background Technology

[0002] Long-distance single-lane two-way traffic sections are an important part of the transportation network in specific areas. However, the contradiction between single-lane and two-way traffic demands leads to problems such as low capacity, large vehicle delays, and high safety risks, which are particularly prominent during peak tourist seasons. Existing control methods are mostly "two-end control strategies," encompassing fixed times, fixed queues, platoon control, adaptive control, human flag bearers, portable traffic lights, Automatic Flag Assist Devices (AFADs), and guide vehicles. However, these strategies become drastically less efficient as the road length increases, because vehicle clearance time increases linearly with distance, leading to longer signal cycles and increased waiting times. For example, as the road length increases from 250 meters to 2000 meters, the average delay increases by 3-4 times. Even the optimal adaptive control cannot completely avoid this limitation.

[0003] While some studies have explored segmented control using induction coils and traffic lights within road sections, these methods suffer from high deployment costs, limited applicability, and a failure to achieve effective coordination. Therefore, existing technologies struggle to efficiently, safely, and adaptively address the control challenges of long-distance, single-lane, two-way traffic. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for adaptive cooperative optimization of multi-point signals in long-distance single-lane two-way traffic based on multi-agent reinforcement learning. This method achieves adaptive cooperative control of long-distance single-lane two-way traffic, significantly improving traffic efficiency, reducing delays, and ensuring traffic safety.

[0005] The technical method adopted by this invention to solve its technical problem is as follows:

[0006] A method for adaptive cooperative optimization of multi-point signals in long-distance single-lane bidirectional traffic based on multi-agent reinforcement learning, comprising:

[0007] Step S100: Infrastructure deployment and regional division;

[0008] Step S200: Construction of the multi-agent system;

[0009] Step S300: Define the motion space and define safety constraints;

[0010] Step S400, Design of reward function;

[0011] Step S500: Multi-agent reinforcement learning training;

[0012] Step S600: Real-time control and policy deployment.

[0013] In some preferred embodiments, step S100, infrastructure deployment and regional division, includes:

[0014] Step S110: Meeting points are set at both ends and inside the long-distance single-lane two-way traffic section. The meeting points include one-way arrival and departure platforms at both ends and two-way arrival and departure platforms inside. A controllable traffic section is formed between adjacent meeting points.

[0015] Step S120: Install traffic lights and traffic information sensing devices at the entrance and exit of each meeting point. The sensing devices are used to collect traffic data such as the number of vehicles in the queue, vehicle speed, vehicle type, and estimated arrival time.

[0016] In some preferred embodiments, step S200, multi-agent construction, includes:

[0017] Step S210: Define each traffic light or traffic light group at each meeting point as an agent Agenti, and construct a multi-agent system MAS={Agent1,Agent2,…,Agentn}, where n is the total number of agents;

[0018] Step S220, the state space S of each agent Agenti i Represented as: ,in, This indicates the current phase status and duration of the traffic light. This represents the number of vehicles queuing at the entrance. For downstream section occupancy status, Information on vehicles that are about to arrive upstream. This refers to the state information of neighboring agents.

[0019] In some preferred embodiments, in step S300, the action space is defined and safety constraints are set, wherein the agent's action space A i This includes: maintaining the current green light phase (action a1), switching to the opposite green light phase (action a2, which requires a safe clearing time t). clear ), extend the current green light phase (action a3, limited by the minimum green light time t) min and maximum green light time t max Phase switching must meet safety constraints: there are no vehicles waiting in the current green light direction, or waiting vehicles are within t... clear The contents can be cleared.

[0020] In some preferred embodiments, in step S400, the reward function design method is to design a multi-objective reward function R=R delay +Rsafety +R throughput +R cooperation Among them, the delay reward is: , This represents the current queue length at the entrance. >0 represents the weighting coefficient; safety penalty: if a conflict occurs, then... , , Parameters indicating the occurrence of a conflict Indicates a conflict has occurred; Passage reward: When the queue is cleared or traffic flows efficiently, , The number of vehicles passing through. >0; Collaborative reward: based on global total delay D total Total traffic volume C total , , , >0;

[0021] In some preferred embodiments, in step S500, the multi-agent reinforcement learning training is performed using a multi-agent reinforcement learning algorithm (such as QMIX, MAPPO), where the agents interact with a traffic simulation environment (such as SUMO, VISSIM) to maximize long-term cumulative rewards. ,in ∈[0,1] is the discount factor. During training, the agent synchronously updates the policy network π. θ (a|s) or value function network The parameter θ is optimized through empirical replay and gradient descent.

[0022] In some preferred embodiments, step S600, real-time control and strategy deployment, includes:

[0023] Step S610: Deploy the trained agent policy model (deep neural network) to the roadside computing unit (edge ​​computing device);

[0024] Step S620: The computing unit acquires sensor data in real time and generates a state vector S. i The optimal action is obtained through strategy model reasoning. Control the traffic lights to perform phase switching or extension;

[0025] In step S630, the system supports online fine-tuning of learning, updating model parameters through real-time data feedback to adapt to dynamic changes in traffic flow.

[0026] In a second aspect, the present invention proposes a long-distance single-lane bidirectional traffic multi-point signal adaptive collaborative optimization system based on multi-agent reinforcement learning, comprising an infrastructure module, an agent module, a control execution module, and a training and optimization module;

[0027] The infrastructure module is configured as a meeting point and traffic light unit and a sensing device unit. The meeting point and traffic light unit consists of bidirectional / one-way arrival and departure platforms set up at both ends and inside the road segment. Traffic lights are deployed at the entrance and exit of each platform, supporting red, green and yellow phase switching and safe clearing time control. The sensing device unit includes radar, camera and induction coil, which collects data such as vehicle count, queue length, speed and vehicle type in real time, with an accuracy error of no more than 5%.

[0028] The intelligent agent module is configured as a state perception subunit, a decision-making and reasoning subunit, and a communication subunit. The state perception subunit generates an intelligent agent state vector S by integrating local traffic light states (phase, duration) and sensor data. i It receives state information from neighboring agents, and the decision-making and reasoning subunit has a pre-trained reinforcement learning policy model built in, based on the state vector S. i Output action a i It supports minimum / maximum green light time constraints and safe clearance time calculation. The communication subunit enables the exchange of state information between adjacent intelligent agents (such as queue length and release direction), and the communication delay does not exceed 100ms.

[0029] The control execution module is configured as a signal control unit and a computing unit. The signal control unit receives action commands from the intelligent agent and executes signal light phase switching or extension to ensure safety constraints (such as t). clear The computing unit adopts a distributed edge computing architecture, with each agent equipped with an independent computing device (CPU ≥ 2.0 GHz, memory ≥ 4 GB) to support real-time policy reasoning (single reasoning time < 50 ms).

[0030] The training and optimization module is configured as a simulation environment subunit, an algorithm training subunit, and an online learning subunit. The simulation environment subunit constructs a high-fidelity simulation scenario based on road segment physical parameters (length, distance between meeting points) and traffic rules, and supports multi-agent collaborative training. The algorithm training subunit integrates multi-agent reinforcement learning algorithms such as QMIX and MAPPO, and supports batch training (≥104 iterations per training session) and model parameter tuning. The online learning subunit collects real-time traffic data and periodically updates the strategy model (update cycle ≤1 hour) to adapt to changes in traffic patterns.

[0031] The beneficial effects of this invention are:

[0032] (1) Strong adaptability: When the controller based on multi-agent reinforcement learning dynamically adjusts the signal according to the real-time traffic flow, there is no need to preset a fixed pattern or a complex explicit model, and it can respond flexibly to traffic fluctuations and emergencies.

[0033] (2) High coordination efficiency: The agent learns to achieve implicit or explicit coordination, discovers complex coordination strategies that go beyond traditional rules or simple optimization models, effectively shortens the signal cycle, reduces vehicle waiting time, and significantly reduces the total system delay.

[0034] (3) Safety assurance: Through carefully designed reward functions (especially safety penalty items) and safety rule coverage layers, ensure that the learned strategies meet basic traffic safety requirements;

[0035] (4) Scalability and robustness: The distributed control architecture has excellent scalability, and it is convenient to add or reduce meeting points / intelligent agents. The system is robust to the failure of some sensors or intelligent agents.

[0036] (5) Real-time decision-making capability: After training, the strategy reasoning speed based on the neural network is very fast, which meets the real-time requirements of traffic signal control. Attached Figure Description

[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 : A flowchart of the method described in this invention;

[0039] Figure 2 : Schematic diagram of traffic light settings and two-way alternating traffic at long-distance single-lane meeting points;

[0040] Figure 3 Optimized signal timing scheme and vehicle trajectory map;

[0041] Figure 4 Sensitivity analysis chart regarding average velocity;

[0042] Figure 5 Sensitivity analysis chart of the proportion of large trucks on delays;

[0043] Figure 6 Sensitivity analysis chart regarding minimum green light time. Detailed Implementation

[0044] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] During peak tourist seasons, scenic area roads generally suffer from low traffic capacity, significant vehicle delays, and high safety risks. Taking a long-distance single-lane road in a scenic area (2000 meters in total length, with 3 internal passing points spaced 500 meters apart) as an example, the specific implementation steps are as follows: Figure 1 As shown:

[0047] Step A100, Infrastructure Deployment and Regional Division, such as Figure 2 The diagram illustrates the physical layout for applying this method, including the meeting points (arrival / departure platforms) at both ends of the road segment and within the segment, along with the corresponding traffic light setup. One-way arrival / departure platforms are set up at both ends of the road segment, each equipped with one set of traffic lights (upbound entrance light and downbound entrance light). Three bidirectional meeting points within the segment each have two sets of traffic lights (upbound and downbound entrance / exit lights). The platform capacity is ≥5 vehicles. A radar-visual fusion sensor is installed to collect real-time data on queue length (accuracy ±1 vehicle), vehicle speed (accuracy ±2km / h), and estimated arrival time (error ±3s).

[0048] Step A200, Data Acquisition and State Construction, Agent State Vector S i Example: [Green light, 15s, 8 vehicles, 0 vehicles, 5 vehicles / 10s, downstream green light status] indicates that the current green light has lasted for 15s, there are 8 vehicles queuing at the entrance, there are no vehicles downstream, 5 vehicles will arrive upstream within 10s, the downstream agent is in a green light state, and adjacent agents exchange queue length and release direction information through LTE-V2X communication, with a communication success rate of ≥99%;

[0049] Step A300: Multi-agent reinforcement learning training using the MAPPO algorithm. The state space dimension is set to 10, the action space dimension to 3, and the reward function weights are λ1=0.5, λ2=100, λ3=1, λ4=0.1, and λ5=0.5. After 100,000 iterations in the SUMO simulation environment, the total system delay at convergence is reduced by 40% compared to fixed-period control, and the traffic efficiency is improved by 35%.

[0050] Step A400, real-time control: When an agent detects that the queue length at the entrance is ≥10 vehicles and the downstream section is idle, the strategy model outputs the action "extend the green light", with an extension time Δt=min(t max -t current ,⌈q i / vavg⌉), where vavg is the current average speed of the traffic flow. When the phase changes, a 5-second safety clearance time is forcibly inserted to ensure that no vehicles are left behind and enter the opposite lane.

[0051] Step A500 involves online optimization, with offline retraining performed every morning at midnight. Model parameters are updated based on traffic data from the previous 24 hours. In the event of sudden congestion (such as queue length exceeding platform capacity), online fine-tuning is triggered with an adjustment cycle of 10 minutes to quickly respond to traffic fluctuations.

[0052] Figure 3 The optimized signal timing scheme and vehicle trajectory diagram are shown. In a specific case, the optimal signal timing scheme calculated using the method of this invention is illustrated (green bars represent green light phases), along with the spatiotemporal trajectories of uphill (blue trajectory) and downhill (red trajectory) vehicles within the corresponding time period.

[0053] Figure 4 Sensitivity analysis regarding average speed is presented, showing that the average error gradually decreases as the average speed increases. This reveals that the system of this invention can maintain or improve control efficiency at different average vehicle speeds (manifested as the average error decreases with increasing speed), demonstrating the strong environmental adaptability and efficient optimization capability of its core multi-agent reinforcement learning method under varying traffic flow speeds.

[0054] Figure 5 The figure illustrates the sensitivity analysis of delays caused by the proportion of large trucks, showing that the average error gradually increases with the increase in the proportion of large trucks. This highlights the necessity and superiority of employing multi-agent reinforcement learning for adaptive cooperative control in this invention to address the impact of complex traffic configurations.

[0055] Figure 6 The figure illustrates a sensitivity analysis regarding the minimum green light time, showing that the average delay gradually decreases as the minimum green light time increases. This demonstrates that the multi-agent reinforcement learning system of this invention can effectively learn and optimize decisions under engineering constraints, showcasing the practicality and sophistication of its intelligent decision-making.

[0056] Through the above implementation methods, the present invention achieves adaptive cooperative control of long-distance single-lane two-way traffic, significantly improving traffic efficiency, reducing delays, and ensuring traffic safety.

[0057] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for adaptive cooperative optimization of multi-point signals in long-distance single-lane bidirectional traffic based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step S100, Infrastructure Deployment and Area Division: Meeting points are set up at both ends and inside the long-distance single-lane two-way traffic section. The meeting points include one-way arrival and departure platforms at both ends and two-way arrival and departure platforms inside. A controllable traffic section is formed between adjacent meeting points. Traffic lights and traffic information sensing devices are installed at the entrance and exit of each meeting point. The sensing devices are used to collect the number of vehicles in the queue, vehicle speed, vehicle type, and estimated arrival time. Step S200, Multi-Agent System Construction: Define each traffic light or traffic light group at each meeting point as an agent. i Construct a multi-agent system MAS={Agent1,Agent2,…,Agent2} n }, where n is the total number of agents; each agent is an agent. i State space S i Represented as: ,in, This indicates the current phase status and duration of the traffic light. This represents the number of vehicles queuing at the entrance. For downstream section occupancy status, Information on vehicles that are about to arrive upstream. This refers to the state information of neighboring agents; Step S300, Action Space Definition and Safety Constraints: Agent Action Space A i This includes actions such as maintaining the current green light phase, switching to the opposite green light phase, and performing a safe clearing time t. clear The action extends the current green light phase and is limited to the minimum green light time t. min and maximum green light time t max The phase switching action must meet safety constraints: there are no vehicles lingering in the current green light direction or the lingering vehicles are within t... clear The contents can be cleared; Step S400, Reward Function Design: Design a multi-objective reward function R=R delay +R safety +R throughput +R cooperation Among them, the delay reward is: , This represents the current queue length at the entrance. >0 represents the weighting coefficient; safety penalty: if a conflict occurs, then... , , Parameters indicating the occurrence of a conflict Indicates a conflict has occurred; Passage reward: When the queue is cleared or traffic flows efficiently, , The number of vehicles passing through. >0; Collaborative reward: based on global total delay D total Total traffic volume C total , , , >0; Step S500, Multi-agent Reinforcement Learning Training: Training is performed using the QMIX or MAPPO multi-agent reinforcement learning algorithm. The agents interact with the SUMO or VISSIM traffic simulation environment to maximize long-term cumulative rewards. ,in ∈[0,1] is the discount factor. During training, the agent synchronously updates the policy network π. θ (a|s) or value function network The parameter θ is optimized through empirical replay and gradient descent. Step S600, Real-time Control and Strategy Deployment: The trained agent strategy model based on deep neural networks is deployed to the roadside edge computing unit; the computing unit collects sensor data in real time to generate the state vector S. i The optimal action is obtained through strategy model reasoning. The system controls traffic lights to perform phase switching or extension; it supports online fine-tuning and learning, updating model parameters through real-time data feedback to adapt to dynamic changes in traffic flow.

2. A long-distance single-lane bidirectional traffic multi-point signal adaptive cooperative optimization system based on multi-agent reinforcement learning obtained by the method of claim 1, characterized in that, include: The infrastructure module includes a meeting point and traffic light unit and a sensing device unit, which are used to deploy meeting points, traffic lights and sensors to collect traffic data; The intelligent agent module includes a state perception subunit, a decision reasoning subunit, and a communication subunit, which are used to generate intelligent agent state vectors, output control actions, and realize communication between neighboring intelligent agents. The control execution module includes a signal control unit and a computing unit, which are used to execute traffic light control commands and run strategy models to make real-time decisions. The training and optimization module includes a simulation environment subunit, an algorithm training subunit, and an online learning subunit, which are used to build simulation scenarios, train reinforcement learning models, and support online parameter updates.

3. The system according to claim 2, characterized in that, The sensing device unit includes radar, camera, and induction coil, used to collect vehicle count, queue length, speed, vehicle type, and estimated arrival time in real time, with a collection accuracy error of no more than 5%.

4. The system according to claim 2, characterized in that, The computing unit adopts a distributed edge computing architecture, with each agent equipped with an independent computing device. The time for a single policy inference is less than 50ms, and it supports a minimum green light time t. min and maximum green light time t max Constraints, safe clear time t clear ≥5s.

Images