Traffic corridor signal coordination control method based on single-agent reinforcement learning

Abstract

The application provides a traffic corridor signal cooperative control method based on single-agent reinforcement learning, and relates to the technical field of traffic management and control, the control method comprising: acquiring real-time traffic state data of a traffic corridor comprising a plurality of signalized intersections; and a current signal control scheme of the traffic corridor; constructing a state vector according to the real-time traffic state data and the current signal control scheme; inputting the state vector into a pre-trained single-agent reinforcement learning model to obtain a corresponding action vector; based on the action vector, synchronously adjusting the phase division of each signalized intersection to generate a new signal control scheme; by using the centralized single-agent architecture to cooperatively control the signal timing of all signalized intersections in the traffic corridor, the system complexity and training instability problems caused by local observation, distributed decision-making and communication coordination between agents in the multi-agent scheme are fundamentally avoided.

Description

Technical Field

[0001] This invention relates to the field of traffic management and control technology, and more specifically, to a traffic corridor signal cooperative control method based on single-agent reinforcement learning. Background Technology

[0002] In the field of urban traffic management, traffic signal control is one of the core means to optimize road network operation efficiency and alleviate traffic congestion. Among them, coordinated signal control for traffic corridors consisting of multiple consecutive intersections is a key issue in improving the capacity of main roads. Traditional methods usually generate timing schemes based on preset traffic flow models (such as timed control and sensor control) or offline optimization algorithms. These schemes rely on simplified theoretical assumptions and static traffic data, making it difficult to adapt to the actual scenario of random and dynamic changes in traffic flow, especially in dealing with sudden congestion and tidal traffic flow.

[0003] In recent years, reinforcement learning has been introduced into traffic signal control research due to its powerful environmental interaction and sequential decision-making capabilities. To address the problem of multi-intersection coordination, most existing studies employ a multi-agent framework, assigning an agent to each intersection or local area and achieving regional coordination through observation sharing and communication collaboration among agents. However, this approach inherently deviates from the real-world architecture of traffic signal control, which is typically centrally managed. Multi-agent systems naturally face challenges such as unstable environments, unstable training, and complex communication mechanism design. Furthermore, their regional divisions are often static and fixed, making it difficult to flexibly respond to spatiotemporal changes in traffic conditions, thus increasing system complexity and practical deployment difficulties.

[0004] Therefore, how to design a multi-intersection signal control method that not only meets the characteristics of centralized traffic control management, but also achieves high efficiency, adaptive collaboration, and is easy to deploy in practice has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The problem solved by this invention is one or more of the aforementioned related technical problems.

[0006] To address the aforementioned problems, this invention provides a traffic corridor signal cooperative control method based on single-agent reinforcement learning.

[0007] In a first aspect, the present invention provides a traffic corridor signal cooperative control method based on single-agent reinforcement learning, comprising:

[0008] Acquire real-time traffic status data of a traffic corridor containing multiple signalized intersections, as well as the current signal control scheme of the traffic corridor;

[0009] A state vector is constructed based on the real-time traffic status data and the current signal control scheme;

[0010] The state vector is input into a pre-trained single-agent reinforcement learning model to obtain the corresponding action vector;

[0011] Based on the action vector, the phase division of each signal intersection is synchronously adjusted to generate a new signal control scheme;

[0012] The phase division process includes: controlling the sum of the green light time for the main direction straight and right turn phases of the signalized intersection and the green light time for the corresponding opposite left turn phase.

[0013] Optionally, the real-time traffic status data includes queue length; obtaining real-time traffic status data for a traffic corridor containing multiple signalized intersections includes:

[0014] Based on the traffic corridor, a plurality of signalized intersections are determined to be connected in sequence, and corresponding road segments are determined based on the plurality of signalized intersections;

[0015] Obtain the queue length for each road segment; wherein the queue length is estimated based on vehicle travel time data collected by the floating car.

[0016] Optionally, the current signal control scheme includes the current phase division of each of the signalized intersections; the step of constructing a state vector based on the real-time traffic state data and the current signal control scheme includes:

[0017] The first sub-vector is obtained by arranging the queue lengths of each road segment in order;

[0018] The current phase divisions of each of the aforementioned signal intersections are arranged in order to obtain the second sub-vector;

[0019] The first sub-vector and the second sub-vector are concatenated to obtain the state vector.

[0020] Optionally, the process of constructing the pre-trained single-agent reinforcement learning model includes:

[0021] Construct a reinforcement learning environment, wherein the observation space of the reinforcement learning environment is defined based on the state vector, and the action space is defined based on the action vector;

[0022] A reward function is set, the output value of which is calculated based on the queue length of each segment in the traffic corridor;

[0023] The proximal policy optimization algorithm is used to iteratively train the single-agent reinforcement learning model in the reinforcement learning environment until the policy converges, thus obtaining the pre-trained single-agent reinforcement learning model.

[0024] Optionally, the reward function includes:

[0025] The reward function is determined based on the sum of the rewards for each of the aforementioned road segments;

[0026] The rewards for each of the aforementioned road sections are as follows:

[0027] ;

[0028] Where r is the reward for the corresponding road segment. , These are the preset thresholds for mild congestion and severe congestion, respectively. This corresponds to the queue length of the aforementioned road segment. The default penalty coefficient is greater than 1.

[0029] Optionally, the reward function includes:

[0030] The reward function is determined based on the sum of the rewards for each of the aforementioned road segments;

[0031] The rewards for each of the aforementioned road sections are as follows:

[0032] ;

[0033] Where r is the reward for the corresponding road segment. This refers to the average travel time for the corresponding road segment. The preset saturation flow rate, The effective green light time for the upstream signalized intersection under the current phase division. This refers to the default green light time for the upstream signalized intersection under the current phase division. , These are the preset thresholds for mild congestion and severe congestion, respectively. This corresponds to the queue length of the aforementioned road segment. The default penalty coefficient is greater than 1.

[0034] Optionally, each element in the action vector corresponds to the adjustment action of the signal intersection.

[0035] Optionally, the traffic corridor signal cooperative control method based on single-agent reinforcement learning further includes: the execution cycle for synchronously adjusting the phase division of the signalized intersection is consistent with the signal cycle of the traffic corridor.

[0036] Secondly, the present invention provides a traffic corridor signal cooperative control system based on single-agent reinforcement learning, comprising:

[0037] The acquisition unit is used to acquire real-time traffic status data of a traffic corridor containing multiple signalized intersections, as well as the current signal control scheme of the traffic corridor.

[0038] The construction unit is used to construct a state vector based on the real-time traffic state data and the current signal control scheme;

[0039] The processing unit is used to input the state vector into a pre-trained single-agent reinforcement learning model to obtain the corresponding action vector;

[0040] The control unit is used to synchronously adjust the phase division of each signalized intersection based on the action vector to generate a new signal control scheme; wherein, the phase division process includes: controlling the sum of the green light time of the main direction straight and right turn phases of the signalized intersection and the green light time of the corresponding opposite left turn phases in the same direction.

[0041] Thirdly, the present invention provides a traffic corridor signal cooperative control device based on single agent reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the traffic corridor signal cooperative control method based on single agent reinforcement learning described in the first aspect.

[0042] The beneficial effects of the traffic corridor signal cooperative control method based on single-agent reinforcement learning of the present invention are:

[0043] By employing a centralized single-agent architecture to collaboratively control the signal timing of all signalized intersections within a traffic corridor, this method fundamentally avoids the system complexity and training instability issues caused by local observation, distributed decision-making, and inter-agent communication coordination problems inherent in multi-agent schemes, significantly simplifying the system design and optimization process. By abstracting the overall state of the intersection group into a unified vector representation and directly outputting collaborative control commands for all intersections, this method achieves centralized, global, and dynamic optimization of corridor traffic flow, enabling more effective responses to real-time traffic demand changes, thereby improving overall traffic efficiency and congestion mitigation capabilities. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a traffic corridor signal cooperative control method based on single-agent reinforcement learning according to an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of an iterative process for coordinated control of traffic corridor signals based on single-agent reinforcement learning, according to an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of a signal phase scheme and phase division according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a traffic corridor signal cooperative control system based on single-agent reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0049] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0050] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0051] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0052] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0053] like Figure 1 As shown in the figure, an embodiment of the present invention provides a traffic corridor signal cooperative control method based on single-agent reinforcement learning, comprising:

[0054] Step S100: Obtain real-time traffic status data of a traffic corridor containing multiple signalized intersections, as well as the current signal control scheme of the traffic corridor.

[0055] Specifically, real-time traffic status data reflecting the current overall road conditions is first collected from the target traffic corridor (i.e., a main road segment consisting of multiple consecutive signalized intersections). This data is a key indicator characterizing the traffic operation status within the corridor. Simultaneously, the current signal control scheme being implemented in this traffic corridor is also acquired, defining the basic operating parameters of the traffic lights at each intersection. By simultaneously acquiring these two types of information, a foundation is laid for subsequently constructing a decision-making basis that comprehensively reflects the "current traffic conditions - existing control status."

[0056] By integrating real-time traffic conditions with existing control schemes, an accurate and consistent input benchmark is provided for subsequent reinforcement learning-based collaborative decision-making. This design enables the entire control system to make decisions based on a complete global situation, avoiding the information lag or bias that may arise from traditional multi-sensor distributed data collection or historical model extrapolation. Simultaneously, the definition of input data in this step is both general and flexible, making it compatible with various common traffic detection data sources and control signal systems. This enhances the applicability and feasibility of the solution under different urban infrastructure conditions, providing reliable data support for subsequent efficient and adaptive corridor-level signal collaborative optimization.

[0057] Step S200: Construct a state vector based on the real-time traffic state data and the current signal control scheme.

[0058] Specifically, the two types of heterogeneous information acquired in step S100—real-time traffic state data reflecting road conditions and the current control scheme defining signal operation—are transformed into a unified, machine-readable numerical representation, namely, a state vector. This process, through specific data organization and fusion methods, integrates all key information within the corridor (such as traffic flow conditions on each road segment and signal status at each intersection) into a structured vector. This vector fully characterizes the "traffic environment in which the system operates under the current control strategy," providing comprehensive and consistent observational input for subsequent single-agent decision-making.

[0059] By employing a state vector construction method, efficient fusion and dimensionality reduction representation of multi-source, multi-dimensional traffic and control information are achieved. This design transforms the complex traffic corridor collaborative control problem into a standard sequence decision problem suitable for reinforcement learning. It not only retains all the key information required for decision-making but also greatly simplifies the observation space, avoiding the difficulties caused by the high dimensionality and heterogeneity of the original data for model training. This directly improves the agent's learning efficiency and policy stability, laying a solid foundation for achieving accurate and rapid centralized collaborative decision-making. Furthermore, this vector construction method is logically clear and computationally lightweight, facilitating efficient and stable online computation and deployment in practical control systems.

[0060] Step S300: Input the state vector into the pre-trained single-agent reinforcement learning model to obtain the corresponding action vector.

[0061] Specifically, the state vector comprehensively representing the current state of the system, constructed in step S200, is input into a pre-trained, centralized single-agent reinforcement learning model. This model can be an Actor-Critic architecture model based on deep neural networks. As the sole decision-making center, this model processes and analyzes the input state information in real-time based on optimization strategies learned through extensive offline training. The model output is an action vector. Each element in this vector is not generated independently, but rather represents a coordinated control command generated by the model for each intersection after comprehensively considering the global state of the entire traffic corridor. This process achieves a mapping from complex environmental states to a set of coordinated control commands.

[0062] By employing a centralized single-agent model for decision-making, the global consistency and cooperative optimality of control commands are fundamentally guaranteed. This design avoids the policy conflicts, training instability, and communication overhead problems caused by distributed decision-making in multi-agent systems. Based on complete corridor state information, the model can instantly calculate a set of coordinated action commands, achieving synchronous and cooperative optimization of signal parameters at multiple intersections. This not only significantly improves decision-making efficiency and response speed, enabling the system to quickly respond to dynamically changing traffic flow, but also ensures the overall coordination of control commands, thereby more effectively guiding and dispersing traffic flow, fundamentally improving the effectiveness and reliability of corridor-level cooperative control. This step is the key intelligent hub for achieving efficient, adaptive global optimization.

[0063] Step S400: Based on the action vector, the phase division of each signalized intersection is synchronously adjusted to generate a new signal control scheme; wherein, the phase division process includes: controlling the sum of the green light time of the main direction straight and right turn phases of the signalized intersection and the green light time of the corresponding opposite left turn phases.

[0064] Specifically, the abstract action vector output in step S300 is transformed into concrete control parameters that can be actually issued to each traffic signal controller. This step parses the action vector, where each element corresponds to an adjustment command for a specific intersection. Based on these commands, the phase division of all intersections is adjusted synchronously and uniformly. Here, "phase division" is a core control parameter, defined as: the sum of the green light time allocated to the straight-ahead and right-turn phases of the main traffic direction (e.g., north-south) at an intersection within a fixed cycle, and the green light time allocated to the opposite left-turn phase in the same direction. By directly and synchronously adjusting this key parameter, a completely new signal timing scheme applicable to the entire traffic corridor can be generated and executed.

[0065] By directly manipulating the key and efficient control variable of "phase division," precise and coordinated regulation of the traffic corridor's operational status is achieved. This design simplifies the complex problem of multi-intersection coordinated control into the synchronous optimization of a unified set of parameters, greatly reducing the complexity of the control logic and the difficulty of implementation. All intersections adjust synchronously based on global decisions, ensuring the spatiotemporal consistency of control commands and effectively guiding traffic flow to pass and dissipate in an orderly manner within the corridor. Compared to traditional methods that adjust multiple parameters such as cycle and phase sequence, this step focuses on the core variable directly related to congestion, making the control response more direct and agile. This allows for faster adaptation to dynamic traffic demands, offering significant advantages in alleviating congestion and improving traffic efficiency, and is easy to deploy and apply quickly in actual signal control systems.

[0066] In this embodiment, by employing a centralized single-agent architecture to collaboratively control the signal timing of all signalized intersections within the traffic corridor, the system complexity and training instability issues caused by local observation, distributed decision-making, and inter-agent communication coordination in multi-agent schemes are fundamentally avoided, significantly simplifying the system design and optimization process. By abstracting the overall state of the intersection group into a unified vector representation and directly outputting collaborative control commands for all intersections, this method achieves centralized and global dynamic optimization of corridor traffic flow, enabling more effective response to real-time changes in traffic demand, thereby improving overall traffic efficiency and congestion mitigation capabilities.

[0067] Optionally, the real-time traffic status data includes queue length; obtaining real-time traffic status data for a traffic corridor containing multiple signalized intersections includes:

[0068] Based on the traffic corridor, a plurality of signalized intersections are determined to be connected in sequence, and corresponding road segments are determined based on the plurality of signalized intersections;

[0069] Obtain the queue length for each road segment; wherein the queue length is estimated based on vehicle travel time data collected by the floating car.

[0070] Optionally, the current signal control scheme includes the current phase division of each of the signalized intersections; the step of constructing a state vector based on the real-time traffic state data and the current signal control scheme includes:

[0071] The first sub-vector is obtained by arranging the queue lengths of each road segment in order;

[0072] The current phase divisions of each of the aforementioned signal intersections are arranged in order to obtain the second sub-vector;

[0073] The first sub-vector and the second sub-vector are concatenated to obtain the state vector.

[0074] In some embodiments, real-time traffic status data for a traffic corridor preferably uses queue length as the core indicator. First, the target corridor is structurally defined: the road section between multiple sequentially connected signalized intersections is defined as an independent road segment (i.e., only the road segment between intersections is counted, excluding side road segments entering and exiting the corridor). For example... Figure 2 As shown, a corridor consisting of four consecutive intersections will generate six road segments, divided as follows: Assume the four intersections are numbered I1, I2, I3, and I4 along the road. Segment 1 (Link 1): The direction of travel from intersection I1 to intersection I2 (e.g., northbound or eastbound). Segment 2 (Link 2): The direction of travel from intersection I2 to intersection I1 (opposite direction, e.g., southbound or westbound). Segment 3 (Link 3): The direction of travel from intersection I2 to intersection I3. Segment 4 (Link 4): The direction of travel from intersection I3 to intersection I2. Segment 5 (Link 5): The direction of travel from intersection I3 to intersection I4. Segment 6 (Link 6): The direction of travel from intersection I4 to intersection I3.

[0075] Subsequently, the real-time queue length for each road segment is obtained. This data acquisition relies on widely available floating car data (such as ride-hailing vehicles, taxis, and private cars with navigation enabled). By receiving the travel time information of these vehicles on specific road segments, combined with known parameters such as road length and free-flow speed, mature traffic flow theory models (e.g., based on queuing theory or speed-density relationship models) can be used to estimate the average or maximum queue length for that road segment. This is an indirect but efficient sensing method.

[0076] Current signal control schemes focus on a core adjustable parameter: the current phase division of each signalized intersection. This parameter directly determines the time allocation of the intersection in different traffic directions.

[0077] When constructing the state vector, an intuitive and efficient vector concatenation method was adopted. Specifically: to generate the first sub-vector (traffic state vector): the queue length values ​​(queue lengths) of all L road segments in the corridor are arranged in the spatial order of the road segments (e.g., from upstream to downstream) to form an L-dimensional vector [q1, q2, ..., qL].

[0078] The current phase division values ​​of all M intersections (M signalized intersections) within the corridor (traffic corridor) are arranged sequentially according to the spatial order of the intersections to form an M-dimensional vector [s1, s2, ..., sM].

[0079] By directly concatenating the two sub-vectors, a one-dimensional composite vector of length (L+M) is formed, namely [q1, q2, ..., qL, s1, s2, ..., sM]. This vector represents the complete observation of the environment by a single agent.

[0080] At the data processing level, the original values ​​before splicing can be normalized, for example, by dividing all queue lengths by the preset maximum possible queue length, and dividing all phase divisions by the signal period length, so that the value of each dimension falls into the [0,1] interval, thereby improving the stability and convergence speed of model training.

[0081] By selecting queue length—an indicator directly related to congestion levels and reliably inverted from floating car data—as the core of state perception, the system's reliance on traditional fixed detectors (such as coils and video) is greatly reduced. This allows the scheme to be quickly adapted to most urban roads in my country that already possess a certain scale of floating car data, demonstrating excellent practical deployability and scalability. Simultaneously, the use of vector concatenation to construct the state transforms the complex spatiotemporal traffic and control states into a structured, low-dimensional numerical vector. This method is extremely concise and lossless of key information, perfectly matching the input requirements of reinforcement learning algorithms, significantly reducing the model's learning complexity, and its generation process has minimal computational overhead. This ensures that the control system can perform high-frequency real-time state updates and decisions, laying a solid foundation for achieving efficient, adaptive real-time collaborative control.

[0082] Optionally, the process of constructing the pre-trained single-agent reinforcement learning model includes:

[0083] Construct a reinforcement learning environment, wherein the observation space of the reinforcement learning environment is defined based on the state vector, and the action space is defined based on the action vector;

[0084] A reward function is set, the output value of which is calculated based on the queue length of each segment in the traffic corridor;

[0085] The proximal policy optimization algorithm is used to iteratively train the single-agent reinforcement learning model in the reinforcement learning environment until the policy converges, thus obtaining the pre-trained single-agent reinforcement learning model.

[0086] Specifically, this process is completed before the system is officially deployed and belongs to the offline training phase. Its core lies in creating a virtual environment and learning framework that can simulate real traffic interactions and guide the agent to learn the optimal strategy.

[0087] Constructing a reinforcement learning environment: This environment is a virtual sandbox for the agent to learn through trial and error. Its design strictly corresponds to the actual control problem.

[0088] Observation space: Constructed based on the state vector defined in the preceding steps. This space is a (L+M) dimensional continuous space, where each dimension corresponds to a road segment queue length or an intersection phase division, and has a reasonable range of values ​​(e.g., queue length from 0 to maximum queue capacity, phase division from minimum green light time to cycle duration).

[0089] Action space: Based on the aforementioned definition of action vectors. This is a discrete action space with a size of 3. M Let represent all possible combinations resulting from M signal intersections independently performing three adjustment actions (decrease, maintain, increase). Each time, the agent must choose a global action from this exponential number of possibilities.

[0090] Environmental dynamics: The environment needs to integrate a simulation model that reflects the basic laws of traffic flow (e.g., using a cellular transmission model, queuing model, or microscopic simulation software interface). When an agent submits an action vector, the environment simulates vehicle operation over a future control cycle (e.g., a few minutes) based on the current traffic state and new signal scheme, calculates the new queue length for road segments, updates the state, and provides a reward.

[0091] Setting a reward function: The reward function acts as a "command stick" guiding the agent's learning direction. The core design principle of this scheme is that the value of the reward function is directly and closely related to the queue length of each segment within the traffic corridor. Its fundamental goal is to encourage the agent to take actions that reduce overall queues and prevent further congestion.

[0092] The function construction tends to impose extremely high negative rewards (penalties) on "severe congestion," negative rewards proportional to queue length on "mild congestion," and zero or tiny positive rewards on "smooth flow." This design allows the agent to naturally learn to avoid causing congestion and actively alleviate existing queues during exploration.

[0093] The specific calculation of rewards can take the form of piecewise linear functions, exponential functions, etc. For example, in addition to directly using the negative value of the queue length, the change in queue length (the current queue minus the previous queue) can also be used as part of the reward to encourage continuous improvement. A smoothing term can also be introduced to impose a small penalty on drastic changes in signal timing schemes between adjacent cycles, thereby improving the smoothness of the control scheme and driver comfort.

[0094] Iterative training using the PPO algorithm: The advanced policy gradient algorithm, Proximal Policy Optimization (PPO), is used to train a single agent.

[0095] Training Process: The agent (a deep neural network) interacts with the environment in numerous rounds. In each iteration step, the agent collects a batch of interaction data (state, action, reward, new state) based on the current policy. Then, the PPO algorithm uses this data and its unique "truncated" objective function to safely and efficiently update the neural network parameters, enabling the new policy to improve in the direction of obtaining higher expected cumulative rewards, while avoiding deviating too far from the old policy and causing training failure.

[0096] Convergence Determination: Training continues until the policy "converges." This is typically manifested as the agent's average cumulative reward (or a corresponding traffic metric, such as average queue length) stabilizing near a high value across multiple evaluation rounds, without significant further improvement. At this point, the neural network parameters are fixed, forming a pre-trained model ready for subsequent online deployment.

[0097] In other embodiments, such as Figure 2 As shown, in the traffic signal control (TSC) process of the single-agent reinforcement learning model, a closed-loop interaction continuously occurs between the agent and the traffic signal control environment. Specifically, at each decision time t, the agent first observes and receives the current state S from the environment. t This state fully encapsulates the congestion level of each road segment within the traffic corridor and the current signal timing information at each intersection. Based on S t The agent outputs a cooperative control action 'a' through its internal policy network. t And issue the order to the environment for execution. The environment executes the order based on action a. t Update signal timings and simulate traffic flow evolution to generate a new environmental state S. t Simultaneously, a scalar reward signal r is calculated. t+1 The reward is fed back to the agent and directly evaluates action a. tThe immediate effect on alleviating corridor congestion. By continuously collecting such interaction experiences and using policy optimization algorithms to constantly update its decision-making strategies, the agent ultimately learns the optimal cooperative control strategy that minimizes overall congestion and improves traffic efficiency.

[0098] The current signal control scheme employs a concise and efficient parameterized representation method. The signal phase scheme for each signalized intersection in the traffic corridor is uniformly abstracted into a scalar value s called "phase division". m (where m = 1, 2, ..., M), and M is the total number of signalized intersections in the corridor). The phase divisions of all intersections together form an M-dimensional vector, which serves as a complete description of the current control scheme.

[0099] Within this framework, each intersection employs a typical four-phase control structure (e.g., north-south straight / right turn, north-south left turn, east-west straight / right turn, east-west left turn), such as... Figure 3 As shown. To focus on the core variables of congestion control, the model assumes that parameters such as signal cycle duration, yellow and all-red light times, phase sequence, and phase differences between intersections are pre-set and fixed. Phase allocation here specifically refers to the straight / right-turn phase assigned to a specific mainstream direction (e.g., north-south) within a cycle. Figure 3 (Middle phase 1) and its opposite left-turning phase (e.g.) Figure 3 Phase 1 in the middle direction, its corresponding left-turn phase is... Figure 3 The sum of the green light times for phase 2 in the signal timing scheme. Since other time parameters are fixed (such as the signal cycle of the traffic corridor, the phase scheme, the yellow and red light times, and the phase difference between intersections are all preset fixed values), the green light duration for each phase of the intersection can be completely determined by this single variable, thus uniquely determining its signal timing scheme.

[0100] The advantage of this representation method lies in its direct connection to traffic flow control capabilities. For example, as... Figure 3 As shown, when congestion is detected on the road segment from intersection m to intersection m+1, the following strategies can be implemented collaboratively: appropriately increase the phase division time of the upstream intersection m to suppress vehicles from entering the congested segment; simultaneously, reduce the phase division time of the downstream intersection m+1 to accelerate vehicle departure. By coordinating and adjusting the phase division values ​​of each intersection within the corridor, proactive traffic flow management and coordinated congestion mitigation can be achieved.

[0101] This model construction scheme transforms the complex traffic cooperative control problem into a highly efficient learning task through a meticulously designed closed-loop training framework. Its beneficial effects are significant: First, by defining a state and action space that closely matches the essence of the problem, the accuracy of the learning objective is ensured. Second, by designing a reward function centered on queue length, a clear and direct learning objective is provided to the agent, guiding it to focus on the core task of alleviating congestion and avoiding the conflicts and confusion that may arise from multi-objective optimization. Finally, the PPO algorithm is used for training, effectively overcoming common problems in reinforcement learning training such as poor stability and low sample efficiency, ensuring that the training process reliably converges to a high-performance cooperative control strategy. This entire offline training mechanism is the key guarantee for ultimately obtaining an "intelligent brain" capable of understanding complex traffic dynamics and making globally optimal cooperative decisions, laying a solid foundation for the actual effectiveness of the scheme.

[0102] Optionally, the reward function includes:

[0103] The reward function is determined based on the sum of the rewards for each of the aforementioned road segments;

[0104] The rewards for each of the aforementioned road sections are as follows:

[0105] (1);

[0106] Where r is the reward for the corresponding road segment. , These are the preset thresholds for mild congestion and severe congestion, respectively. This corresponds to the queue length of the aforementioned road segment. The default penalty coefficient is greater than 1.

[0107] Optionally, the reward function includes:

[0108] The reward function is determined based on the sum of the rewards for each of the aforementioned road segments;

[0109] The rewards for each of the aforementioned road sections are as follows:

[0110] (2);

[0111] Where r is the reward for the corresponding road segment. This refers to the average travel time for the corresponding road segment. The preset saturation flow rate, The effective green light time for the upstream signalized intersection under the current phase division. This refers to the default green light time for the upstream signalized intersection under the current phase division. , These are the preset thresholds for mild congestion and severe congestion, respectively. This corresponds to the queue length of the aforementioned road segment. The default penalty coefficient is greater than 1.

[0112] Specifically, the reward function is designed with two types of rewards, corresponding to two objectives. The first is to simply alleviate the congestion of each road segment within the control area (based on the reward function obtained from formula (1)); the second is to minimize the total travel time as much as possible while alleviating the congestion of each road segment (based on the reward function obtained from formula (2)).

[0113] The two reward functions, designed with different optimization focuses, together constitute the core advantage of this invention in achieving efficient and robust cooperative control:

[0114] The beneficial effects of the first reward function (calculated directly based on queue length):

[0115] This design directly defines the reward function as a piecewise function of the queue length for each road segment. Its core advantages lie in its high focus and computational efficiency. Since the reward signal is directly and linearly correlated with the most intuitive congestion indicator—queue length (in cases of mild congestion)—and imposes an exponential penalty for severe congestion, the optimization objective of the reinforcement learning agent becomes extremely clear and singular: avoid severe congestion at all costs and minimize the total number of vehicles queuing on each road segment. This design greatly simplifies the objective complexity of the agent's policy learning, enabling it to quickly and stably learn basic strategies for alleviating congestion. Simultaneously, the function calculation relies only on the estimated queue length, resulting in minimal computational overhead, making it suitable for real-time control systems requiring high-frequency decision-making and ensuring the algorithm's online operational efficiency.

[0116] The second reward function (based on travel time and saturation flow estimation): This design further incorporates the average travel time of the road segment on top of queue length. and based on saturation flow The estimated traffic flow is calculated by comparing the traffic volume to the green light time. Its core benefit lies in elevating the optimization from "alleviating overt congestion" to a higher level of "minimizing total system cost." This is achieved through... Multiplying by the estimated flow rate gives an approximate representation of the total delay time for all vehicles on that road segment (number of vehicles × average delay per vehicle). This allows the agent to spontaneously optimize total vehicle travel time while alleviating congestion, achieving a deeper level of optimization from "visible queues" to "actual travel efficiency." Furthermore, saturation flow is used in severely congested sections. As a penalty base, it is a reasonable approximation of the system's throughput limit under extremely high traffic conditions, making the reward function more robust and physically realistic under extreme congestion. This design guides the agent to learn a better and more economical global strategy that balances queue length and throughput efficiency.

[0117] It is important to note that the above formula does not use actual traffic flow. Instead, for mild congestion, an adjustment value based on saturation flow is used. This is used to represent the actual traffic flow, primarily to reduce the impact of traffic fluctuations under the same phase division. Furthermore, during severe congestion, due to... It is a relatively large penalty value, in order to simplify its use This is used to replace the actual traffic flow, meaning that the actual effective green light time was not used for correction.

[0118] Two reward functions offer optimization paths with different granularities and objectives. The first, with its simplicity, robustness, and strong guidance, is particularly suitable for scenarios where rapid congestion suppression is the primary goal or in the early stages of training. The second, with its economy and systematic approach, is suitable for further refining overall network efficiency optimization during periods of basic smooth traffic or mild congestion. This invention provides both designs simultaneously, giving the technical solution greater adaptability and scalability. Users can choose the most suitable reward mechanism based on the specific stage objectives of urban traffic management (emergency mitigation or long-term optimization) and data conditions. Thus, both can achieve more stable and global collaborative control effects than traditional multi-agent methods within a unified single-agent framework.

[0119] Optionally, each element in the action vector corresponds to the adjustment action of the signal intersection.

[0120] Specifically, the action vector is a direct reflection of the model's decision-making results. Its design aims to transform the agent's learning strategy into precise and coordinated control of signal timing at all intersections within the traffic corridor.

[0121] The structure and mapping relationship of the action vector: The action vector a is an M-dimensional vector, i.e., a = [a1, a2, ..., a...]. M ] ), where M is the total number of signalized intersections in the traffic corridor. The core design principle of this vector is one-to-one correspondence: the m-th element a in the vector m This structured output format, specifically and uniquely corresponding to the adjustment command for the m-th signal intersection, enables a centralized agent to simultaneously issue differentiated control commands to multiple intersections located in different positions, making it a key vehicle for achieving global collaborative control.

[0122] Adjust the semantic definition of the action: each element a m The value of is derived from a discrete, predefined set of actions. This scheme employs an intuitive and robust ternary action set:

[0123] Action 0 (Decrease): Decrease the current phase division of the corresponding intersection m by a fixed step. .

[0124] Action 1 (Hold): Do not adjust the phase division of intersection m.

[0125] Action 2 (Addition): Increase the current phase division of the corresponding intersection m by a fixed step size. .

[0126] in, It is a preset parameter (e.g., 5 seconds) that represents the smallest unit of adjustment, balancing the fineness of control with the stability of the system.

[0127] Example of execution process:

[0128] Suppose a corridor consists of 3 intersections (M=3). The current state vector has been constructed and input into the model. After calculation, the model outputs an action vector a = [2, 0, 1]. The interpretation and execution of this vector are as follows:

[0129] For intersection 1, Perform the "increase" action: increase its phase division. Second.

[0130] For intersection 2, Perform the "reduce" action: reduce its phase division. Second.

[0131] For intersection 3, The "hold" action is performed: its phase division remains unchanged within this period.

[0132] After decoding this vector, new phase division values ​​are synchronously sent to the signal controllers at the three intersections. All adjustments take effect at the beginning of a control cycle to ensure coordinated control.

[0133] Other options include: layered motion space: to accommodate finer control, a larger set of motions can be defined, such as five-element motions: {significant decrease, slight decrease, hold, slight increase, significant increase}, each corresponding to a different adjustment step size.

[0134] Relative vs. Absolute Actions: The above design represents a relative adjustment. Another option is to define absolute actions, where the action directly represents the interval or specific value of the target phase division, and the agent directly selects a final timing scheme for each intersection.

[0135] Continuous action space: When using an algorithm that supports continuous actions, the action vector a m It can itself be a continuous value (e.g., between [-1, 1]), and the adjustment amount for phase division can be directly obtained by multiplying it by a scaling factor (e.g., ...). This enables stepless adjustment.

[0136] Action vectors are sets of instructions that distribute the centralized decisions of an agent to each intersection. By designing each element to correspond to an adjustment action at an intersection, this scheme transforms the complex spatial collaborative optimization problem into a well-structured and easily executable control instruction generation problem. This is one of the core mechanisms for achieving efficient and precise collaborative control of multiple intersections by a single agent.

[0137] Optionally, the traffic corridor signal cooperative control method based on single-agent reinforcement learning further includes: the execution cycle for synchronously adjusting the phase division of the signalized intersection is consistent with the signal cycle of the traffic corridor.

[0138] Specifically, the triggering and execution time of control decisions must be strictly synchronized with the existing traffic signal operation cycle.

[0139] Under this scheme, phase division adjustments do not occur randomly at any given time, but are constrained within the inherent operational framework of the traffic signal system. Specifically:

[0140] Decision-making and execution cycles are linked: the basic time unit of the decision-making (i.e., generating action vectors) of the single-agent model and the execution of the phase division adjustment instructions based thereon is exactly the same as the preset signal cycle duration of the traffic corridor.

[0141] Operation timing: Within a typical signal cycle (e.g., duration C seconds), a fixed timing sequence of "evaluation-decision-execution" is followed:

[0142] Within a brief time window before the end of a cycle (or using data from within the cycle), the system acquires the average or typical traffic conditions (such as queue length) for that cycle and combines this with the current phase division to form a state vector S. t .

[0143] S t The input is fed into a trained single-agent model, which calculates the action vector a. t .

[0144] At the start of the next signal cycle, the control system will a t The data is decoded into specific phase division adjustments and simultaneously sent to all signal intersections within the corridor. The new phase division will remain in effect throughout the entire new cycle (cycle t+1).

[0145] This means that the phase division value remains constant within each signal cycle, and adjustments only occur at the instant of cycle switching. The agent learns to make cycle-level collaborative control decisions based on cycle-level "snapshot" states.

[0146] Example: Assume the signal period is fixed at 90 seconds. The control system's clock is synchronized with the signal controller. Around the 85th second of each Nth cycle, the system collects data and triggers an agent decision. Starting from second 0 of the N+1th cycle, all intersections operate according to the newly adjusted phase division, lasting for 90 seconds.

[0147] The decision trigger point can be selected at a fixed moment at the beginning, end, or middle of the cycle, as long as the decision result is applied to the next complete cycle. For example, to allow time for computation and communication, the trigger can be set 5% before the end of the cycle.

[0148] This solution is not only applicable to fixed-cycle systems, but also adaptable to "semi-fixed-cycle" systems (i.e., the cycle duration can be switched between several preset values). In this case, the execution cycle remains consistent with the adjusted new cycle duration.

[0149] For more complex scenarios, the decision interval of the agent can be defined as multiple signal cycles (e.g., making a decision every two cycles), but execution still takes effect at the beginning of the cycle following the decision. This reduces the decision frequency, potentially enhancing the stability of the policy, and is suitable for scenarios where traffic conditions change slowly.

[0150] By aligning the execution cycle of phase division adjustments with the signal cycle, this design brings crucial system stability, engineering feasibility, and traffic flow predictability. It ensures that the signal timing scheme is fixed and consistent within each cycle, avoiding driver confusion, safety hazards, and traffic flow disruption caused by frequent green light switching within a cycle. Simultaneously, this synchronization mechanism allows the intelligent control layer of this invention to seamlessly integrate with the standard operating logic of existing traffic signal controllers without modifying the underlying hardware timing mechanism, significantly reducing the complexity of system integration and deployment. Furthermore, the periodic-level stable control provides the reinforcement learning agent with a clear, discrete timescale for evaluating and optimizing long-term rewards (cumulative rewards), facilitating the training of more robust policies and ultimately achieving efficient collaborative control that is smoothly compatible with existing traffic infrastructure.

[0151] like Figure 4 As shown in the figure, an embodiment of the present invention provides a traffic corridor signal cooperative control system based on single-agent reinforcement learning, comprising:

[0152] The acquisition unit is used to acquire real-time traffic status data of a traffic corridor containing multiple signalized intersections, as well as the current signal control scheme of the traffic corridor.

[0153] The construction unit is used to construct a state vector based on the real-time traffic state data and the current signal control scheme;

[0154] The processing unit is used to input the state vector into a pre-trained single-agent reinforcement learning model to obtain the corresponding action vector;

[0155] The control unit is used to synchronously adjust the phase division of each signalized intersection based on the action vector to generate a new signal control scheme; wherein, the phase division process includes: controlling the sum of the green light time of the main direction straight and right turn phases of the signalized intersection and the green light time of the corresponding opposite left turn phases in the same direction.

[0156] This invention provides a traffic corridor signal cooperative control device based on single-agent reinforcement learning, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the traffic corridor signal cooperative control method based on single-agent reinforcement learning as described above when the computer program is executed.

[0157] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the traffic corridor signal cooperative control method based on single-agent reinforcement learning as described above.

[0158] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A traffic corridor signal cooperative control method based on single-agent reinforcement learning, characterized in that, include: Acquire real-time traffic status data of a traffic corridor containing multiple signalized intersections, as well as the current signal control scheme of the traffic corridor; The real-time traffic status data includes queue length; The acquisition of real-time traffic status data for a traffic corridor containing multiple signalized intersections includes: Based on the traffic corridor, a plurality of signalized intersections are determined to be connected in sequence, and corresponding road segments are determined based on the plurality of signalized intersections; Obtain the queue length for each of the road segments; wherein the queue length is estimated based on vehicle travel time data collected by the floating car; A state vector is constructed based on the real-time traffic status data and the current signal control scheme; The state vector is input into a pre-trained single-agent reinforcement learning model to obtain the corresponding action vector; Based on the action vector, the phase division of each signal intersection is synchronously adjusted to generate a new signal control scheme; The phase division process includes: the sum of the green light time for the main direction straight and right turn phases of the signalized intersection and the green light time for the corresponding opposite direction left turn phase; The current signal control scheme includes the current phase division of each signalized intersection; the construction of a state vector based on the real-time traffic state data and the current signal control scheme includes: The first sub-vector is obtained by arranging the queue lengths of each road segment in order; The current phase divisions of each of the aforementioned signal intersections are arranged in order to obtain the second sub-vector; The first sub-vector and the second sub-vector are concatenated to obtain the state vector; The traffic corridor signal cooperative control method based on single-agent reinforcement learning further includes: an execution cycle for synchronously adjusting the phase division of the signalized intersection, which is consistent with the signal cycle of the traffic corridor; The process of constructing the pre-trained single-agent reinforcement learning model includes: Construct a reinforcement learning environment, wherein the observation space of the reinforcement learning environment is defined based on the state vector, and the action space is defined based on the action vector; A reward function is set, the output value of which is calculated based on the queue length of each segment in the traffic corridor; The proximal policy optimization algorithm is used to iteratively train the single agent reinforcement learning model in the reinforcement learning environment until the policy converges, thus obtaining the pre-trained single agent reinforcement learning model. The reward function includes: The reward function is determined based on the sum of the rewards for each of the aforementioned road segments; The rewards for each of the aforementioned road sections are as follows: ； Where r is the reward for the corresponding road segment. This refers to the average travel time for the corresponding road segment. The preset saturation flow rate, The effective green light time for the upstream signalized intersection under the current phase division. This refers to the default green light time for the upstream signalized intersection under the current phase division. , These are the preset thresholds for mild congestion and severe congestion, respectively. This corresponds to the queue length of the aforementioned road segment. The default penalty coefficient is greater than 1.

2. The traffic corridor signal cooperative control method based on single-agent reinforcement learning according to claim 1, characterized in that, Each element in the action vector corresponds to the adjustment action of the signal intersection.

3. A traffic corridor signal cooperative control system based on single-agent reinforcement learning, characterized in that, The traffic corridor signal cooperative control method based on single-agent reinforcement learning as described in any one of claims 1 to 2, wherein the control system comprises: The acquisition unit is used to acquire real-time traffic status data of a traffic corridor containing multiple signalized intersections; and the current signal control scheme of the traffic corridor; the real-time traffic status data includes queue length; and the current signal control scheme includes the current phase division of each signalized intersection. A construction unit is configured to construct a state vector based on the real-time traffic state data and the current signal control scheme; the construction of the state vector based on the real-time traffic state data and the current signal control scheme includes: The first sub-vector is obtained by arranging the queue lengths of each road segment in order; The current phase divisions of each of the aforementioned signal intersections are arranged in order to obtain the second sub-vector; The first sub-vector and the second sub-vector are concatenated to obtain the state vector; The processing unit is used to input the state vector into a pre-trained single-agent reinforcement learning model to obtain the corresponding action vector; The control unit is used to synchronously adjust the phase division of each signalized intersection based on the action vector to generate a new signal control scheme; wherein, the phase division process includes: controlling the sum of the green light time of the main direction straight and right turn phases of the signalized intersection and the green light time of the corresponding opposite left turn phase; the traffic corridor signal cooperative control method based on single agent reinforcement learning also includes: the execution cycle of synchronously adjusting the phase division of the signalized intersection is consistent with the signal cycle of the traffic corridor.

4. A traffic corridor signal cooperative control device based on single-agent reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the traffic corridor signal cooperative control method based on single-agent reinforcement learning as described in any one of claims 1 to 2.

Images