HierachicalLight: urban traffic state modeling method based on hierarchical reinforcement learning and traffic signal control method

Abstract

The invention provides an urban traffic state modeling method based on hierarchical reinforcement learning and a traffic signal control method based on hierarchical reinforcement learning. The road intersection traffic signal control method based on the hierarchical reinforcement learning HRL comprises the steps that the first traffic state of a road intersection is acquired, and the first traffic state comprises a plurality of elements; generating a first plurality of weights according to the first traffic state, each weight corresponding to each of a plurality of elements of the first traffic state; generating a first quantifiable traffic state (QTS) according to the first plurality of weights and the plurality of elements; generating a traffic signal phase for the road intersection according to the first quantifiable traffic state QTS; wherein the state agent comprising the deep Q network is used for generating a first plurality of weights and a first quantifiable traffic state QTS according to the first traffic state; comprising a reinforcement learning module is used for generating a first traffic signal phase for the road intersection according to the first quantifiable traffic state QTS; the state agent and the traffic agent are respectively a relatively high level agent and a relatively low level agent in hierarchical reinforcement learning (HRL).

Description

Technical Field

[0001] The present application relates to machine learning technology, and in particular, to an urban traffic state modeling method, a traffic signal control method and an information processing device thereof based on hierarchical reinforcement learning. Background Art

[0002] With the continuous increase in traffic demand, it is crucial to improve the efficiency of existing transportation infrastructure to reduce urban congestion and exhaust emissions. Among them, real-time adaptive traffic signals are one of the main tools to improve the efficiency of my country's urban transportation network. However, due to the complexity and randomness of the traffic system, traffic signal control (TSC) remains a challenging task.

[0003] Traditional traffic signal control (TSC) methods, such as fixed-time signal control introduced by Webster in 1958 [https: / / trid.trb.org / view / 113579], have shown limitations in adapting to dynamic traffic conditions. Therefore, researchers have proposed various adaptive traffic signal control (TSC) methods, such as SCOOT [https: / / trid.trb.org / view / 179439], SCATS [https: / / trid.trb.org / view / 488852], and Self-Organizing Traffic Lights (SOTL) [https: / / arxiv.org / abs / nlin / 0411066]. TSC methods based on maximum pressure (MP) [https: / / www.sciencedirect.com / science / article / abs / pii / S0968090X13001782] and effective pressure (EP) [https: / / arxiv.org / abs / 2112.02336] target the length of vehicle queues and the pressure of queues, respectively. In recent years, methods based on reinforcement learning (RL) have become a promising trend in TSC applications. In reinforcement learning, the traffic signal optimization problem is formulated as a Markov decision process (MDP), where S represents the set of road states, A represents the set of traffic signal control actions, and R represents the reward. The traffic signal control agent decides the control action a∈A of the traffic signal to be taken in a given state s∈S to maximize the expected reward. The traffic signal control agent is, for example, a traffic light control device or a control system. While FRAP [https: / / dl.acm.org / doi / 10.1145 / 3357384.3357900] adopts the phase competition principle, Preslight [https: / / dl.acm.org / doi / 10.1145 / 3292500.3330949] maximizes throughput by utilizing MP-based state representation and reward function. Colight [https: / / dl.acm.org / doi / 10.1145 / 3357384.3357902] utilizes graph attention neural network, and MetaLight improves learning efficiency through meta-reinforcement learning technology. Optimizing reward function and state representation is the key to RL model in TSC application.The Advance d-MP[https: / / proceedings.mlr.press / v162 / zhang22ah.html] algorithm considers the competition between moving vehicles and queuing vehicles, and Dynamiclight[https: / / arxiv.org / abs / 2211.01025] utilizes the maximum queue length for phase selection and implements deep Q-learning to determine the phase duration.

[0004] Complex systems are ubiquitous in nature and society and are characterized by nonlinear and dynamic interactions between components, leading to emergent phenomena. Understanding complex systems presents a fundamental challenge that spans multiple disciplines, including physics, biology, economics, and social sciences. Wolfram has introduced a new paradigm for understanding complex systems through cellular automata (CA) [https: / / www.wolfram-media.com / products / nks / ], contributing a large number of discoveries and insights in different scientific fields. Mitchell provides an overview of complexity science [https: / / www.researchgate.net / publication / 258447899_Complexity_A_Guided_Tour], including various topics such as evolution, computation, and artificial intelligence. TSC is a complex system affected by multiple factors such as traffic demand, signal timing, and driver behavior. To address the complexity of TSC, researchers have proposed various modeling and optimization techniques based on complexity science, such as network theory, cellular automata, and swarm intelligence.

[0005] Hierarchical reinforcement learning (HRL) aims to simplify complex problems by organizing them into hierarchical structures. It has a wide range of applications, such as teaching robots to learn complex tasks such as object manipulation and navigation through manageable subtasks [https: / / proceedings.neur ips.cc / paper / 2016 / file / f442d33fa06832082290ad8544a8da27-Paper.pdf]. HRL also improves autonomous decision-making in vehicles by hierarchically organizing tasks such as lane changing, obstacle avoidance, and path planning. In video games, HRL is used to develop intelligent agents that master games by learning subtasks such as navigation, resource management, and combat strategies. It also enhances natural language processing by building hierarchical structures in language tasks such as sentence parsing and document summarization. Summary of the invention

[0006] This paper introduces a "Rules-Data-ML" paradigm for modeling complex systems, and based on this paradigm and HRL, proposes methods to accurately model the state of road intersections for RL-based TSC. The methods provided by this paper reveal the special rules and states that govern the operation of urban traffic, thereby improving the performance of TSC methods, especially the performance of RL-based state-of-the-art (SOTA) results. In addition, the examples of this paper show the potential for generalization in other RL-based tasks.

[0007] The present invention introduces a new "Rules-Data-ML" paradigm to the field of traffic signal control, solving the problem of how to model the complex and dynamic state of road intersections in real-world traffic systems. The method of the present invention combines machine learning, especially classical reinforcement learning methods, to accurately model the state of traffic intersections. This method not only improves the performance of existing traffic signal control (TSC) methods, but also enhances the most advanced (SOTA) reinforcement learning models. The advantage of the present application lies in the quantifiable state model of road intersections. This has been proven to be effective and robust through experimental data in the embodiments of the present invention. The model of the present invention has the potential to be applied to other fields beyond traffic control systems, providing a general technical solution for complex task processing involving reinforcement learning. The embodiments of the present application, coupled with promising numerical results, are expected to optimize traffic signal control systems and a wider range of machine learning applications. It is expected that the examples, codes and data proposed by the present invention can be used for further exploration in these and other fields.

[0008] The present invention further develops a hierarchical reinforcement learning (HRL) framework based on the quantizable traffic state (QTS) representation of the "Rules-Data-ML" paradigm. Finally, the present invention also provides a traffic signal control method that utilizes QTS to achieve performance improvement.

[0009] According to a first aspect of the present application, a first road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, comprising: obtaining a first traffic state of a road intersection, wherein the first traffic state includes a plurality of elements; generating a first plurality of weights according to the first traffic state, each weight corresponding to each of the plurality of elements of the first traffic state; generating a first quantifiable traffic state QTS according to the first plurality of weights and the plurality of elements;

[0010] A traffic signal phase for the road intersection is generated according to the first quantifiable traffic state QTS; a state agent including a deep Q network is used to generate a first plurality of weights and a first quantifiable traffic state QTS according to the first traffic state; a traffic agent including a reinforcement learning module is used to generate a first traffic signal phase for the road intersection according to the first quantifiable traffic state QTS; the state agent and the traffic agent are respectively a relatively high-level agent and a relatively low-level agent in hierarchical reinforcement learning HRL; wherein, to train the deep Q network, the following steps S1 to S7 are performed; step S1, obtaining a plurality of elements representing a second traffic state from a traffic environment as a state S of the deep Q network; generating a second traffic state according to the state S based on maximizing the Q value of the deep Q network Multiple weights are used as the behavior A of the deep Q network: step S2, generating a second quantifiable traffic state QTS according to multiple elements of the second traffic state and the second multiple weights and providing it to the traffic agent; step S3, the traffic agent implements traffic signal control according to the second quantifiable traffic state QTS; step S4, collecting the average travel time ATT of the traffic environment under the control of the traffic agent, and calculating the reward R for training the deep Q network according to the average travel time ATT; step S5, collecting multiple elements representing the new value of the second traffic state in the traffic environment under the control of the traffic agent as the updated state S' of the deep Q network; step S6, using the state S, the behavior A, the reward R and the state S' to generate training data; and step S7, training the deep Q network from the training data.

[0011] According to the first aspect of the present application, a second road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein in step S3, the traffic agent generates traffic signal phases in multiple traffic environments using multiple traffic signal control methods, wherein the multiple traffic signal control methods correspond one-to-one to the multiple traffic environments, and each of the multiple traffic signal control methods generates a traffic signal phase according to the second quantifiable traffic state QTS; and in step S4, the rewards obtained according to the average travel time ATT in each of the multiple traffic environments are accumulated as the reward R for training the deep Q network.

[0012] According to the first aspect of the present application, a third road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, which also includes: re-training the trained deep Q network; wherein, to re-train the deep Q network, the steps S1 to S7 are executed again; when the step S3 is executed again, the traffic agent generates a traffic signal phase in the corresponding traffic environment using the first traffic signal control method, and the first traffic signal control method generates a traffic signal phase according to the second quantifiable traffic state QTS generated by the state agent including the trained deep Q network; and when the step S4 is executed again, the average travel time (ATT) in the traffic environment corresponding to the first traffic signal control method is collected, and the reward R used to train the deep Q network is calculated.

[0013] According to the first aspect of the present application, a fourth road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein the multiple elements of the first traffic state include effective pressure EP, passing demand PD and local range PR.

[0014] According to the first aspect of the present application, a fifth road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein, according to QTS=f(αEP,βPD)γ PR Calculate the first quantifiable traffic state QTS and the second quantifiable traffic state QTS; wherein α represents the weight corresponding to the effective pressure EP, β represents the weight corresponding to the passing demand PD, and γ represents the weight corresponding to the local range PR; f represents an embedding function for embedding the road intersection state expressed by αEP, βPD, and γPR; in step S4, according to Calculate the reward R, where Indicates the baseline average travel time for comparison.

[0015] According to the first aspect of the present application, a sixth road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein the multiple elements of the first traffic state include effective pressure EP, passing demand PD and local range PR; according to QTS=f(αEP,βPD) γPR Calculate the first quantifiable traffic state QTS and the second quantifiable traffic state QTS; wherein α represents the weight corresponding to the effective pressure EP, β represents the weight corresponding to the passing demand PD, and γ represents the weight corresponding to the local range PR; f represents an embedding function for embedding the road intersection state expressed by αEP, βPD, and γPR; in step S4, according to R=∑Rj, and Calculate the reward R, where j represents the jth traffic control method among the multiple traffic signal control methods, and n represents the number of types of the multiple traffic signal control methods. Represents the benchmark average travel time of the jth traffic control method used for comparison.

[0016] According to the first aspect of the present application, a seventh road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein the multiple traffic signal control methods include a traffic signal control method based on reinforcement learning.

[0017] According to the first aspect of the present application, an eighth road intersection traffic signal control method based on hierarchical reinforcement learning (HRL) is provided, wherein steps S3 to S6 are repeated multiple times to obtain multiple quadruples (S, A, R, S') to generate training data; and steps S1 to S7 are repeated multiple times to train the deep Q network multiple times.

[0018] According to the second aspect of the present application, an information processing device according to the second aspect of the present application is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the road intersection traffic signal control method according to the first aspect of the present application when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present application, together with the preferred mode of use and further objects and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A block diagram of an HRL framework according to an embodiment of the present application is shown.

[0021] Figure 2 A block diagram of an HRL framework according to another embodiment of the present application is shown.

[0022] Figure 3 A flowchart of a training state agent according to an embodiment of the present application is shown.

[0023] Figure 4 A flowchart of training a state agent according to another embodiment of the present application is shown.

[0024] Figure 5 The comparison shows the performance impact of using the MP method as the baseline method.

[0025] Figure 6 The comparison shows the performance impact of using the MPLight method as the baseline method.

[0026] Figure 7 The comparison shows the performance impact of using the CoLight method as the baseline method. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0028] The present invention proposes the "Rules-Data-ML" paradigm, which integrates the knowledge of known "rules" and machine learning techniques to model complex systems. The method proposed in the present invention focuses on improving the accuracy of modeling complex systems. The "Rules-Data-ML" paradigm includes three main components, namely rules, data, and machine learning.

[0029] The rule "Rules" component contains multiple existing rules (R1, R2, ..., Rn), which describe the known behaviors of the complex system. The i-th rule is represented by Ri, where i represents the index of the rule in question. As an example, the complex system here is a road traffic system. In addition to the control system that implements traffic signal control (TSC), the road traffic system also includes various components such as roads and vehicles. Still as an example, in the road traffic system, the known rules are description methods of road intersection states such as effective pressure (EP), passing demand (Passing Demand, PD), and partial range (PartialRange, PR). It can be understood that in the present application, the rule Ri can also represent other ways of describing the state of road intersections, or ways of describing the state of road intersections that will appear in the future. These methods can be integrated into the machine learning framework by the "Rules-Data-ML" paradigm provided in the embodiments of the present application to optimize the performance of complex systems.

[0030] The “data” component represented by Dr embodies the experimental observation and data collection in the complex system based on the above rules. Here, Dr represents the data generated by the rule R in the complex system.

[0031] The “ML” component includes a machine learning model M trained using the above “rules” and “data”. The machine learning model M is used to automatically discover and quantify key information and weight representations in the “rules”.

[0032] Mathematically, the relationship between model, data and rules can be expressed as: M(D r )→(αR 1 , βR 2, ..., γR n ), where (α, β, γ) represents the quantifiable weight information of different rules. It can be understood that the number of weights is not limited to 3, but a corresponding trainable weight is provided for each rule Ri. Therefore, the ML model M(Dr) is trained based on the data Dr, and the weights associated with the "rule" Ri can be automatically learned. And the ML model M(Dr) is a model for learning weights (α, β, γ) based on the data Dr.

[0033] According to an embodiment of the present invention, a quantifiable traffic state representation QTS is also proposed. Recent studies have emphasized that effective pressure (EP) and traffic demand (PD) are effective ways to characterize urban traffic states. In an embodiment of the present invention, EP and PD calculated within a local range (PR) are used, wherein the local range (PR) is located at the center of a road intersection. The selection of the local range (PR) has an impact on the performance of TSC, which is another "rule" integrated by an embodiment of the present invention. Different local ranges (PR) will affect the calculation of EP and PD, and also affect the control effect of TSC. For example, PR is approximately 150 meters, representing a range of 150 meters from the center of the road intersection, and data such as the length, number, and travel time of vehicle queues are collected within this range to calculate EP and PD. However, for complex systems, it is difficult to intuitively describe how rules such as PR affect system performance, and the M(D) of the present invention can be used to describe how rules such as PR affect system performance. r ) models the comprehensive and quantitative expression of multiple rules by learning rule-based data.

[0034] As an example, the quantifiable traffic state (QTS) is a traffic state obtained by using the M(D) r ) can be used to obtain an optimization model for the state of the road intersection. For example, the quantifiable traffic state (QTS) equation is defined as:

[0035] QTS=f(αEP,βPD) γPR (1)

[0036] In formula (1), f is an embedding function used to embed the road intersection state expressed by αEP, βPD, and γPR so as to be processed by a TSC system based on an RL model. Therefore, f can be implemented by a variety of state embedding schemes in the prior art, such as a 2-layer neural network. In formula (1), α and β represent the learnable weights of EP and PD, which balance the importance of "effective pressure" EP and "traffic demand PD" in the representation of road traffic state. γ represents the weight of the local range (PR).

[0037] According to an embodiment of the present invention, an HRL framework following the "rule-data-machine learning" paradigm is provided. The HRL framework is used to identify the best quantifiable combination of rules that express the road intersection state to obtain an optimization model of the road intersection state. Specifically, the proposed HRL framework aims to find the best parameters (such as α, β, γ in the above text) for each known or future road intersection state of QTS.

[0038] Figure 1 A block diagram of an HRL framework according to an embodiment of the present application is shown.

[0039] See also Figure 1 ,The state agent and the traffic agent represent the high-level agent and the low-level agent respectively, each acting as an entity that executes different options.

[0040] Hierarchical reinforcement learning (HRL) organizes complex problems into hierarchical structures to model and simplify problem solving. According to the present application, the traffic control problem is decomposed into two parts, namely, the state agent and the traffic agent, to solve the problem.

[0041] In the HRL framework, the state agent is considered as an entity (high-level agent) that executes high-level options. The task of the state agent is to select appropriate weights (α, β, γ) for various traffic states as rules. Various traffic states are represented by EP, PD, etc. These weights (α, β, γ) can be considered as the high-level selection strategy π of HRL. 0 , because they decide the action to take in each state. The state agent is based on a reward R signal To find the quantifiable state parameters. Based on the state, the action selected by the agent is weighted (α, β, γ). Then, QTS is obtained using equation (1) and provided to the traffic agent.

[0042] The traffic agent is responsible for traffic signal control (TSC). It selects the action to be taken (e.g., traffic signal) based on the QTS provided by the state agent as the traffic state and a variety of optional known or future traffic signal control algorithms, applies the selected traffic signal to the traffic environment, and obtains state feedback of the traffic environment.

[0043] The traffic environment represents a real-world urban road traffic system, or a traffic simulator such as CityFlow (available from https: / / cityflow-project.github.io). Real-world traffic data (available from https: / / traffic-signal-control.github.io) provides an example of a traffic environment.

[0044] Traffic conditions affected by TSC can be obtained from the traffic environment, such as the number of vehicles traveling at the intersection, the number of vehicles in queue, the size of the local range, etc. These traffic conditions are used to generate traffic conditions such as EP, PD, PR, etc. as inputs to the state agent.

[0045] For a set of weights (α, β, γ) generated by the state agent, the corresponding QTS is provided to the traffic agent for implementing TSC, and the corresponding performance evaluation is obtained accordingly. For example, ATT (Average Transfer Time) is used as the performance evaluation. ATT represents the average time taken by vehicles to pass through road intersections in a traffic system, for example. During application or training, each set of weights (α, β, γ) generates a corresponding ATT.

[0046] As an example, formula (2) provides the reward R signal ,in denoted as a benchmark ATT for comparison, for example, the ATT obtained by the traffic signal control algorithm when QTS is not used.

[0047]

[0048] in The average of ATT achieved by existing TSC methods (such as FRAP, PressLight, MetaLight, etc.) in the same dataset.

[0049] By integrating the state agent and the traffic agent, the HRL (hierarchical reinforcement learning) framework according to the embodiment of the present application aims to determine the optimal parameters (d, β, γ) to accurately represent the traffic state and ultimately improve the performance of TSC. Using the "Rules-Data-ML" paradigm and the HRL framework, the embodiment of the present application also provides a machine learning model for the state agent to determine the QTS representation based on the traffic agent. The formula of the model is as shown in formula (3):

[0050] QTS′=State agent (Env(Traffic agent (QTS))) (3)

[0051] In formula (3), Traffic agent (QTS) is the behavior generated by the traffic agent according to the received QTS (for example, used to control the signal phase of the traffic light), Env (Traffic agent(QTS)) represents the result generated by the TSC method based on the QTS used in the traffic environment. The result represented by Env() includes the traffic state obtained from the traffic environment, such as the number of vehicles in motion, the number of vehicles in queue, the size of the local area, etc. at each road intersection, and also includes states such as EP and PD. Env() also includes performance indicators such as ATT. QTS′=State agent (Env(…)) represents the output generated by the state agent based on multiple state representations obtained and calculated from the traffic environment, that is, the updated or enhanced traffic state representation.

[0052] Table 1 shows an exemplary network structure of a state agent. Its embedding layer includes 3 layers of MLP and a connection layer Concat, and the output layer includes 1 layer of MLP.

[0053] Table 1 State agent network structure

[0054]

[0055] In Table 1, Layer represents the network layer, and its parameters are the size of the input data and the size of the output data of the network layer. The three MLPs in the embedding layer represent the three-layer neural network, and the parameter num_lanes represents the number of lanes, len_all_feature represents all the acquired information of the lanes, hidden_size represents the size of the hidden layer data; len_ep_feature represents EP information, and len_pd_feature represents PD information.

[0056] The training process of the state agent includes collecting historical data in the form of (S, A, R, S') quaternions and updating the network parameters using the deep Q network (DQN) algorithm. The Q function is updated according to the Bellman equation. As an example, the state S as the input of the state agent includes the states EP, PD and PR of the road intersection in the traffic environment. The action A as the output of the state agent is a set of weights (α, β, γ) for equation (1). The reward R of the state agent = R signal , obtained by formula (2), and the state S' is the feedback obtained from the traffic environment after the state agent outputs action A and the traffic agent acts on the traffic environment. The feedback is expressed as the updated state EP, PD and PR of the road intersection.

[0057] During the training of the state agent, the appropriate combination of traffic state weights can be found by optimizing the combination of (α, β, γ) to maximize the reward R. The Q function is updated by equation (4).

[0058]

[0059] in It means that the Q value of all possible actions A' in state S' is maximized.

[0060] This training process enables the HRL framework to find the optimal parameters (α, β, γ) to accurately generate QTS and improve the performance of TSC.

[0061] Figure 2 A block diagram of an HRL framework according to another embodiment of the present application is shown.

[0062] and Figure 1 Compared with the examples shown, Figure 2 In the HRL framework, the traffic agent instantiates multiple TSC algorithms (TSC algorithm 1, TSC algorithm 2, ... and TSC algorithm n). These TSC algorithms are the current state-of-the-art traffic signal control methods (e.g., MPLight, FRAP, CoLight, etc.), and also include traffic signal control methods that will emerge in the future. Figure 2 A plurality of traffic environments (traffic environment 1, traffic environment 2, ... and traffic environment n) are also shown. The traffic environments correspond to the TSC algorithms one by one, so that the TSC algorithms are applied to the corresponding traffic environments, so that each traffic environment is used to respond to and evaluate the performance of the corresponding TSC algorithms, and obtain, for example, the ATT index. The ATT index corresponds to the TSC algorithm, so that the ATT corresponding to TSC algorithm 1, TSC algorithm 2, ... and TSC algorithm n is obtained through each traffic environment. 1 ,ATT 2 …and ATT n It can be understood that in order to facilitate the comparison of the performance of each TSC algorithm, for example, the same traffic data set is deployed in each traffic environment.

[0063] Figure 2 In the example, the traffic agent provides multiple QTS switches (QTS switch 1, QTS switch 2, ... and QTS switch n). QTS open source corresponds to the TSC algorithm one by one, and is used to control whether the QTS generated by the state agent is provided to each TSC algorithm.

[0064] In one embodiment, all QTS switches are closed, indicating that the QTS generated by the agent is applied to all listed TSC algorithms. The same traffic dataset is deployed in all traffic environments. And the global reward R Global =∑R signal As a reward for the training state agent. Thus, the training state agent generates universal weights (α, β, γ) that can improve the performance of more or all TSC algorithms (represented by, for example, ATT) GIt can be understood that during training, multiple TSC algorithms can be run simultaneously, and multiple traffic environments can be deployed to interact with the corresponding TSC algorithms simultaneously to collect the global reward R faster. Global Alternatively, a few or a single computing device can be used to run each TSC algorithm and traffic environment in a time-sharing manner, and the reward R of each TSC algorithm can be obtained successively. signal , and then calculate the global reward R GIobal .

[0065] According to a further embodiment of the present application, after training, effective universal weights (α, β, γ) are generated. G After the state agent is trained, multiple QTS switches are disconnected, and only a single QTS (e.g., QTS switch i) is kept closed, so as to specially train the state agent for the TSC algorithm (e.g., TSC algorithm i) corresponding to the retained QTS switch to generate a dedicated weight (αβ, γ) specifically suitable for the TSC algorithm. Si , to optimize the performance of a specific TSC algorithm.

[0066] Close all QTS switches and train universal weights (α, β, γ) that improve the performance of most or all TSC algorithms. G The corresponding QTS is called G-QTS. To distinguish, the dedicated weights (α, β, γ) applicable to a single specific TSC algorithm Si The corresponding QTS is called S-QTS. G-QTS is used to describe the overall state of the road intersection, improve the performance of the TSC algorithm, and provide support for the new TSC algorithm to quickly obtain the optimal results. S-QTS is used for specific road intersections and specific algorithms to obtain the most advanced (SOTA) performance results.

[0067] Figure 3 A flowchart of a training state agent according to an embodiment of the present application is shown.

[0068] The state agent uses a deep Q network (DQN) to generate a set of weights (α, β, γ). The deep Q network is trained using a quadruple (S, A, R, S') as training data. S represents the state, including, for example, EP (effective pressure), PD (traffic demand), and PR (local range) from the traffic environment. A represents the behavior generated by the deep Q network, specifically the weights (α, β, γ). The weights (α, β, γ) are used to calculate QTS according to formula (1), which is provided to the traffic as the output of the state agent. R represents the reward, for example, obtained based on ATT (average time) in the traffic environment. S' is the new traffic state generated after the traffic agent interacts with the traffic environment using QTS.

[0069] See also Figure 3In step 310, the Q network, Q' network and replay buffer D of the state agent are initialized. It can be understood that w and w' represent the network parameters of the Q network and the Q' network respectively. For the purpose of training, the Q' network is used as the target network. During training, the parameters of the Q network are updated at each step, while the Q' network is updated every C steps (i.e., w is used to update w'). And the traffic environment and weights (α, β, γ) are also initialized. As an example, the traffic environment is a traffic simulator.

[0070] In step 320, EP, PD and PR are obtained from the traffic environment as state S. The method of obtaining EP, PD and PR belongs to the prior art. Optionally, other traffic states or state combinations obtained from the traffic environment are used to construct state s.

[0071] In step 330, the state agent generates a corresponding behavior A according to the state S based on maximizing the Q value output by the Q network. Behavior A represents a new value of a selected set of weights (α, β, γ). And to provide disturbance, the new value of the weight (α, β, γ) is randomly selected with a certain probability. Optionally, the number of weights in behavior A is consistent with the number of traffic states constructed in step 320, and each weight of behavior A corresponds to the traffic state constructed in state S.

[0072] In step 340, the traffic state represented by the behavior A generated in step 330 is provided to the traffic agent. Specifically, QTS is generated using the selected weights (α, β, γ) according to formula (1). QTS serves as the input of the traffic agent.

[0073] In step 345, the traffic agent implements traffic signal control, generates, for example, a traffic signal phase based on the input QTS as the traffic state, and applies it to each road intersection in the traffic environment through the traffic signal lamp. Vehicles / pedestrians in the traffic environment pass based on the traffic signal phase of the road intersection. The traffic agent interacts with the traffic environment based on a certain TSC algorithm, obtains the traffic state and generates a traffic signal phase. The interaction between the traffic agent and the traffic environment includes, for example, multiple traffic signal cycles, each of which has the same or different time intervals.

[0074] In step 350, the average travel time (ATT) of the traffic environment under the control of the traffic agent in step 345 is collected. The ATT comes from multiple road intersections, multiple vehicles and / or multiple traffic signal cycles in the traffic environment.

[0075] In step 360, the new EP, PD and PR in the traffic environment under the control of the traffic agent in step 345 are collected as the updated state S'.

[0076] In step 370, the reward R for training the state agent is calculated according to ATT. For example, R is calculated according to formula (2): signal As a bonus R.

[0077] In step 380, the state S obtained in step 320, the behavior A obtained in step 330, the state S' obtained in step 360, and the reward R obtained in step 370 generate a quadruple (S, A, R, S') which is added to the replay buffer D as new training data.

[0078] Steps 340 to 380 are repeated multiple times to generate more quadruplets (S, A, R, S'). The number of times steps 340 to 380 are repeated is recorded as T, where T is a positive integer. After steps 340 to 380 are repeated T times, the process turns to step 390 instead of step 340.

[0079] In step 390, a number of quadruplets (S, A, R, S') are sampled from the replay buffer D to update the parameters w of the Q network and the parameters w' of the Q' network. And steps 320 to 390 are repeatedly executed to train the state agent. Among them, the parameters w of the Q network are updated each time step 390 is executed, and the parameters w' of the Q' network are updated every C times step 390 is executed. The Q' network is updated by, for example, copying the parameters w to the parameters w'. C is a positive integer.

[0080] Next, return to step 320, obtain EP, PD and PR from the traffic environment again as the state S and repeat the above process. Optionally, each time step 320 is entered, the traffic environment can be initialized to obtain the corresponding state S, or a random state S can be obtained, or the state S can be obtained after interacting with the traffic environment in the previous operation.

[0081] Figure 4 A flowchart of training a state agent according to another embodiment of the present application is shown.

[0082] exist Figure 4 In the embodiment, Figure 3 In comparison, steps 445 to 470 are different from corresponding steps 345 to 370 .

[0083] exist Figure 4 In an embodiment of the present invention, the traffic agent interacts with the traffic environment using multiple TSC algorithms to train the state agent to generate weights (α, β, γ) suitable for multiple TSC algorithms.

[0084] In step 445, the multiple TSC algorithms receive the same QTS from the state agent as the current traffic state input for the TSC algorithm and interact with the traffic environment independently. Thus, the different TSC algorithms in the traffic agent do not affect each other in the traffic environment, but act on the traffic environment independently of each other.

[0085] In step 450, the average transit time ATT obtained after each TSC algorithm interacts with the traffic environment under the control of the traffic agent is collected. j , where j represents the jth TSC algorithm.

[0086] In step 460, according to the ATT corresponding to each TSC algorithm j Calculate the global reward R Global For example, R Global =∑R signal,j ,in n represents the number of TSC algorithm types.

[0087] In step 470 , an updated state S′ is obtained after each TSC algorithm acts on the traffic environment.

[0088] The trained state agent can generate QTS according to the state obtained from the traffic environment (e.g., a combination of EP, PD, and PR). Using QTS as an optimized model representing the traffic state obtained according to the present application and providing it to the TSC algorithm can enable the TSC algorithm to achieve better performance in urban road traffic signal control.

[0089] The trained state agent is deployed in the real-world traffic control system. By collecting traffic states (such as EP, PD, and PR), the state agent generates QTS, which is provided to the TSC algorithm used by the traffic control system as the input traffic state, and the TSC algorithm generates the current traffic signal phase to be applied to the road intersection.

[0090] A large number of experiments were also conducted on real-world traffic datasets to evaluate the HRL model according to the embodiments of the present application. TSC results were obtained using the traditional RL model and the QTS obtained based on the embodiments of the present application applied to the RL model (referred to as the QTS-based method). The QTS-based method was divided into G-QTS for all TSC algorithms and specific QTS for RL methods to achieve new SoTA results.

[0091] In the comparative experiments, a real-world dataset (available from https: / / traffic--signal-control.github.io) is used, which includes datasets of three cities: Jinan, Hangzhou, and New York.

[0092] For a clear comparison, these traffic signal control methods are divided into three parts: traditional methods, RL methods, and QTS-based methods. Table 2 reports the experimental results on real datasets in Jinan, Hangzhou, and New York using ATT as the evaluation metric.

[0093] In Table 2, each row in the leftmost table represents a TSC method. The method marked with "G-GTS" indicates the ATT performance obtained by the corresponding TSC method when G-GTS is used as the traffic state. The method marked with "S-GTS" indicates the ATT performance obtained by the corresponding TSC method when S-GTS is used as the traffic state.

[0094] In Table 2, "1", "2" or "3" in the second row represents different data sets. For example, "Jinan 1", "Jinan 2" and "Jinan 3" are different data sets from Jinan.

[0095] Table 2 Comparison of ATT(s) performance of various TSC methods in Jinan, Hangzhou and New York data

[0096]

[0097] Figure 5 The comparison shows the performance impact of using the MP method as the baseline method.

[0098] from Figure 5 In this paper, the ATT performance of the MP method is used as a baseline to compare the changes in the ATT performance of the method using Q-QTS, the Advanced-MP method, and the method using Q-QTS. Each chart represents the experimental results on a dataset.

[0099] pass Figure 5 Different effects of G-QTS on the performance of MP methods are observed. On some datasets (JiNan1, JiNan2, JiNan3, and HangZhou1), ATT increases slightly, while other datasets show a slight decrease in ATT, indicating that G-QTS has an uneven effect on improving the ATT performance of MP methods. However, applying S-QTS to Advanced-MP methods consistently reduces ATT on all datasets (lower ATT means better performance).

[0100] Figure 6 The comparison shows the performance impact of using the MPLight method as the baseline method.

[0101] from Figure 6In this paper, the ATT performance of the MPLight method is used as a baseline to compare the changes in the ATT performance of the method using Q-QTS, the Advanced-MPLight method, and the method using Q-QTS with the baseline performance. Each chart represents the experimental results on a dataset.

[0102] like Figure 6 As shown in Figure 2, incorporating G-QTS into the MPLight method can lead to significant performance enhancements on most datasets. Similarly, applying S-QTS to the Advanced-MPLight method also achieves comprehensive performance upgrades on all datasets. This suggests that S-QTS can help determine the optimal (α, β, γ) composition, thereby improving the performance of the Advanced-MPLight method.

[0103] Figure 7 The comparison shows the performance impact of using the CoLight method as the baseline method.

[0104] from Figure 7 In this paper, the ATT performance of the CoLight method is used as a baseline to compare the changes in the ATT performance of the method using Q-QTS, the Advanced-CoLight method, and the method using Q-QTS with the baseline performance. Each chart represents the experimental results on a dataset.

[0105] exist Figure 7 Comparing G-QTS-CoLight and CoLight shows mixed results, with the overall trend being for ATT performance improvement despite slight performance degradation on some datasets. When comparing Advanced CoLight and S-QTS-CoLight, the latter performs consistently across all datasets and shows smaller fluctuations in error range, indicating performance enhancement, stability, and robustness.

[0106] From the overall results, this application has the following beneficial effects:

[0107] (1) Compared with state-of-the-art TSC methods, including traditional MP and recent RL methods (FRAP, MPLight, and CoLight), G-QTS can better represent the traffic state and achieve better performance;

[0108] (2) S-QTS for MPLight and CoLight can find the optimal (α, β, γ) composition, achieving new state-of-the-art (SOTA) performance based on MPLight and CoLight methods, respectively.

[0109] Although the present application is described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application.

[0110] Those skilled in the art to which these embodiments relate and who benefit from the teachings presented in the above description and associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that the present application is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used only in a general and descriptive sense and are not used for the purpose of limitation.

Claims

1. A road intersection traffic signal control method based on hierarchical reinforcement learning (HRL), including: Acquire a first traffic state of a road intersection, wherein the first traffic state includes a plurality of elements; generating a first plurality of weights according to the first traffic state, each weight corresponding to each of a plurality of elements of the first traffic state; generating a first quantifiable traffic state QTS according to the first plurality of weights and the plurality of elements; generating a traffic signal phase for the road intersection according to the first quantifiable traffic state QTS; A state agent including a deep Q network is used to generate a first plurality of weights and a first quantifiable traffic state QTS according to the first traffic state; A traffic agent including a reinforcement learning module is used to generate a first traffic signal phase for the road intersection according to the first quantifiable traffic state QTS; the state agent and the traffic agent are respectively a relatively high-level agent and a relatively low-level agent in hierarchical reinforcement learning HRL; Wherein, to train the deep Q network, the following steps S1 to S7 are performed; Step S1, obtaining a plurality of elements representing a second traffic state from a traffic environment as a state S of the deep Q network; Generating a second plurality of weights as behaviors A of the deep Q network according to the state S based on maximizing the Q value of the deep Q network; Step S2, generating a second quantifiable traffic state QTS according to a plurality of elements of the second traffic state and the second plurality of weights and providing the second quantifiable traffic state QTS to the traffic agent; Step S3, the traffic agent implements traffic signal control according to the second quantifiable traffic state QTS; Step S4, collecting the average travel time ATT of the traffic environment under the control of the traffic agent, and Calculating a reward R for training the deep Q network; Step S5, collecting multiple elements representing new values ​​of the second traffic state in the traffic environment under the control of the traffic agent as the updated state S' of the deep Q network; Step S6, generating training data using the state S, the behavior A, the reward R and the state S'; and Step S7, training the deep Q network from the training data.

2. The method for controlling traffic signals at road intersections according to claim 1, wherein: In step S3, the traffic agent generates traffic signal phases in multiple traffic environments using multiple traffic signal control methods, wherein the multiple traffic signal control methods correspond to the multiple traffic environments one by one, and each of the multiple traffic signal control methods generates a traffic signal phase according to the second quantifiable traffic state QTS; and In step S4, the rewards obtained according to the average travel time ATT in each of the plurality of traffic environments are accumulated. as the reward R for training the deep Q network.

3. The method for controlling traffic signals at intersections according to claim 2, further comprising: Retraining the trained deep Q network; in Wherein, to train the deep Q network again, the steps S1 to S7 are performed again; When the step S3 is performed again, the traffic agent generates a traffic signal phase in the corresponding traffic environment using a first traffic signal control method, wherein the first traffic signal control method generates a traffic signal phase according to the second quantifiable traffic state QTS generated by the state agent including the trained deep Q network; and When the step S4 is executed again, the average travel time (ATT) in the traffic environment corresponding to the first traffic signal control method is collected, and the reward R for training the deep Q network is calculated.

4. The method for controlling traffic signals at intersections according to any one of claims 1 to 3, wherein: The plurality of elements of the first traffic state include effective pressure EP, passing demand PD, and local range PR.

5. The method for controlling traffic signals at intersections according to claim 4, wherein: According to QTS = f(αEP, βPD)γ PR Calculating the first quantifiable traffic state QTS and the second quantifiable traffic state QTS; Where α represents the weight corresponding to the effective pressure EP, β represents the weight corresponding to the passing demand PD, and γ represents the weight corresponding to the local range PR; f represents the embedding function, which is used to embed the state of the road intersection expressed by αEP, βPD, and yPR; In step S4, according to Calculate the reward R, where Indicates the baseline average travel time for comparison.

6. The method for controlling traffic signals at intersections according to claim 2, wherein: The plurality of elements of the first traffic state include effective pressure EP, passing demand PD and local range PR; According to QTS = f(αEP, βPD) γPR Calculating the first quantifiable traffic state QTS and the second quantifiable traffic state QTS; Where α represents the weight corresponding to the effective pressure EP, β represents the weight corresponding to the passing demand PD, and γ represents the weight corresponding to the local range PR; f represents the embedding function, which is used to embed the state of the road intersection expressed by αEP, βPD, and yPR; In step S4, according to R=∑R j ,as well as Calculate the reward R, where j represents the jth traffic control method among the multiple traffic signal control methods, and n represents the number of types of the multiple traffic signal control methods. Represents the benchmark average travel time of the jth traffic control method used for comparison.

7. The method for controlling traffic signals at intersections according to claim 6, wherein: The multiple traffic signal control methods include a traffic signal control method based on reinforcement learning.

8. The method according to any one of claims 5 to 7, wherein Steps S3 to S6 are repeatedly performed multiple times to obtain multiple quadruples (S, A, R, S') to generate training data; and The steps S1 to S7 are repeatedly performed multiple times to train the deep Q network multiple times.

9. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images