Lightweight reinforcement learning method suitable for few-sample traffic signal control

Abstract

The invention provides a road intersection traffic signal control method LiteLight based on a reinforcement learning model. The road intersection traffic signal control method based on the reinforcement learning model comprises the following steps: acquiring a local state PTS of a road intersection at a specified time interval; wherein the local state PTS of the road intersection comprises local states PTS of all traffic signal phases of the road intersection; the reinforcement learning model generates a next traffic signal phase for the road intersection according to the local state PTS of the road intersection; and setting the traffic signal of the road intersection by using the next traffic signal phase.

Description

Technical Field

[0001] The present application relates to machine learning technology, and in particular, to a traffic signal control method and information processing device based on reinforcement learning. Background Art

[0002] With the continuous increase in traffic demand, it is crucial to improve the efficiency of existing transportation infrastructure to reduce urban congestion and exhaust emissions. Among them, real-time adaptive traffic signals are one of the main tools to improve the efficiency of my country's urban transportation network. However, due to the complexity and randomness of the traffic system, traffic signal control (TSC) remains a challenging task.

[0003] Traditional TSC method

[0004] Traditional traffic signal control (TSC) methods, such as the fixed-time signal control introduced by Webster in 1958 [https: / / trid.trb.org / view / 113579], have shown limitations in adapting to dynamic traffic conditions. Now, in many modern cities, FixedTime [https: / / rosap.ntl.bts.gov / view / dot / 20661], SCOOT [https: / / trid.trb.org / view / 186640] and SCATS [https: / / www.semanticscholar.org / paper / SCATS:-Sydney-Co-Ordinated-Adaptive-Traffic-System:-Lowrie / 93a56f13a698dcab2df2f7a88229ba15091b3004#citing-papers] are still used in TSC systems, which rely heavily on expert-designed traffic signal schemes.

[0005] TSC methods using traffic rules have been implemented, where traffic lights switch at fixed time intervals. These rule-based methods fall into three categories: fixed-time control [https: / / rosap.ntl.bts.gov / view / dot / 20661, https: / / www.sciencedirect.com / science / article / abs / pii / 0378437196001446], adaptive control [https: / / trid.trb.org / view / 186640, https: / / doi.org / 10.1016 / j.trc.2013.08.014], and inductive control [https: / / link.springer.com / chapter / 10.1007 / 978-1-4471-5113-5_3, https: / / arxiv.org / abs / nlin / 0411066]. Fixed-time control relies on a predefined signal scheme (i.e., cycle length, phase division, and offset). Adaptive control takes into account local traffic information from sensors near the intersection. Therefore, it is usually better than fixed-time control. Inductive control affects the performance of TSC through predetermined thresholds (i.e., cycle length, waiting time), where the switching of traffic signals is based on the measured needs. The latest research trend [https: / / www.sciencedirect.com / science / article / pii / S0968090X20307324, https: / / arxiv.org / abs / 2112.02336] is to focus on the accurate analysis of traffic conditions after obtaining more traffic information. Some of these methods not only improve the results of TSC, but even perform better than some RL-based methods [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744, https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900].

[0006] Adaptive traffic signal control TSC methods, such as SCOOT [https: / / trid.trb.org / view / 179439], SCATS [https: / / trid.trb.org / view / 488852] and Self-Organizing Traffic Lights (SOTL) [https: / / arxiv.org / abs / nlin / 0411066]. TSC methods based on maximum pressure (MP) [https: / / www.sciencedirect.com / science / article / abs / pii / S0968090X13001782] and effective pressure (EP) [https: / / arxiv.org / abs / 2112.02336] target the length of the vehicle queue and the pressure of the queue, respectively. Generally speaking, adaptive TSC methods [https: / / link.springer.com / chapter / 10.1007 / 978-1-4471-5113-5_3,https: / / doi.org / 10.1016 / j.trc.2013.08.014] need to obtain the traffic conditions of vehicles from the traffic flow at the intersection before making adaptive decisions. Intuitively, the more observations we have about the traffic conditions, the better the performance of adaptive TSC methods will be.

[0007] RL-based TSC method

[0008] In recent years, reinforcement learning (RL)-based methods have become a promising trend in TSC applications. In RL, the traffic signal optimization problem is formulated as a Markov decision process (MDP), where S represents the set of traffic states, A represents the set of traffic signal control actions, and R represents the reward. The traffic signal control agent decides the control action a∈A of the traffic signal to be taken in a given state s∈S to maximize the expected reward. The traffic signal control agent is, for example, a traffic light control device or control system. While FRAP[https: / / dl.acm.org / doi / 10.1145 / 3357384.3357900] adopts the phase competition principle, Preslight[https: / / dl.acm.org / doi / 10.1145 / 3292500.3330949] maximizes throughput by utilizing MP-based state representation and reward function. Colight[https: / / dl.acm.org / doi / 10.1145 / 3357384.3357902] uses graph attention neural network, MetaLight improves learning efficiency through meta reinforcement learning technology. Optimizing reward function and state representation is the key to RL model in TSC application. Advanced-MP[https: / / proceedings.mlr.press / v162 / zhang22ah.html] algorithm considers the competition between moving vehicles and queuing vehicles, Dynamiclight[https: / / arxiv.org / abs / 2211.01025] uses the maximum queue length for phase selection and implements deep Q learning to determine the phase duration.

[0009] RL-based models can learn how to generate subsequent actions by observing feedback from the environment through trial and error and rewards, without the need for an accurate traffic model.

[0010] Q-learning [https: / / link.springer.com / article / 10.1007 / BF00992698] is one of the most commonly used value-based methods in RL, which updates its value function based on the Bellman equation [https: / / books.google.com.hk / books?hl=en&lr=&id=dHrsHz0VocUC&oi=fnd&pg=PA1&dq=Optimization+in+Economic+Theory+(2nd+ed.)&ots=-0O9_P7jsn&sig=NUgkP7OYNjaAMl47e3NZhiBo2kk&redir_esc=y#v=onepage&q=Optimization%20in%20Economic%20Theory%20(2nd%20ed.)&f=false]. In the field of computer games, Q-learning has achieved great success, such as Atari [https: / / arxiv.org / abs / 1312.5602]. In addition, Q-learning is an off-policy learner that learns the value of the optimal policy independently of the agent's actions. Due to its powerful discrete decision-making ability, Q-learning can also be applied to the TSC problem of a single (or isolated) intersection [https: / / www.semanticscholar.org / paper / Intelligent-Traffic-Light-Control-Wiering-Veenen / 02053b02ca9d98e57319f7376aead1b1a064d6ec]. Given its scalability, the combination of multiple Q-learning [https: / / digital-library.theiet.org / content / journals / 10.1049 / iet-its.2009.0070] models has also been applied to the TSC problem of several intersections.

[0011] In recent years, RL has attracted more and more research interests and has been widely used in solving TSC problems [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744,https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900]. Some RL-based collaborative methods have extended their observation of traffic conditions and shown superior performance for controlling large-scale traffic signals. Recently, some new methods [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744,https: / / dl.acm.org / doi / abs / 10.1145 / 3292500.3330949] have been proposed to accurately represent traffic conditions and achieve better performance. Deep neural networks further adapt them to various environmental conditions. Therefore, for a single intersection, most RL-based methods perform better than traditional methods.

[0012] Recent works, such as PressLight[https: / / dl.acm.org / doi / abs / 10.1145 / 3292500.3330949], are deep Q-learning networks that can solve the multi-intersection TSC problem. On the other hand, MPLight[https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744] is based on the FRAP framework, which allows its parameters to be shared between different intersections. This approach achieves better performance than PressLight. IntelliLight[https: / / dl.acm.org / doi / abs / 10.1145 / 3219819.3220096] develops a complex neural network with a traffic signal phase selector. CoLight[https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902] uses graph attention networks[https: / / arxiv.org / abs / 1710.10903] in the setting of TSC and achieves good results on large-scale road networks. MetaLight[https: / / ojs.aaai.org / index.php / AAAI / article / view / 5467] is a value-based meta-reinforcement learning workflow based on the representative gradient meta-learning algorithm (MAML), which includes periodically alternating individual-level adaptation and global-level adaptation. In addition, HiLight[https: / / ojs.aaai.org / index.php / AAAI / article / view / 16147] uses a hierarchical RL model and outperforms MPLight and CoLight.

[0013] In 2019, Zheng et al. [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900] proposed a Q-learning-based FRAP framework that is invariant to symmetric situations in traffic flow (such as flipping and rotation). In addition, the FRAP framework can handle different road topologies, traffic flows, and complex real-world phase settings, which are similar to recent methods such as MPLight [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744] and MetaLight [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5467].

[0014] In the field of artificial intelligence (AI), pre-training techniques use parameters to initialize models for new tasks. These parameters have been trained in other tasks. Pre-training methods have achieved great success in some challenging tasks such as natural language processing (NLP) [https: / / proceedings.neurips.cc / paper / 2020 / hash / 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html,https: / / arxiv.org / abs / 1810.04805]. Among them, the generalization ability of AI models is crucial for practical applications. However, due to the differences in traffic flow distribution and traffic terrain, pre-training methods cannot be directly used in RL environments. Therefore, the latest RL-based TSC model [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744, https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902] did not achieve good generalization performance as expected.

[0015] Traffic status display

[0016] Traffic state representation has an important impact on the quality of both traditional TSC models and RL-based TSC models. In order to better obtain information about large-scale traffic conditions, graph convolutional neural networks can be added to the TSC model [https: / / ieeexplore.ieee.org / abstract / document / 8569301]. For example, PressLight [https: / / dl.acm.org / doi / abstract / 10.1145 / 3292500.3330949] incorporates pressure into the design of state and reward; MPLight [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744] uses traffic flow pressure as state information and intersection pressure as reward.

[0017] In recent research, a new concept, “Effective Pressure (EP)”, was introduced in [https: / / arxiv.org / abs / 2112.02336], and its powerful traffic state representation capability can achieve the state-of-the-art (SOTA) in TSC.

[0018] Intuitively, observing comprehensive traffic conditions is crucial for designing traffic signal control (TSC) based on reinforcement learning (RL). To the best of our knowledge, few researchers have studied the observation range of traffic state representation, and a common assumption is that the larger the observation data, the better the performance. Most of the latest RL methods try to capture more traffic network conditions to design adaptive models, which leads to an explosive growth of the state space in actual deployment, making the problem more difficult.

[0019] In practical applications, learning the optimal TSC policy becomes more difficult in more complex scenarios. Figure 1 A schematic diagram of a road intersection is shown. Figure 1 In the figure, there is a standard 4-way intersection, each with left turn, straight ahead, and right turn. According to the traffic rules, there are four signal phases for the combination of different traffic movements. Assuming the number of observed lanes is n, when all four phases are considered, the exploration space will increase to O(n 4 ). Therefore, reducing the complexity of the state space in TSC design is a crucial issue, as well as how to design a lightweight RL model to handle the TSC problem based on the reduced complexity of the TSC state space.

[0020] The limitations of current RL-based TSC methods can be summarized as follows:

[0021] They require more observations of traffic conditions, further increasing the complexity of the TSC model

[0022] Their training process is usually long and they have poor generalization performance to new road intersections outside the acquired training dataset

[0023] They require more computing resources and expensive equipment to deploy in the real world Summary of the invention

[0024] According to a first aspect of the present application, a first road intersection traffic signal control method based on a reinforcement learning model is provided, comprising: obtaining the local state PTS of the road intersection at a specified time interval; wherein the local state PTS of the road intersection includes the local state PTS of all traffic signal phases of the road intersection, each traffic signal phase corresponds to a traffic movement i and a traffic movement j without conflict, and the local state PTS of each traffic signal phase includes the local state PTS of the corresponding traffic movement i and the local state PTS of the traffic movement j; wherein, according to the local queue lengths of the traffic movement i and the traffic movement j, and and the number of local vehicles in traffic movement i and traffic movement j and Generate a local state PTS of the traffic signal phase to which traffic movement i and traffic movement j belong; wherein the local queue length is the number of vehicles queued in a traffic movement within the local range L of the road intersection, denoted as q L (l,m), where l and m represent the entry and exit roads of a traffic movement; the local number of vehicles is the number of vehicles in a traffic movement within the local range L of the road intersection, denoted by r L (l,m), the local range L is the maximum distance that a vehicle passing through the intersection can pass within a specified time; through a fully connected neural network, and Encoding, the result is recorded as and connect The local state PTS of the traffic signal phase to which traffic movement i and traffic movement j belong is obtained from the traffic signal phase to which traffic movement i and traffic movement j belong; the reinforcement learning model generates a next traffic signal phase for the road intersection according to the local state PTS of the road intersection; and the traffic signal of the road intersection is set using the next traffic signal phase.

[0025] According to the second road intersection traffic signal control method based on reinforcement learning model according to the first aspect of the present application, the reinforcement learning model includes a FRAP model and a CoLight model.

[0026] According to the third road intersection traffic signal control method based on reinforcement learning model of the first aspect of the present application, wherein the reinforcement learning model uses the local state PTS of the road intersection as the traffic state to generate the next traffic signal phase for the road intersection; and the fully connected neural network is a one-layer fully connected neural network.

[0027] According to the fourth road intersection traffic signal control method based on the reinforcement learning model of the first aspect of the present application, the phase of the traffic signal phase to which the traffic movement i and the traffic movement j belong is expressed as Through the fully connected neural network Encoding, the result is recorded as

[0028] Through the fully connected neural network and Encoding, the result is recorded as d i ; Through the fully connected neural network and Encoding, the result is recorded as d j ,in Combination d i With dj The local state PTS of the traffic signal phase to which traffic movement i and traffic movement j belong is obtained.

[0029] According to the fifth road intersection traffic signal control method based on the reinforcement learning model of the first aspect of the present application, the local state PTS of the road intersection is used as the traffic state of the reinforcement learning model; the next traffic signal phase for the road intersection is generated as the action of the reinforcement learning model; after the traffic signal of the road intersection is set with the next traffic signal phase, the total queue length of the road intersection after a specified time interval is used as the reward r of the reinforcement learning model, wherein Lin is the set of all incoming lanes at the road intersection, is the queue length within the local range L entering lane ì; training samples are generated according to the traffic state, action and reward r of the reinforcement learning model to train the reinforcement learning model.

[0030] According to the second aspect of the present application, a first road intersection traffic signal control method based on a reinforcement learning model according to the second aspect of the present application is provided, comprising: obtaining the local queue length and the number of local running vehicles of each traffic movement at the road intersection; wherein the local queue length is the number of vehicles queued in a traffic movement within the local range L of the road intersection, denoted as q L (l,m), where l and m represent the entry and exit roads of a traffic movement; the local number of vehicles is the number of vehicles in a traffic movement within the local range L of the road intersection, denoted by r L (l,m), the local range L is the maximum distance that vehicles passing through the intersection can pass within a specified time; generate an input vector q according to the local queue length of each traffic movement at the road intersection, wherein the number of rows of vector q is 2, and the number of columns is equal to the number of traffic movement groups, wherein each traffic movement group includes 2 traffic movements that do not conflict with one of the traffic signal phases; generate an input vector r according to the number of local vehicles traveling for each traffic movement, wherein the number of rows of vector r is 2, and the number of columns is equal to the number of traffic movement groups; encode q and r respectively through a layer of fully connected neural network, and then sum the obtained results according to the column vector to obtain V including two vectors pts ; Generate multiple input phase representations according to each traffic signal phase, wherein each input phase representation represents an allowed traffic signal phase and multiple prohibited other traffic signal phases; Generate a phase embedding vector V according to each input phase representation phase ; Embed each phase into vector V phase Respectively with V pts Connect to obtain the state embedding vector H corresponding to each traffic signal phasek ; Embedding vector H for multiple states k The competition result vector is reorganized and generated through a layer of fully connected neural network, wherein each row of the competition result vector corresponds to one of the traffic signal phases; the competition result vector is summed up by column to obtain a phase score corresponding to each traffic signal phase; the reinforcement learning model generates a next traffic signal phase for the road intersection according to the phase scores of each traffic signal phase at the road intersection; and the traffic signal at the road intersection is set using the next traffic signal phase.

[0031] According to the second method for controlling traffic signals at road intersections based on a reinforcement learning model according to the second aspect of the present application, the number of vehicles queued in a traffic movement within a local range L of the road intersection is For any other entry lanes in a traffic movement TM except the right turn lane, The sum of the queue lengths in the local range L of all entry lanes excluding the right-turn lane representing the traffic movement TM; the number of vehicles traveling in a traffic movement in the local range L of the road intersection The sum of the number of vehicles traveling in the local range L of all entry lanes excluding the right-turn lane representing the traffic movement TM.

[0032] According to the third road intersection traffic signal control method based on the reinforcement learning model of the second aspect of the present application, the method further includes embedding the state into a vector H k As the traffic state of the reinforcement learning model, the next traffic signal phase for the road intersection generated according to the phase score and the corresponding phase score are used as the action and the corresponding reward of the reinforcement model respectively; training samples are generated according to the traffic state, action and reward of the reinforcement learning model to train the reinforcement learning model.

[0033] According to a third aspect of the present application, a traffic signal control method for a traffic system including multiple road intersections is provided, comprising: at one, multiple or each of the multiple road intersections, using one of the road intersection traffic signal control methods based on a reinforcement learning model according to the first aspect of the present application, setting the traffic signal of the road intersection where the traffic signal is located. .

[0034] According to the fourth aspect of the present application, an information processing device according to the third aspect of the present application is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, one of the road intersection traffic signal control methods according to the first to third aspects of the present application is implemented.

[0035] Inspired by the fact that each bird in a flock of birds takes actions based on its local perception, according to the embodiments of this application, (1) PTS (Partial State) is proposed as the traffic state in TSC to achieve the collection of traffic conditions with less observations; (2) PTS is used as the traffic state to improve the state input of current reinforcement learning algorithms (such as MaxPressure, FRAP and CoLight, etc.); (3) Using PTS and a simpler RL architecture, a lightweight RL structure with prior knowledge, LiteLight, is designed. Comprehensive experimental results from multiple real-world datasets show that: (1) PTS can significantly improve the results of current RL-based methods and has better generalization; (2) The LiteLight model achieves the best results (SOTA) in large-scale TSC with lower complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present application, together with the preferred mode of use and further objects and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a road intersection is shown.

[0038] Figure 2 Three simple turning behaviors of bird flocks are demonstrated.

[0039] Figure 3 The local range of the observation environment of the road intersection agent according to an embodiment of the present application is shown.

[0040] Figure 4 The network structure diagram of the reinforcement learning model LiteLight according to an embodiment of the present application is shown.

[0041] Figure 5 The results of the migration test are shown.

[0042] Figure 6 Model performance during the parameter updating step of the training process is shown for PTS-FRAP and LiteLight.

[0043] Figure 7 The model performance under different state settings is demonstrated. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0045] The movement of a flock of birds appears complex, yet presents smooth and harmonious dynamics. In fact, this complex yet smooth group movement is simply the aggregated result of the actions of each individual bird, although it appears to be centrally controlled to handle the complexity. However, a simulated Flock of Bird (FOB) movement is developed from a distributed behavior model [https: / / doi.org / 10.1145 / 37402.37406]. Each bird acts only based on its local perception of the surrounding birds.

[0046] Figure 2 Three simple turning behaviors of bird flocks are demonstrated.

[0047] Craig Reynolds et al. [https: / / doi.org / 10.1145 / 37402.37406] developed a computer model based on 3D computational geometry, called Boids, to monitor FOBs. Boids can coordinate animal movements, such as flocks of birds and schools of fish, and consist of three simple turning behaviors that describe how individual birds adjust their movements based on the positions and speeds of their neighbors (e.g. Figure 2 shown).

[0048] Figure 2 In the figure, (a), (b) and (c) correspond to one of the three steering behaviors respectively.

[0049] (a) Separation: Each bird attempts to maintain a reasonable distance from all nearby birds to avoid overcrowding

[0050] (b) Alignment: Birds try to change their positions to align with the average alignment of other nearby birds

[0051] (c) Cohesion: Each bird tries to move toward the average position of other nearby birds.

[0052] Similarly, urban traffic control systems consist of a large number of traffic lights. It can be said that the entire urban traffic movement is like a flock of birds (FOB), a complex aggregate movement controlled by traffic lights. Unfortunately, these random movements often lead to traffic jams. Therefore, smarter traffic signal control (TSC) is essential to improve traffic efficiency and alleviate traffic congestion.

[0053] Inspired by the bird flock motion model, in order to meet these challenges, the embodiment of the present application proposes a small amount of observation method called PTS (Partial State) to obtain traffic environment information. The proposed PTS improves the state input of current RL algorithms (such as FRAP [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900], MPLight [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744], CoLight [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902]), and obtains good results. PTS is also used to design a lightweight reinforcement learning method called LiteLight, which is simpler in structure and runs faster than the latest RL architecture while maintaining good performance.

[0054] Definition of terms

[0055] Definition 1: Traffic network, road intersection, road, entering road, leaving road, lane

[0056] A traffic network is described as a directed graph, where each node represents a road intersection (also referred to as an intersection) and each edge represents a road. Each intersection has an entry road and an exit road. Each road consists of one or more lanes that determine how vehicles pass through the intersection, such as turning left, going straight, and turning right. The set of entry lanes and exit lanes of intersection i are represented as and Let l, m, k represent the road, and l′, m′, k′ represent the lane. Figure 1 (a) in FIG. 5 describes an intersection consisting of four roads.

[0057] Definition 2: Traffic Movement (Traffic Flow)

[0058] Traffic movements (also called traffic flows) are defined as the behavior of vehicles moving through an intersection from one incoming road to one outgoing road. A traffic movement from road l to road m is represented by (l,m). At an intersection with three lanes per road, there are three different traffic movements for each incoming road, one for each of the three outgoing roads. Figure 1 There are 12 possible traffic movements at the road intersection shown. For traffic movement (l,m), Figure 1In the example, road l and road m each have three lanes, and the behavior of passing through the road intersection from any lane of road l to any lane of road m belongs to traffic movement (l,m). Therefore, (l,m) can be expressed as set{(l′,m′)}, where set{()} represents the set composed of the elements in ().

[0059] Definition 3: Signal Phase

[0060] Each signal phase is a set of non-conflicting traffic movements. Use s to represent a phase. Where s = set{(l,m)}, s∈S i . Si represents the set of signal phases at road intersection i. Figure 1 (c) in Figure 4 describes the four most commonly used signal phases.

[0061] Definition 4: Markov decision process

[0062] Markov decision process (MDP) is a mathematical model used to describe sequential decision problems, which is itself a dynamic optimization method. MDP consists of the following five elements:<S,A,P,R,T> The typical TSC problem can be modeled as an MDP, including the state set State(S), the action set Action(A), the reward function Reward(R) and the decision time set(T). The agent learns a policy π. This policy determines the best action a to take when the state s is given. Therefore, the subsequent expected discounted return is maximized. Among them G t is the expected return at time t, and γ (γ < 1) is a discount factor. R t+n+1 Represents the reward obtained from the environment at n+1 steps after time t.

[0063] Figure 3 The local range of the observation environment of the road intersection agent according to an embodiment of the present application is shown.

[0064] Inspired by the observation that each bird in the flock of birds only takes action based on its local perception, according to the embodiment of the present application, only the road conditions near the intersection (such as Figure 3 In the embodiment of the present application, a new concept - partial state (PTS) is introduced to set a more reasonable observation range for traffic signal control TSC.

[0065] In order to accurately describe the local state (PTS), several new concepts are proposed based on the basic definitions provided above, as follows:

[0066] Definition 6: Local Scope

[0067] Partial Range is the distance between each vehicle at t duration The maximum distance that can be passed within can be described as

[0068] L=V max ×t duration (1)

[0069] Among them, L represents the local range, V max is the maximum speed of the vehicle, and t is the specified length of time. For example, if the maximum speed of the vehicle is 12 m / s, then t duration When =10s, the corresponding local range L=120m.

[0070] Definition 7: Local Queue Length

[0071] The partial queue length refers to the number of vehicles queued in a traffic movement (see Definition 2) within the local range L of an intersection, which can be expressed as q L (l,m) description.

[0072] q L (l,m)=∑q(l') (2)

[0073] where q(l') is an entry lane l'(l'∈L in ) is the queue length within the local range L of the queue. The queue length can be expressed by the number of queued vehicles or the length of the queued vehicles.

[0074] Definition 8: Number of local vehicles

[0075] The number of partial running vehicles refers to the number of vehicles running in a traffic movement (see Definition 2) within the local range L of an intersection, which can be expressed as r L (l,m) to describe.

[0076] r L (l,m)=∑r(l') (3)

[0077] Among them, r(l') is an entry lane l'(l'∈L in ) is the number of vehicles traveling within a distance L.

[0078] According to an embodiment of the present application, a PTS representation of traffic movement is proposed, including the local queue length (Definition 7) and the local number of traveling vehicles (Definition 8).

[0079] PTS←{q L (l,m);r L(l,m)}(l∈L in ) (4)

[0080] In addition, when complex intersections are involved, the local queue length q L (l,m) and the number of local vehicles r L (l,m) are easier to obtain and do not require further calculation compared to pressures (such as the traffic pressure of a road intersection defined in the prior art). Using PST as the traffic state of the road intersection, the generated traffic state is applied to a variety of TSC methods to generate signal phase control for the road intersection.

[0081] Formula (4) shows a local state (PTS) of a traffic movement. For a traffic signal phase, it includes multiple non-conflicting traffic movements, and the PTSs of these traffic movements together constitute the PTS of the signal phase. Similarly, a road intersection has multiple traffic signal phases, which together include all traffic movements at the road intersection, and the PTSs of these traffic movements together constitute the PTS of the road intersection. In some embodiments, the signal phase includes or does not include traffic movements representing right turns. For example, in order to simplify the representation of the state or simplify the complexity of the state acquisition process, the right-turn traffic movement that has less impact on traffic is ignored. Accordingly, the PTS of the signal phase may include or exclude the PTS of the traffic movement representing the right turn.

[0082] According to an embodiment of the present application, a method for controlling traffic signals using PTS as a traffic state is provided, which is called PTS-XLight. Referring also to Table 1, PTS-XLight in pseudo code form is shown, which includes 11 steps indicated by reference numerals, wherein steps 1 to 11 constitute a loop and are repeatedly executed.

[0083] In the PTS-XLight method, for a certain road intersection, t duration is the time interval, every time t duration Time (also refer to step 1-step 3 of Table 1), obtain the PTS of the road intersection as the traffic state (indicated by step 7), provide the PTS as the traffic state input to the traffic signal control TSC method, and obtain the next traffic signal phase from the TSC method (indicated by step 8) to set the traffic signal of the road intersection. The TSC method used can be a traditional TSC method in the prior art, or a TSC method based on a reinforcement learning (RL) model. The RL model here can be a variety of RL models used for TSC in the prior art. The continuous setting of the traffic signal at the road intersection is achieved by repeatedly executing the above process.

[0084] In step 4 to step 7, the specific method of obtaining PTS is shown. It should be pointed out that the characteristic of PTS itself is that it includes or only includes the local queue length and the local number of vehicles running at the traffic movement at the road intersection, thereby simplifying the traffic state. The traffic models used for TSC in the prior art are diverse. In principle, each model receives the traffic state as input, but the forms of the accepted traffic states are different. For example, for the traditional TSC method MaxPressure (MP), it can receive the PTS directly expressed by the local queue length and the local number of vehicles running for each traffic movement at the road intersection as output, and output the traffic signal phase. Some RL models used for TSC need to receive encoded PTS to meet their requirements on the form and structure of the input traffic state, which is also called PTS embedding.

[0085] Referring back to Table 1, in step 4, the local queue length (Definition 7) and the local number of vehicles (Definition 8) of the traffic movement i at the intersection are embedded in the state. Specifically, let represents the local queue length of traffic movement i, and represents the number of local vehicles in traffic movement i, W v and W r Represents the weight of a layer of the neural network, b v and b r Respectively represent the bias of a layer of the neural network, and Relu() represents the activation function of the layer of the network. In step 4, two one-layer fully connected neural networks are used to respectively and Encode and the output is recorded as and have Where i represents a traffic movement at the intersection. Traffic movement j is a traffic movement that does not conflict with traffic movement i, so traffic movement i and traffic movement j correspond to the same signal phase. The signal phase of traffic movement j is calculated in a similar way. and (not shown in Table 1). In order to obtain the PTS embedding of each signal phase of the road intersection, for each signal phase of the intersection, the h of each corresponding traffic movement is calculated. v and h r (Not shown in Table 1).

[0086] In step 5, the signal phase s including traffic movements i and j is state-embedded. The phase representation of the signal phase s is represented by a fully connected neural network. Encode and the output is recorded as have Where W sRepresents the weight of a layer of the neural network, b s Represents the bias of a layer of the neural network. It can be understood that

[0087] In step 6, for traffic movements i and j belonging to the same signal phase, the Relu() function is used to calculate their demand representation d i and d j ,in Where W represents the weight of a layer of the neural network, and b represents the bias of a layer of the neural network.

[0088] In step 7, for each signal phase p at each intersection, calculate d(p)=d i +d j The PTS embedding of the signal phase and the PTS embedding of all signal phases of the road intersection are collectively used as the traffic state of the intersection.

[0089] In step 8, the traffic state of the road intersection is provided as input to the RL model, and the next traffic signal phase obtained from the RL model is used.

[0090]

[0091] Table 1 PTS-based reinforcement learning methods

[0092] When PTS is provided as traffic status to the RL model, PTS-XLight includes the following key parts:

[0093] State: Each agent at an intersection observes the PTS embedding of its intersection as the traffic state;

[0094] Action: At time t, each agent chooses a signal phase As its action t , then the traffic signal will be set to phase

[0095] Reward: The reward r is the sum of the local queue lengths of all the entry lanes at the intersection. where q(l') is an entry lane l'(l'∈L in ) is the queue length within the local scope L of .

[0096] PTS-XLight provides a method for calculating the state, action, and reward of the RL model. The obtained state, action, and reward can be used in the existing RL model-based traffic signal control method and improve its performance. For example, the state, action, and reward provided by PTS-XLight can be used in the existing RL models FRAP [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900] and CoLight [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902], and the results are recorded as PTS-FRAP and PTS-CoLight. The performance of PTS-FRAP and PTS-CoLight models in processing traffic data is compared experimentally in the following text.

[0097] In a traffic system including multiple road intersections, each road intersection can independently use PTS-XLight to control traffic signals, thereby eliminating the need to synchronize state data and control information between agents at each road intersection, simplifying the deployment of the TSC method in a practical environment. Optionally, among the various road intersections in the traffic system, the PTS-XLight method can be used to control traffic signals at one or some road intersections, while other road intersections use the TSC method of the prior art or other improved TSC methods.

[0098] PTS-XLight generates actions to control the signal phase according to the traffic state of the road intersection (PTS embedding) and obtains rewards from the road intersection. Through this process, training samples are also obtained and can be used to train the neural networks used to generate PTS embedding in Algorithm 1. In addition, the trained neural network has demonstrated excellent generalization ability in the comparative experiments below, so it can be easily migrated and deployed to various road intersections to implement TSC.

[0099] PTS-XLight is not limited to FRAP and CoLight, but can also be integrated into other RL-based models. According to an embodiment of the present application, a lightweight RL-based reinforcement learning model is also provided, called the LiteLight model (described below), which utilizes PTS-XLight.

[0100] Figure 4 The network structure diagram of the reinforcement learning model LiteLight according to an embodiment of the present application is shown.

[0101] LiteLight takes the PTS of the road intersection as the input traffic state and outputs the phase score of each candidate signal phase. The phase score can be used as the basis for selecting the next signal phase (action). In the LiteLight model, the prediction of the phase score (i.e., the Q value in the RL model) is divided into two stages: (a) state embedding, and (b) phase competition (phase score). The LiteLight model is demonstrated by taking the four signal phases at the road intersection as an example.

[0102] See also Figure 4 In part (a), the state input to the LiteLight model includes the road intersection PTS and phase information. The PTS includes the local queue length and the number of local vehicles according to the embodiment of the present application. It can be understood that the defined local queue length and the number of local vehicles correspond to a traffic movement at the intersection, and Figure 4 When the state of (a) is embedded, the intersection PTS includes the local queue length and local number of vehicles corresponding to the four signal phases. The phase information input to LiteLight includes the state representation of each of the four candidate signal phases, such as using a one-hot method to indicate whether each candidate signal phase is selected. In the representation of each phase information, only one phase is selected (green light), and all other phases represent red lights.

[0103] Before the state is input, the PTS on the right turn lane is deleted, while FRAP uses the state of the right turn lane as the model input. Therefore, LiteLight further simplifies the traffic state compared to FRAP. The input PTS includes 8 groups of traffic movements (see Figure 4 The traffic movements represented by numbers 1-8 in the upper left corner of part (a) are the local queue lengths and the local number of vehicles traveling. PTS embedding includes two inputs: a 2×4 local queue length, denoted as q(PTS: queue length), and a 2×4 local number of vehicles traveling, denoted as r(PTS: number of vehicles traveling). Here, the 8 traffic movements are divided into 4 pairs, and the 2 traffic movements in each pair correspond to the same signal phase, thereby forming a 2x4 structure input state representation for the local queue length and the local number of vehicles traveling.

[0104] Phase information is also used as input for PTS embedding. Figure 4 In part (a) of the figure, the input phase represents four candidate signal phases represented by a one-hot method, where the dark squares represent that the corresponding signal phase is selected (green light), and the light squares represent that the corresponding signal phase is red. It can be understood that for the four phases of the intersection, the two traffic movements corresponding to the selected signal phases are allowed (green light), while the traffic movements corresponding to the other three signal phases are prohibited (red light).

[0105] With the fully connected layer (by Figure 4 FC represents) encodes q and r respectively, and then sums the output of the fully connected layer through the column vector (represented by Figure 4 ADD in the figure) obtains two 1x4 vectors (respectively recorded as queue length embedding and number of vehicles embedded, and the whole is recorded as PTS embedding V pts ), which is convenient for phase embedding connection.

[0106] For the input phase representation representing the phase information of the intersection, a one-hot vector with a structure of 1×4 is obtained by embedding (denoted as phase embedding V phase ). For PTS embedded V pts and phase embedded in V phase Make the connection, according to formula (5) and Figure 4 The Concatenate representation of part (a) shows that the result H k It has a 3X4 structure.

[0107] H k =FC(concatenate(V pts ,V phase )) (5)

[0108] Figure 4 Part (a) shows the state embedding vector (H) obtained by taking an input phase representation and the PST of the road intersection as input k ), as Figure 4 The input of part (b) of the example. In this input phase representation, the second square is green. For this intersection, there are three other input phase representations, corresponding to the first square being green, the third square being green, and the fourth square being green. Figure 4 In the manner shown in part (a) of , the other three input phase representations are also used as inputs to obtain three corresponding phase embeddings. These three phase embeddings are respectively combined with V pts Concatenate to get another 3 state embedding vectors, also as Figure 4 Enter part (b) of .

[0109] Figure 4 Part (b) shows the phase competition stage of the LiteLight model. This part receives the state embedding vectors (H) corresponding to the four different input phase representations generated in part (a). k ). The four state embedding vectors are reorganized (by Figure 4 The Re-combine representation of part (b) of ) gives a state competition representation with a structure of 3x4x4. After a fully connected (FC) layer (by Figure 4The FC representation of part (b) of the competition result vector is obtained by summing the column vectors of the competition result vector (given by Figure 4 The ADD representation of part (b) of ) obtains a 4x1 vector as the phase score representing the four different signal phases. The phase score of each phase can be directly used as the basis for selecting the next signal phase, for example, the higher the score, the better. The selected phase is used to generate the traffic control signal at the road intersection.

[0110] The phase score is also used as a reward for Q-Learning to update the Q value for RL learning. Figure 4 The network parameters in part (b).

[0111] Optionally, Figure 4 The network parameters of part (a) and part (b) of can be learned separately to reduce network complexity and speed up learning. For example, Figure 4 The network of part (a) can be from Algorithm 1 shown in Table 1. The network parameters obtained in the network training of Algorithm 1 are used for Figure 4 (a) of the network. And fixed Figure 4 The network parameters of part (a) are trained during the interaction with the traffic environment. Figure 4 (b) Part of the network.

[0112] The LiteLight model simplifies the PTS-XLight model using the FRAP model. At least in (1) the state of the traffic movement corresponding to the right-turn lane is deleted before the state is input, while FRAP uses the state of the right-turn lane as the model input. (2) In the model structure design, a fully connected (FC) layer is used in LiteLight to process the input states q and r, instead of using the convolution layer with a 1×1 convolution kernel in FRAP.

[0113] A large number of experiments are also conducted on real-world traffic datasets to evaluate the embodiments of the present application.

[0114] In the comparative experiments, datasets of real-world environments (available from https: / / traffic-signal-control.github.io) are used, including datasets of three cities: Jinan, Hangzhou, and New York. There are six real-world traffic datasets in total (three in the Jinan dataset (DJiNan1, DJiNan2, and DJiNan3 in Table 2), two in the Hangzhou dataset (DHangZhou1 and DHangZhou2 in Table 2), and one in the New York dataset (DNewYork). These datasets are generated in real time from surveillance cameras near intersections. As shown in Table 2, the average arrival rates of these traffic datasets are different.

[0115]

[0116] Table 2

[0117] Jinan dataset: This traffic network has 12 (3×4) intersections. Each intersection is set as a 4-way intersection with two 400m (east-west) long road segments and two 800m (south-north) long road segments. There are three traffic flow datasets under this traffic road network.

[0118] Hangzhou dataset: This traffic network has 16 (4×4) intersections. Each intersection is set as a 4-way intersection with two 800m (east-west) long road segments and two 600m (south-north) long road segments. There are two traffic flow datasets under this traffic road network.

[0119] New York dataset. This traffic network has 192 (28×7) intersections. Each intersection is four-lane, with two 300-meter (east-west) road segments and two 300-meter (south-north) road segments. There is only one traffic flow dataset under this traffic road network.

[0120] The TSC method according to the embodiment of the present application is compared with the baseline TSC method in the prior art. In order to make a clear comparison, these traffic signal control methods are divided into three groups: traditional methods, RL-based methods, and PTS-based methods of the embodiment of the present application. The methods used for comparison are listed below.

[0121] (1) Traditional methods

[0122] FixedTime[https: / / rosap.ntl.bts.gov / view / dot / 20661]: uses a fixed signal phase cycle length and a pre-set phase length allocation strategy.

[0123] MaxPressure(MP)[https: / / doi.org / 10.1016 / j.trc.2013.08.014]: A strategy for selecting the phase with the maximum pressure.

[0124] Efficient-MP[https: / / arxiv.org / abs / 2112.02336]: A strategy for selecting the phase with the maximum effective pressure.

[0125] (2) RL-based methods

[0126] FRAP[https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900]: uses an improved network structure to construct phase features and attempts to capture the phase competition relationship between different traffic flows.

[0127] MPLight[https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744]: uses FRAP as the base model and introduces pressure in the design of states and rewards.

[0128] CoLight[https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902]: Use graph attention network as the basic model to achieve cooperative control between intersections.

[0129] Efficient-MPLight[https: / / arxiv.org / abs / 2112.02336]: uses FRAP as the base model and adopts effective pressure in state design. It shows better performance than MPLight.

[0130] Efficient-CoLight[https: / / arxiv.org / abs / 2112.02336]: uses graph attention network as the base model and adopts efficient pressure in state design. It shows better performance than CoLight.

[0131] (3) PTS method based on the embodiment of the present application

[0132] PTS-FRAP: FRAP is used as the basic model and PTS as the state.

[0133] PTS-CoLight: CoLight is the basic model and PTS is the state.

[0134] LiteLight: uses a lightweight RL framework as the base model and PTS as the state.

[0135] Experiments were conducted on CityFlow3, which supports large-scale TSC. CityFlow is a platform for urban traffic simulation. CityFlow has been widely used in multiple RL-based methods, such as PressLight, CoLight, and MPLight [https: / / dl.acm.org / doi / abs / 10.1145 / 3292500.3330949, https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902].

[0136] The evaluation indicators used include average travel time and generalization.

[0137] Travel time is the difference in time (in seconds) between each vehicle entering and exiting a traffic area. Average travel time is the most commonly used indicator in recent TSC methods to measure the performance of traffic signal control methods. Reducing the average travel time is the goal of the TSC method, so the smaller the average travel time, the better the performance.

[0138] Generalization: The generalization performance of a model is represented by calculating the relationship between its migration results (travel time in seconds) Trans on a previously untrained dataset and its training results (travel time in seconds) Train on a previously trained dataset, which can be described as formula (6). The obtained Performance transfer The smaller the value, the better the generalization performance of the model.

[0139]

[0140] In order to make a fair comparison, the same hyperparameters (learning rate, playback buffer size, sample size, etc.) were set for all RL-related training processes in the experiment. Each round is a 60-minute simulation, and the average of the last ten rounds of testing is used as a result set. The final result set is the average of three independent results. The duration of the signal phase is set to t = 15 seconds, and the signal phase is set to 4 phases. Each green light is followed by three seconds of yellow light and two seconds of red light.

[0141] Table 3 shows the performance comparison of various TSC methods on the Jinan, Hangzhou and New York datasets. The values ​​in Table 3 represent the average driving time in seconds, with the performance of the Efficient-CoLight method as the benchmark because it has the most advanced performance in the past. The percentages in Table 3 are the percentages relative to the baseline performance, which are calculated by formula (7).

[0142]

[0143]

[0144] Table 3

[0145] As can be seen from Table 3, the performance of FixedTime methods is poor, as expected. However, the traditional methods MaxPressure and Efficient-MP even outperform some RL-based methods such as FRAP and MPLight. Influenced by PTS, the RL-based methods generated by PTS-XLight (PTS-FRAP, PTS-CoLight, LiteLight) consistently outperform other methods that do not use PTS. Every RL-based model used in the experiment can be improved by PTS compared to the original method. Although PTS-FRAP has a longer average driving time than Efficient-CoLight in the New York dataset, PTS-FRAP outperforms the original FRAP framework by nearly 22.71%. On the other hand, PTS-CoLight achieves the least driving time compared to the baseline method. Fewer observations actually lead to better results.

[0146] In addition, LiteLight is a lightweight model with no significant performance tradeoff compared to PTS-CoLight. It is also easier to train and deploy in industrial settings, while PTS-CoLight has more computational burden introduced by the attention mechanism [https: / / proceedings.neurips.cc / paper_files / paper / 2017 / hash / 3f5ee243547dee91fbd053c1c4a845a a-Abstract.html].

[0147] The New York dataset has 192 intersections, which is the largest dataset among the three datasets used in this study. To the best of our knowledge, only a few studies have conducted experiments using datasets larger than the New York dataset, which can produce more stable experimental results.

[0148] For simplicity, further experiments were conducted on the Jinan and Hangzhou datasets to reveal the advantages of speed performance and generalization performance when using the TSC model of the embodiments of the present application.

[0149] The generalization of the model is very important for it to adapt to different environments. An ideal RL model should be robust to different traffic conditions (such as small-scale traffic environments and large-scale traffic environments) after being trained in one traffic situation. The generalization (transferability) of the LiteLight, MPLight, and FRAP models was trained and tested on real datasets in Jinan and Hangzhou. These models were trained on one of the five datasets (3 datasets in Jinan and 2 datasets in Hangzhou) using the experimental environment and setting parameters described above, and the models trained by these methods were directly transferred to other datasets for testing.

[0150] Figure 5 The results of the migration test are shown.

[0151] Figure 5 In the figure, the vertical axis of each grid represents the dataset used for training (Train on), and the horizontal axis represents the dataset used for testing (Transfer on). In order to obtain more reliable transfer performance results, each transfer result (evaluated by driving time in seconds) is the average of three independent experiments. The smaller the value, the better the generalization performance.

[0152] like Figure 5 As shown, comparing the generalization capabilities of LiteLight, MPLight and FRAP models, the LiteLight migration result of the embodiment of the present application is almost ten times higher than that of other methods.

[0153] From the generalization results of the model, it can be inferred that in reinforcement learning-based methods, a small amount of observation data and a lightweight framework can help combat the overfitting of the Q function.

[0154] The speed of various models is also evaluated in the examples of this application.

[0155] Training a reinforcement learning-based model mainly requires four steps:

[0156] (1) Use ε-greedy to explore and generate samples;

[0157] (2) prepare and organize these samples;

[0158] (3) updating the parameters of the neural network in the RL model; and

[0159] (4) Test the updated model.

[0160] The parameter update (step (3)) step can be significantly affected by the model performance, so it is often used to evaluate the speed of the model.

[0161] Experiments were conducted on two datasets randomly selected from Jinan and Hangzhou (JiNan1 and HangZhou1), and the difference between the time of entering the parameter update step (step (3)) and exiting the step during model training was recorded as the update speed of the model. The final results were obtained from the average of the first 20 rounds.

[0162] Figure 6 The model performance of PTS-FRAP and LiteLight in the parameter update step during training is shown. It can be seen that during the RL model training process, LiteLight updates 8.7% faster than PTS-FRAP, which enables it to maintain faster adaptability in different heterogeneous scenarios.

[0163] When the RL model is actually deployed, the process of determining whether each vehicle is driving or waiting also requires a lot of computing resources. Therefore, in order to simplify the state input, the embodiment of this application only considers the total number of vehicles in the local range. For example, with the queue length as the reward, three state representations are designed, which show the different challenges of obtaining such traffic condition observation data in actual deployment:

[0164] State 1: The total number of vehicles entering the lane within the local range (165 meters) of the current intersection. It does not require information about adjacent intersections.

[0165] State 2: The total number of vehicles entering the lane within the local range (165 meters) of each intersection. State 2 needs to communicate with the simulation environment to obtain the status of adjacent intersections.

[0166] State 3: The total number of vehicles entering the lane at the current intersection, which is what CoLight and FRAP use.

[0167] In the experiment, CoLight and FRAP were used as the basic models and Figure 7 The results are shown in . From the experimental results, it can be seen that state 1 simplifies the state input into the state within the local range, has the best performance and the simplest state acquisition complexity, so it can be the most suitable state calculation method for industrial deployment.

[0168] Judging from the results of overall performance, model generalization, model training speed and model deployment, the TSC method using a small number of observation states and a lightweight RL model according to the embodiments of the present application can improve the performance by 11.46%, increase the generalization ability by up to 100%, increase the training speed by 8.70%, and improve the stability of deployment capabilities by 6.89%.

[0169] This application proposes a traffic state acquisition method using a small amount of observations called PTS (Partial State), which is used to obtain traffic states in traffic signal control (TSC) methods. In addition, a lightweight reinforcement learning method, namely LiteLight, is designed. Experimental results show that LiteLight according to an embodiment of the present application uses less traffic state information, achieves better performance in traffic signal control, and can enhance the industrial deployment capabilities of the TSC method. Therefore, it can be concluded that fewer observations and a lightweight framework can lead to better performance.

[0170] Although the present application is described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application.

[0171] Those skilled in the art to which these embodiments relate and who benefit from the teachings presented in the above description and associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that the present application is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used only in a general and descriptive sense and are not used for the purpose of limitation.

[0172] References

[0173] [https: / / digital-library.theiet.org / content / journals / 10.1049 / iet-its.2009.0070]Itamar Arel,C.Liu,T.Urbanik,and Airton Kohls.2010.Reinforcementlearning based multi-agent system for network traffic signalcontrol.Intelligent Transport Systems,IET 4(2010),128–135.

[0174] [https: / / ieeexplore.ieee.org / abstract / document / 9392372]AzzedineBoukerche,Dunhao Zhong,and Peng Sun.2021.A Novel Reinforcement

[0175] Learning-based Cooperative Traffic Signal System through Max-pressureControl.IEEE Transactions on Vehicular Technology(2021),1–1.

[0176] [https: / / proceedings.neurips.cc / paper / 2020 / hash / 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html]Tom Brown,Benjamin Mann,Nick Ryder,MelanieSubbiah,Jared D Kaplan,Prafulla Dhariwal,Arvind Neelakantan,Pranav Shyam,Girish Sastry,Amanda Askell,Sandhini Agarwal,Ariel Herbert-Voss,GretchenKrueger,Tom Henighan,Rewon Child,Aditya Ramesh,Daniel Ziegler,Jeffrey Wu,Clemens Winter,Chris Hesse,Mark Chen,Eric Sigler,Mateusz Litwin,Scott Gray,Benjamin Chess,Jack Clark,Christopher Berner,Sam McCandlish,Alec Radford,IlyaSutskever,and Dario Amodei.2020.Language Models are Few-Shot Learners,H.Larochelle,M.Ranzato,R.Hadsell,M.F.Balcan,and H.Lin(Eds.).Advances inNeural Information Processing Systems33(NeurIPS)33,1877–1901

[0177] [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5744]Chacha Chen,Hua Wei,Nan Xu,Guanjie Zheng,Ming Yang,Yuanhao Xiong,Kai Xu,and ZhenhuiLi.2020.Toward a thousand lights:Decentralized deep

[0178] reinforcement learning for large-scale traffic signal control.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.34.3414–3421

[0179] [https: / / link.springer.com / chapter / 10.1007 / 978-1-4471-5113-5_3]Seung-Bae Cools,Carlo s Gershenson,and Bart D’Hooghe.2013.Self-organizing

[0180] traffic lights:A realistic simulation.In Advances in applied Self-organizing systems.Springer,45–55

[0181] [https: / / arxiv.org / abs / 1810.04805]Jacob Devlin,Ming-Wei Chang,KentonLee,and Kristina Toutanova.2018.BERT:Pre-training of Deep BidirectionalTransformers for Language Understanding.

[0182] [https: / / books.google.com.hk / books?hl=en&lr=&id=dHrsHz0VocUC&oi=fnd&pg=PA1&dq=Optimizat ion+in+Economic+Theory+(2nd+ed.)&ots=-0O9_P7jsn&sig=NUgkP7OYNjaAMl47e3NZhiBo2kk&redir_esc=y#v=onepage&q=Optimization%20in%20Econo mic%20Theory%20(2nd%20ed.)&f=false]AvinashK.Dixit.1990.Optimization in Economic Theory(2nd ed.).Oxford University Press(1990),164

[0183] [https: / / arxiv.org / abs / nlin / 0411066]Carlos Gershenson.2004.Self-organizing traffic lights.arXiv preprint nlin / 0411066(2004)

[0184] [https: / / trid.trb.org / view / 186640]PB Hunt,DI Robertson,RD Bretherton,and M Cr Royle.1982.The SCOOT on-line traffic signal optimisationtechnique.Traffic Engineering&Control23,4(1982)

[0185] [https: / / rosap.ntl.bts.gov / view / dot / 20661]Peter Koonce and LeeRodegerdts.2008.Traffic signal timing manual.Technical Report.UnitedStates.Federal Highway Administration.

[0186] [https: / / www.sciencedirect.com / science / article / pii / S0968090X20307324]Michael W Levin,Jeffrey Hu,and Michael Odell.2020.Max-pressure signal controlwith cyclical phase structure.Transportation Research Part C:EmergingTechnologies 120(2020),102828

[0187] [https: / / www.semanticscholar.org / paper / SCATS:-Sydney-Co-Ordinated-Adaptive-Traffic-System:-Lowrie / 93a56f13a698dcab2df2f7a88229ba15091b3004#citing-papers]PR Lowrie.1992.SCATS:A traffic responsive method of controllingurban traffic control.Roads and Traffic Authority(1992).

[0188] [https: / / arxiv.org / abs / 1312.5602]Volodymyr Mnih,Koray Kavukcuoglu,David Silver,Alex Graves,Ioannis Antonoglou,Daan Wierstra,and MartinRiedmiller.2013.Playing Atari with Deep Reinforcement Learning.(2013).http: / / arxiv.org / abs / 1312.5602 cite arxiv:1312.5602Comment:NIPS Deep LearningWorkshop 2013.

[0189] [https: / / ieeexplore.ieee.org / abstract / document / 8569301]Tomoki Nishi,Keisuke Otaki,Keiichiro Hayakawa,and Takayoshi Yoshimura.2018.Traffic signalcontrol based on reinforcement learning with graph convolutional neuralnets.In 2018 21st International conference on intelligent transportationsystems(ITSC).IEEE,877–883

[0190] [https: / / doi.org / 10.1145 / 37402.37406]Craig W.Reynolds.1987.Flocks,Herds and Schools:A Distributed Behavioral Model.SIGGRAPH Comput.Graph.21,4(1987),25–34.

[0191] [https: / / ieeexplore.ieee.org / book / 6267343]A.G.Sutton,R.S.;Barto.2017.Reinforcement Learning:An Introduction.Cambridge,MA:The MIT Press.(2017)

[0192] [https: / / doi.org / 10.1109 / TCYB.2019.2904742]Tian Tan,Feng Bao,YueDeng,Alex Jin,Qionghai Dai,and Jie Wang.2020.Cooperative Deep ReinforcementLearning for Large-Scale Traffic Grid Signal Control.IEEE Transactions onCybernetics 50,6(2020),2687–2700.

[0193] [https: / / www.sciencedirect.com / science / article / abs / pii / 0378437196001446]János and János Kertész.1996.The green wave model oftwo-dimensional traffic:Transitions in the flow properties and in thegeometry of the traffic jam.Physica A:Statistical Mechanics and itsApplications 231,4(1996),515–533

[0194] [https: / / doi.org / 10.1016 / j.trc.2013.08.014]Pravin Varaiya.2013.Maxpressure control of a network of signalized intersections.TransportationResearch Part C:Emerging Technologies36(11 2013),177–195.

[0195] [https: / / proceedings.neurips.cc / paper_files / paper / 2017 / hash / 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html]Ashish Vaswani,Noam Shazeer,NikiParmar,Jakob Uszkoreit,Llion Jones,Aidan N Gomez, ukasz Kaiser,and IlliaPolosukhin.2017.Attention is All you Need.In Advances in Neural InformationProcessing Systems(NIPS),Vol.30.

[0196] [https: / / arxiv.org / abs / 1710.10903]Petar Guillem Cucurull,Arantxa Casanova,Adriana Romero,Pietro Liò,and Yoshua Bengio.2017.GraphAttention Networks.6th International Conference on Learning Representations(ICLR)(2017).

[0197] [https: / / link.springer.com / article / 10.1007 / BF00992698]C.J.Watkins andP.Dayan.1992.Q-learning.Mach.Learn.3-4(1992)

[0198] [https: / / dl.acm.org / doi / abs / 10.1145 / 3292500.3330949]Hua Wei,ChachaChen,Guanjie Zheng,Kan Wu,Vikash Gayah,Kai Xu,and Zhenhui Li.2019.Presslight:Learning max pressure control to coordinate traffic signals in arterialnetwork.In Proceedings of the 25th ACM SIGKDD International Conference onKnowledge Discovery&Data Mining.1290–1298

[0199] [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357902]Hua Wei,Nan Xu,Huichu Zhang,Guanjie Zheng,Xinshi Zang,Chacha Chen,Weinan Zhang,Yanmin Zhu,Kai Xu,and Zhenhui Li.2019.Colight:Learning network-level cooperation fortraffic signal control.In Proceedings of the 28th ACM InternationalConference on Information and Knowledge Management.1913–1922.

[0200] [https: / / dl.acm.org / doi / abs / 10.1145 / 3219819.3220096]Hua Wei,GuanjieZheng,Huaxiu Yao,and Zhenhui Li.2018.IntelliLight:A Reinforcement LearningApproach for Intelligent Traffic Light Control.In Proceedings of the 24th ACMSIGKDD International Conference on Knowledge Discovery&Data Mining(London,United Kingdom)(KDD’18).2496–2505.

[0201] [https: / / www.semanticscholar.org / paper / Intelligent-Traffic-Light-Control-Wiering-Veenen / 02053b02ca9d98e57319f7376aead1b1a064d6ec]M.Wiering,J.V.Veenen,J.Vreeken,and A.Koopman.2004.Intelligent traffic lightcontrol.Utrecht Univ.(2004).

[0202] [https: / / arxiv.org / abs / 2112.02336]Qiang Wu,Liang Zhang,Jun Shen,Linyuan Lü,Bo Du,and Jianqing Wu.2021.Efficient Pressure:Improving efficiencyfor signalized intersections.arXiv:2112.02336[cs.LG]

[0203] [https: / / ojs.aaai.org / index.php / AAAI / article / view / 16147]B.Xu,Y.Wang,Z.Wang,H.Jia,and Z.Lu.2021.Hierarchically and Cooperatively Learning TrafficSignal Control.Association for the Advancement of Artificial Intelligenc(AAAI)(2021),669–677

[0204] [https: / / ojs.aaai.org / index.php / AAAI / article / view / 5467]Xinshi Zang,Huaxiu Yao,Guanjie Zheng,Nan Xu,Kai Xu,and Zhenhui Li.MetaLight:Value-BasedMeta-Reinforcement Learning for Traffic Signal Control.Proceedings of theAAAI Conference on Artificial Intelligence 34,01(2020),1153–1160.

[0205] [https: / / dl.acm.org / doi / abs / 10.1145 / 3357384.3357900] Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. 2019. Learning phase competition for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1963–1972.

Claims

1. A road intersection traffic signal control method based on a reinforcement learning model, comprising: At a specified time interval, a local state PTS of the road intersection is obtained; wherein the local state PTS of the road intersection includes the local state PTSs of all traffic signal phases of the road intersection, each traffic signal phase corresponds to a traffic motion i and a traffic motion j without conflict, and the local state PTS of each traffic signal phase includes the local state PTS of the corresponding traffic motion i and the local state PTS of the traffic motion j; wherein, according to the local queue lengths f of the traffic motion i and the traffic motion j, respectively i v and and the number of local vehicles f for traffic movement i and traffic movement j i r and Generate a local state PTS of the traffic signal phase to which traffic movement i and traffic movement j belong; wherein the local queue length is the number of vehicles queued in a traffic movement within the local range L of the road intersection, denoted as q L (l,m), where l and m represent the entry and exit roads of a traffic movement; the local number of vehicles is the number of vehicles in a traffic movement within the local range L of the road intersection, denoted by r L (l,m), the local range L is the maximum distance that a vehicle passing through the intersection can pass within a specified time; f is respectively i v , f i r and Encoding, the result is recorded as and connect The local state PTS of the traffic signal phases of the traffic motion i and the traffic motion j is obtained by combining the traffic signal phases of the traffic motion i and the traffic motion j; The reinforcement learning model generates a next traffic signal phase for the road intersection according to the local state PTS of the road intersection; and A traffic signal at the road intersection is set using the next traffic signal phase.

2. The method for controlling traffic signals at road intersections according to claim 1, wherein: The reinforcement learning model includes a FRAP model and a CoLight model.

3. The method for controlling traffic signals at road intersections according to claim 1 or 2, wherein: The reinforcement learning model uses the local state PTS of the road intersection as the traffic state to generate the next traffic signal phase for the road intersection; The fully connected neural network is a one-layer fully connected neural network.

4. The method for controlling traffic signals at road intersections according to claim 3, wherein: The phase of the traffic signal phase to which traffic movement i and traffic movement j belong is represented by f i s ; Through the fully connected neural network i s Encoding, the result is recorded as Through the fully connected neural network and Encoding, the result is recorded as d i ; Through the fully connected neural network and Encoding, the result is recorded as d j ,in Combination d i With d j The local state PTS of the traffic signal phase to which traffic movement i and traffic movement j belong is obtained.

5. The method for controlling traffic signals at road intersections according to claims 1-4, wherein: Using the local state PTS of the road intersection as the traffic state of the reinforcement learning model; generating a next traffic signal phase for the road intersection as an action of the reinforcement learning model; After setting the traffic signal of the road intersection with the next traffic signal phase, the total queue length of the road intersection after a specified time interval is used as the reward r of the reinforcement learning model, where L in is the set of all incoming lanes at the road intersection, Entering the driveway The queue length within the local range L of the reinforcement learning model is generated according to the traffic state, action and reward r of the reinforcement learning model to train the reinforcement learning model.

6. A road intersection traffic signal control method based on a reinforcement learning model, including: Obtain the local queue length and local number of vehicles traveling for each traffic movement at the road intersection; The local queue length is the number of vehicles queued in a traffic movement within the local range L of the road intersection, denoted as q L (l,m), where l and m represent the entry and exit roads of a traffic movement; the local number of vehicles is the number of vehicles in a traffic movement within the local range L of the road intersection, denoted by r L (l,m), the local range L is the maximum distance that a vehicle passing through the intersection can pass within a specified time; Generating an input vector q according to the local queue lengths of each traffic movement at the road intersection, wherein the number of rows of the vector q is 2 and the number of columns is equal to the number of traffic movement groups, wherein each traffic movement group includes 2 traffic movements that do not conflict with one of the traffic signal phases; Generate an input vector r according to the number of local vehicles in each traffic movement, where the number of rows of vector r is 2 and the number of columns is equal to the number of traffic movement groups; Through a layer of fully connected neural network, q and r are encoded respectively, and the results are summed according to the column vector to obtain V including two vectors. pts ; generating a plurality of input phase representations according to each traffic signal phase, wherein each input phase representation represents an allowed traffic signal phase and a plurality of other prohibited traffic signal phases; Generate a phase embedding vector V based on each input phase representation phase ; Embed each phase into a vector V phase Respectively with V pts Connect to obtain the state embedding vector H corresponding to each traffic signal phase k ; Embedding vector H for multiple states k Reorganize and generate a competition result vector through a layer of fully connected neural network, each row of the competition result vector corresponds to one of the traffic signal phases; Summing the competition result vector by column to obtain a phase score corresponding to each traffic signal phase; The reinforcement learning model generates a next traffic signal phase for the road intersection according to the phase scores of each traffic signal phase at the road intersection; and A traffic signal at the road intersection is set using the next traffic signal phase.

7. The method according to claim 6, wherein: The number of vehicles queued in a traffic movement within the local area L of the road intersection For any other entry lanes in a traffic movement TM except the right turn lane, The sum of the queue lengths in the local area L of all entry lanes excluding the right-turn lane representing the traffic movement TM; The number of vehicles traveling in a traffic movement within the local area L of the road intersection The sum of the number of vehicles traveling in the local range L of all entry lanes excluding the right-turn lane representing the traffic movement TM.

8. The method according to claim 7, further comprising: Embed the state into vector H k As the traffic state of the reinforcement learning model, the next traffic signal phase for the road intersection generated according to the phase score and the corresponding phase score are used as the action of the reinforcement model and the corresponding reward respectively; The reinforcement learning model is trained by generating training samples according to the traffic status, actions and rewards of the reinforcement learning model.

9. A traffic signal control method for a traffic system including a plurality of road intersections, comprising: At one, more than one or each of the plurality of road intersections, a traffic signal at the road intersection is set using the road intersection traffic signal control method based on a reinforcement learning model according to one of claims 1-8.

10. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images