Traffic signal phase and duration cooperative control method based on double residual reinforcement learning

Abstract

The invention provides a traffic signal phase and duration cooperative control method based on double residual reinforcement learning. The method comprises the following steps: acquiring a traffic state vector of an intersection, wherein the traffic state vector comprises a current signal phase, the number of vehicles on each entrance lane and the number of vehicles on each lane section; a basic phase module and a basic duration module respectively generate an initial phase action value vector and an initial duration action value vector based on the traffic state vector; a dual residual error correction module generates a phase residual error value vector and a duration residual error value vector respectively based on the initial phase action value vector, the initial duration action value vector and the traffic state vector; generating a final phase action value vector and a final duration action value vector based on the initial phase action value vector, the initial duration action value vector, the phase residual value vector and the duration residual value vector; and respectively selecting a phase action and a duration action based on the final phase action value vector and the final duration action value vector so as to form a traffic signal control instruction.

Description

Technical Field

[0001] This disclosure belongs to the field of traffic signal control technology, specifically relating to a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning. Background Technology

[0002] With the rapid development of intelligent transportation systems, deep reinforcement learning, due to its powerful perception and decision-making capabilities, has become an important research direction in the field of adaptive traffic signal control. Existing technologies have explored various approaches to improve control efficiency and collaborative capabilities. For example, some methods employ graph convolutional networks and attention mechanisms to optimize intersection feature extraction, or utilize transfer learning strategies to improve the training efficiency of models at new intersections; other studies use Transformer sequence models or Bayesian deep Q-networks to address multi-intersection collaborative control problems, aiming to improve network-level traffic efficiency; still other solutions focus on combining traffic flow prediction or improving reward function design, aiming to enhance the system's adaptability to dynamic traffic environments.

[0003] However, these methods still have significant limitations. When signal control involves two closely coupled decision dimensions—phase selection and duration adjustment—most existing methods either treat them separately or perform simple joint decisions, failing to deeply model their inherent synergistic relationship. This makes it difficult for phase and duration strategies to work effectively together, the effect of coordinated control cannot be accurately quantified, and consequently, the joint optimization process of multiple strategies becomes unstable and convergence difficult, ultimately hindering further improvements in control performance under complex traffic scenarios. Therefore, how to achieve efficient and stable coordinated optimization of phase and duration strategies has become an urgent technical challenge to be solved. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, this disclosure aims to provide a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning.

[0005] To achieve one or more of the above objectives, this disclosure provides the following aspects.

[0006] According to one aspect of this disclosure, a traffic signal phase and duration coordinated control method based on dual residual reinforcement learning is provided, comprising: acquiring a traffic state vector of an intersection including the current signal phase, the number of vehicles in each approach lane, and the number of vehicles in each lane segment; generating an initial phase action value vector and an initial duration action value vector respectively by a basic phase module and a basic duration module based on the traffic state vector; generating a phase residual value vector and a duration residual value vector respectively by a dual residual correction module based on the initial phase action value vector, the initial duration action value vector, and the traffic state vector; generating a final phase action value vector and a final duration action value vector based on the initial phase action value vector, the initial duration action value vector, the phase residual value vector, and the duration residual value vector; and selecting a phase action and a duration action based on the final phase action value vector and the final duration action value vector to form a traffic signal control command.

[0007] According to an exemplary embodiment, the basic phase module and the basic duration module generate an initial phase action value vector and an initial duration action value vector based on the traffic state vector, respectively: the traffic state vector is embedded with features based on a multi-head self-attention mechanism to generate an initial feature representation; the initial feature representation is nonlinearly transformed based on a multilayer perceptron to generate a high-dimensional feature representation; and the high-dimensional feature representation is processed by a phase linear mapping layer and a duration linear mapping layer, respectively, to generate the initial phase action value vector and the initial duration action value vector.

[0008] According to an exemplary embodiment, the dual residual correction module generates phase residual value vectors and duration residual value vectors based on initial phase action value vectors, initial duration action value vectors, and traffic state vectors, respectively, including: concatenating feature vectors obtained by feature mapping of the initial phase action value vectors and initial duration action value vectors based on a multilayer perceptron with feature vectors obtained by feature embedding of the traffic state vectors based on a multi-head self-attention mechanism; extracting features from the concatenated feature vectors based on a multilayer perceptron to generate fused feature vectors; and mapping the fused feature vectors to the action spaces of phase and duration respectively through a multilayer perceptron, thereby generating phase residual value vectors and duration residual value vectors.

[0009] According to an exemplary embodiment, in selecting phase actions and duration actions to form traffic signal control instructions, an ε-greedy strategy is adopted when selecting phase actions and duration actions based on the final phase action value vector and the final duration action value vector: phase actions and duration actions are uniformly and randomly selected from the action space with probability ε, and the phase action and duration action with the highest current value are selected with probability 1-ε.

[0010] According to an exemplary embodiment, the method further includes: obtaining the individual phase reward and the individual duration reward before correction, wherein the individual phase reward and the individual duration reward before correction are the individual phase reward and the individual duration reward obtained based on Shapley value theory when the dual residual correction module is not introduced; obtaining the individual phase reward and the individual duration reward after correction, wherein the individual phase reward and the individual duration reward after correction are the individual phase reward and the individual duration reward obtained based on Shapley value theory after the dual residual correction module is introduced; and using the difference between the individual phase reward after correction and the individual phase reward before correction, and the difference between the individual duration reward after correction and the individual duration reward before correction, as residual rewards for updating the dual residual correction module.

[0011] According to an exemplary embodiment, the method further includes: obtaining a first environmental reward value when only a phase action is performed and the duration is fixed at a preset default value; obtaining a second environmental reward value when only a duration action is performed and the phase is fixed at a preset default value; obtaining a third environmental reward value when both a phase action and a duration action are performed simultaneously; and calculating phase individual rewards and duration individual rewards based on the first to third environmental reward values ​​using Shapley value theory.

[0012] According to an exemplary embodiment, the residual reward is used to update the dual residual correction module using the following loss function:

[0013]

[0014] in, Q represents the residual reward. x (s t ,a x ) represents the final phase or duration action value vector, γ represents the discount factor, and D x This represents the dataset acquired during the phase of updating the dual residual correction module, and E represents the expectation for the samples.

[0015] According to an exemplary embodiment, the method further includes a training process that employs a progressive optimization strategy, including: a first stage, a fixed-duration control strategy for training the basic phase module separately; and a fixed-phase control strategy for training the basic duration module separately; and a second stage, after completing the first stage of training, introducing and jointly training a dual residual correction module.

[0016] The technical solutions based on the exemplary embodiments of this disclosure can achieve the following beneficial effects:

[0017] This disclosure proposes a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning. By simultaneously modeling two closely coupled control variables, traffic signal phase and duration, within a reinforcement learning framework, joint optimization control of traffic signals is achieved, which can significantly improve the coordination of signal timing and overall traffic efficiency in complex traffic environments.

[0018] This method first uses a multi-head self-attention mechanism to embed features and model traffic state information globally, fully capturing the spatial relationships between different lanes and providing a more representative high-dimensional feature representation for subsequent decision-making. On this basis, the constructed dual residual correction module can dynamically correct phase and duration action values ​​respectively, capturing the mutual influence and synergistic characteristics between the two, thereby improving the robustness and stability of the control strategy.

[0019] Furthermore, this disclosure introduces an individual residual reward mechanism based on Shapley value theory, which achieves fair quantification and differential update of the contributions of the phase module and the duration module. This not only ensures that the performance after collaborative correction is better than the basic control, but also effectively enhances the interpretability and training stability of the model.

[0020] In addition, the method employs an ε-greedy strategy to balance exploration and exploitation, avoiding the strategy from getting trapped in local optima too early, and uses a progressive optimization strategy to divide the complex joint training process into stages, thereby reducing training instability and improving convergence speed.

[0021] Through the synergistic effect of the above-mentioned technical features, this disclosure can achieve adaptive optimization of signal control in complex traffic environments with multiple intersections, and has significant advantages and wide application value in improving traffic flow efficiency, reducing vehicle delays, and improving intersection coordination.

[0022] The above-disclosed technical solutions only need to achieve one of the aforementioned effects, and it is not required that each technical solution achieve all of the aforementioned technical effects.

[0023] Furthermore, the effects of this disclosure include not only those set forth herein, but also other effects that will be apparent to those skilled in the art upon reference to the claims, the specification, and the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a schematic diagram of a traffic intersection controlled by traffic lights according to an exemplary embodiment of the present disclosure.

[0026] Figure 2 This is a schematic diagram of a four-phase configuration according to an exemplary embodiment of the present disclosure.

[0027] Figure 3 This is a flowchart of a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning according to an exemplary embodiment of the present disclosure.

[0028] Figure 4 This is a schematic diagram of a traffic signal phase and duration collaborative control architecture based on dual residual reinforcement learning according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that these embodiments are merely examples and not exhaustive. The components shown in the drawings can be arranged and designed in various ways. Furthermore, the technical features of different embodiments can be combined with each other without conflict.

[0030] Please note that similar reference numerals in the figures represent similar elements or steps, and therefore their definitions will not be repeated after their first appearance. Terms such as "first" and "second" used in this specification are for distinction only and do not imply any order or importance.

[0031] This disclosure presents a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning. Addressing the problems of difficulty in coordinating phase and duration strategies, difficulty in quantifying the collaborative effect, and instability in multi-strategy joint optimization in existing reinforcement learning methods for traffic signal control, this method proposes a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning. This method first constructs two basic decision modules for phase and duration, generating initial phase and duration actions respectively. Based on this, a dual residual correction module is designed, capturing the coordination relationship and mutual influence between phase and duration through a residual learning mechanism, and collaboratively correcting the initial actions to improve decision accuracy and coordination. This disclosure further proposes an individual residual reward mechanism, utilizing Shapley values ​​from game theory to decompose the global reward according to individual contributions. By comparing the difference in rewards before and after correction, the coordination correction effect is quantified, effectively improving the stability of multi-strategy collaborative optimization. In terms of model training, this disclosure adopts a phased progressive optimization strategy, first training the basic decision modules independently, and then gradually introducing residual correction and reward decomposition, effectively improving the instability problem in the joint optimization process and improving overall performance. Through the above technical solutions, this disclosure can achieve efficient coordinated control of traffic signal phase and duration, and significantly improve traffic operation efficiency and signal control stability in multi-intersection and complex traffic environments.

[0032] The traffic intersections disclosed herein are as follows Figure 1 As shown, the road includes 4 entrance sections and 4 exit sections. Each section has 3 lanes for left turns, straight ahead, and right turns, for a total of 12 entrance lanes and 12 exit lanes. Lane access is controlled by traffic lights: green light allows passage, red light prohibits passage. Figure 1 The content shown is merely an example, and this disclosure is not limited thereto.

[0033] This disclosure defines the following terms:

[0034] Phase: refers to a combination of one or more traffic flows that simultaneously gain the right-of-way without traffic conflict (excluding right-turn lanes). Although Figure 2 The phase set shown includes four phases, but this disclosure is not limited thereto.

[0035] Duration: Duration represents the time interval allocated to a specific signal phase, determining the time during which vehicles are allowed or prohibited from passing under that phase. This disclosure defines the duration value set as {10, 15, 20, 25, 30, 35, 40} seconds, and the appropriate duration can be dynamically selected according to real-time traffic conditions to achieve flexible control.

[0036] Number of vehicles in the entrance lanes: refers to the total number of vehicles in each entrance lane at the current moment.

[0037] Lane segment vehicle count: Divide the entrance lane into several lane segments, for example, every 100 meters, and count the number of vehicles in each lane segment.

[0038] This disclosure models phase-duration joint control as a multi-agent cooperative reinforcement learning problem, setting phase agent and duration agent to make decisions on signal phase and duration, respectively. At each time step, the two agents select their respective control actions (i.e., phase and duration) based on the current traffic state, evaluate the effects of the actions through environmental feedback, and optimize their respective control strategies to maximize cumulative rewards. Formally, this process can be represented as a Markov decision process.<S,A,r,P,γ> .

[0039] State space S: At time step t, state s t This includes the current phase, the number of vehicles in each approach lane, and the number of vehicles in each lane segment.

[0040] Action space A: For an intersection containing P phases, the phase action set is defined as A. p ={a1,…,a P Meanwhile, define D discrete values ​​to form a duration action set A. d ={a1,…,a D}

[0041] Reward r: At time step t, take the queue length of all entrance lanes as the reward r. t The cumulative return R is calculated after T time steps.

[0042] State transition P(s) t+1 |s t ,a t ): Indicates that action a is being performed. t The transition probability of the system state after the transition.

[0043] Discount factor γ∈(0,1): used to weigh the importance of current rewards against future rewards.

[0044] Through the above modeling, this disclosure can achieve coordinated optimization control of traffic signal phase and duration, improve intersection traffic efficiency and reduce vehicle delays.

[0045] Road Network Traffic Environment: Based on the real road network topology and vehicle travel path data of the target area, this disclosure utilizes the SUMO traffic simulation platform to construct a road network traffic environment highly similar to actual traffic conditions. Through SUMO's TraCI interface and sumolib tool, traffic operation information for each road segment and lane segment during the simulation can be extracted in real time. Furthermore, the signal control strategy generated by the model can be transmitted to the traffic light system in the simulation, dynamically adjusting the phase and duration configuration of each intersection. In this environment, the evolution of traffic flow and the execution of control strategies occur simultaneously. The system continuously records the traffic state data of each signalized intersection in the road network and feeds it back to the control model for further decision-making and optimization. The road network traffic environment constructed in this way not only realistically reflects actual traffic operation patterns but also provides reliable data support and test scenarios for the training and verification of signal control strategies.

[0046] Basic decision-making module: Based on the traffic state s at the current time t t The values ​​of the two initial control actions are calculated. (Phase action value) Evaluate the potential benefits of selecting different signal phases. Duration-based action value. The potential benefits of selecting different signal durations (e.g., green light duration of 15 seconds, 30 seconds, etc.) are evaluated. Feature extraction unit: Using a multi-head self-attention mechanism (MHA) and a multilayer perceptron (MLP), high-level features are extracted from complex traffic conditions, with a particular focus on capturing the interactions between different lanes. Numerical prediction unit: Using a linear mapping layer, the extracted high-level features are mapped to action values ​​for phase and duration, respectively.

[0047] According to one aspect of this disclosure, a traffic signal phase and duration coordinated control method based on dual residual reinforcement learning is provided, comprising: acquiring a traffic state vector of an intersection including the current signal phase, the number of vehicles in each approach lane, and the number of vehicles in each lane segment (S1); generating an initial phase action value vector and an initial duration action value vector by a basic phase module and a basic duration module based on the traffic state vector, respectively (S2); generating a phase residual value vector and a duration residual value vector by a dual residual correction module based on the initial phase action value vector, the initial duration action value vector, and the traffic state vector, respectively (S3); generating a final phase action value vector and a final duration action value vector based on the initial phase action value vector, the initial duration action value vector, the phase residual value vector, and the duration residual value vector (S4); and selecting a phase action and a duration action based on the final phase action value vector and the final duration action value vector to form a traffic signal control command (S5).

[0048] like Figure 4 As shown, the basic decision-making module of this disclosure includes a phase module and a duration module, both of which consist of a feature extraction unit and a numerical prediction unit. The feature extraction unit is used to capture the correlation of traffic states between different lanes. By introducing a multi-head self-attention mechanism to embed features into the input state, it can perceive the interaction relationship of traffic flows in each lane in the spatial dimension and obtain a high-dimensional representation vector. The numerical prediction unit then uses a linear mapping layer to process the embedded features, outputting the initial action values ​​for phase and duration, respectively.

[0049] According to an exemplary embodiment, the basic phase module and the basic duration module generate an initial phase action value vector and an initial duration action value vector based on the traffic state vector, respectively: the traffic state vector is embedded with features based on a multi-head self-attention mechanism to generate an initial feature representation; the initial feature representation is nonlinearly transformed based on a multilayer perceptron to generate a high-dimensional feature representation; and the high-dimensional feature representation is processed by a phase linear mapping layer and a duration linear mapping layer, respectively, to generate the initial phase action value vector and the initial duration action value vector.

[0050] For example, in step S1, at time t, the traffic state vector s is input. t (Including current signal phase, number of vehicles in approach lanes, and number of vehicles in each lane segment). Feature extraction is performed using a multi-head self-attention mechanism (MHA) to understand the spatial correlation between traffic flows (i.e., how congestion in one lane affects other lanes). The calculation process is as follows:

[0051] Create a query (Q), key (K), and value (V): Q = W Q s t K = W K s tV = W V s t

[0052] The traffic state vector s t Each is associated with three learnable weight matrices W. Q W K W V Multiplying them produces three vectors: Query, Key, and Value.

[0053] Calculate single-head attention:

[0054]

[0055] QK T Calculate the similarity (dot product) between the query and all keys. A higher similarity indicates a more relevant information corresponding to that key. Divide by... The dimension of the key vector is to prevent the gradient vanishing problem caused by an excessively large dot product. Then, the similarity score is normalized into a probability distribution (the sum of the weights is 1) using the Softmax function. This distribution is the "attention weight", which indicates which parts of the input information should be "focused" on when generating new features.

[0056] Weighted summation: The obtained attention weights are multiplied by the values ​​(V) and summed. This assigns higher weights to information more relevant to the current query, resulting in greater representation in the output. Each attention head... i The output of each feature is a new feature representation that incorporates global information.

[0057] Merging multiple heads: Concatenate the outputs of multiple (h) attention heads and then pass them through a learnable linear projection matrix W. O To perform fusion and dimensionality reduction.

[0058] MHA(s t )=Concat(head1,head2,…,head h W O

[0059] Among them, W Q W K W V For learnable parameters, head i is the output of the i-th attention head, h is the number of attention heads, and Softmax(·) represents the exponential operation and normalization of each element in the input vector so that each output element is non-negative and the sum is 1, which is used to represent the relative importance or attention weight of each element.

[0060] Different attention heads can learn to focus on different types of relationships. For example, one head can focus on conflicting relationships between traffic flows in opposite directions, while another head can focus on cooperative relationships between adjacent traffic flows. This multi-head mechanism allows the model to understand traffic conditions from different perspectives in parallel, thereby obtaining richer and more robust feature representations z. t This step yields the embedded feature representation z. t =MHA(s) t ).

[0061] Multilayer perceptron (MLP) outputs features z from a self-attention mechanism t The input is a fully connected neural network (MLP), which typically contains hidden layers. The resulting embedded feature representation z... t High-dimensional features are further extracted using a multilayer perceptron (MLP):

[0062]

[0063] Where W1, W2, b1, and b2 are learnable parameters, and σ(·) is a nonlinear activation function.

[0064] First layer: W1z t +b1 undergoes a linear transformation, then passes through a non-linear activation function σ(·) (such as ReLU). This introduces non-linearity, enabling the model to learn more complex patterns. Second layer: W2 (first layer output) +b2 undergoes another linear transformation, and can again pass through an activation function. Based on the interaction relationships captured by the self-attention mechanism, further non-linear feature combinations and transformations are performed to obtain the final, highly abstract feature vector used for decision-making.

[0065] Extracted high-dimensional features The inputs are fed into two linear mapping layers, and the outputs are the initial action values ​​of phase and duration:

[0066]

[0067] This is a vector of length p (the total number of optional phases). Each value in the vector... This represents the state s t The estimated long-term return from choosing the i-th phase is given. A higher value indicates that the phase is likely to be better in the current situation.

[0068]

[0069] This is a vector of length D (the total number of optional duration levels). Each value This represents the estimated long-term return for choosing the j-th duration.

[0070] Among them, Wp W d b p b d Here, P represents the number of phase actions, and D represents the number of duration actions.

[0071] Based on the obtained action value and The basic phase module and the basic duration module select the initial control action by maximizing the value of the corresponding action:

[0072]

[0073] For both phase and duration, the basic modules employ a greedy strategy to select the initial action. That is, they directly choose the action with the highest value. For example, for the initial phase action... choose The phase corresponding to the maximum value in the vector, and for the initial duration action. choose The duration corresponding to the maximum value in the vector. In the following description of this disclosure, x is used to refer to both the phase (p) and duration (d) control components.

[0074] The initial action value obtained and As input to the dual residual correction module.

[0075] Through the above design, this disclosure can output the initial actions of phase and duration respectively within the reinforcement learning framework, providing basic decision support for the subsequent residual correction module.

[0076] The dual residual correction module disclosed herein includes a phase-duration co-modeling unit and two types of residual numerical units. Its core function is to further model the co-relationship between phase and duration based on the initial action value output by the basic decision module, and to perform residual correction on the action value, thereby obtaining a more accurate and stable control signal.

[0077] According to an exemplary embodiment, the dual residual correction module generates phase residual value vectors and duration residual value vectors based on initial phase action value vectors, initial duration action value vectors, and traffic state vectors, respectively, including: concatenating feature vectors obtained by feature mapping of the initial phase action value vectors and initial duration action value vectors based on a multilayer perceptron with feature vectors obtained by feature embedding of the traffic state vectors based on a multi-head self-attention mechanism; extracting features from the concatenated feature vectors based on a multilayer perceptron to generate fused feature vectors; and mapping the fused feature vectors to the action spaces of phase and duration respectively through a multilayer perceptron, thereby generating phase residual value vectors and duration residual value vectors.

[0078] The phase-duration co-modeling unit comprehensively analyzes various combinations of phase actions and duration actions, and devalues ​​the basic actions. and The input is processed by a multilayer perceptron (MLP) for feature mapping, and combined with a multi-head self-attention mechanism to process the original traffic state s. t Feature extraction is performed to obtain the fused feature vector h. t :

[0079]

[0080] The phase value vector given by the basic module can be analyzed separately using a neural network to extract its advanced features. Similarly, the duration-value vector is analyzed separately to extract its features.

[0081] MHA(s t The multi-head self-attention mechanism is used again to analyze the original traffic state s. t Feature extraction is performed.

[0082] The symbol || indicates concatenation. The three feature vectors mentioned above are concatenated into a longer, more comprehensive fused feature vector. This integrates information from the "preliminary phase judgment," "preliminary duration judgment," and "actual situation on site." The concatenated long vector is then fed into the final MLP for dimensionality reduction and deep fusion, yielding a fused feature vector h representing the "phase-duration synergy relationship." t .

[0083] Based on this, the two types of residual numerical units respectively use a multilayer perceptron (MLP) to fuse the features h. t Projecting onto the action space of phase and duration, learn the phase residual value ΔQ p Duration residual ΔQ d :

[0084]

[0085] fusing feature h t Input two independent MLP networks respectively. Phase Residual MLP: Outputs a vector ΔQ. p Its dimension and fundamental phase value Q p Exactly the same. Each value in the vector This refers to the correction amount (positive or negative) for the value of the i-th phase. Similarly, the duration residual MLP outputs the correction vector ΔQ for the duration value. d .

[0086] Ultimately, the action value of phase and duration is obtained by adding the base value and the residual value:

[0087]

[0088] The initial motion value calculated by the basic module is added element-wise to the residual correction value calculated by the correction module to obtain the final motion value.

[0089] The phase agent and duration agent determine the final action based on the corrected action value. To balance exploration and exploitation, this disclosure adopts an ε-greedy strategy: starting from the action space A with probability ε... x In a uniformly randomized manner, select the action with the highest current value with probability 1-ε:

[0090]

[0091] Through the aforementioned dual residual correction mechanism, this disclosure can effectively capture the synergistic relationship between phase and duration decisions, dynamically correct the value of basic actions, and improve the accuracy and stability of traffic signal control strategies.

[0092] According to an exemplary embodiment, in selecting phase actions and duration actions to form traffic signal control instructions, an ε-greedy strategy is adopted when selecting phase actions and duration actions based on the final phase action value vector and the final duration action value vector: phase actions and duration actions are uniformly and randomly selected from the action space with probability ε, and the phase action and duration action with the highest current value are selected with probability 1-ε.

[0093] Individual Residual Reward Mechanism: In this disclosure, the reward signal not only determines the optimization direction and objective of each control module (phase and duration), but also directly affects the collaborative control effect between the two. If a shared global reward is directly used to simultaneously optimize both phase and duration agents, it can easily lead to inertia in both agents, making it difficult to leverage their respective advantages. A global reward refers to a single reward that is simultaneously fed back from the environment by both phase and duration agents; this reward cannot clearly distinguish the effectiveness of each agent. This disclosure proposes an individual residual reward mechanism to encourage agents to continuously improve the collaborative control effect while maintaining basic control performance.

[0094] According to an exemplary embodiment, the method further includes: obtaining the individual phase reward and the individual duration reward before correction, wherein the individual phase reward and the individual duration reward before correction are the individual phase reward and the individual duration reward obtained based on Shapley value theory when the dual residual correction module is not introduced; obtaining the individual phase reward and the individual duration reward after correction, wherein the individual phase reward and the individual duration reward after correction are the individual phase reward and the individual duration reward obtained based on Shapley value theory after the dual residual correction module is introduced; and using the difference between the individual phase reward after correction and the individual phase reward before correction, and the difference between the individual duration reward after correction and the individual duration reward before correction, as residual rewards for updating the dual residual correction module.

[0095] This mechanism first uses Shapley value theory to fairly distribute the global reward according to each agent's actual contribution to the control effect, thereby quantifying the individual reward of each agent. Based on this, the individual residual reward mechanism splits the reward into two components: a base reward and a residual reward. The base reward ensures that the control effect after collaborative correction is not lower than the performance of the base control module; the residual reward measures the effective gain brought by the correction and guides each agent to continue improving upon this.

[0096] The Shapley value is calculated as follows:

[0097]

[0098]

[0099] Where, φ p φ d Let r(p∪d) represent the individual rewards allocated to the phase agent and the duration agent from the global reward r(p∪d), respectively, and let r(·) represent the rewards obtained under different actions. r(p) represents the reward obtained by performing only the phase action, r(d) represents the reward obtained by performing only the duration action, and r(p∪d) represents the global reward obtained by performing both the phase and duration actions.

[0100] In actual operation, the basic reward is the global reward directly returned by the environment. The allocation method described above ensures the basic performance after the correction action is performed.

[0101]

[0102]

[0103] in, These are the base rewards for phase proxy and duration proxy, respectively. By their respective reward models R p (·), R d (·) The predictions are obtained, and the reward model is implemented through a multilayer perceptron (MLP).

[0104] According to an exemplary embodiment, the method further includes: obtaining a first environmental reward value when only a phase action is performed and the duration is fixed at a preset default value; obtaining a second environmental reward value when only a duration action is performed and the phase is fixed at a preset default value; obtaining a third environmental reward value when both a phase action and a duration action are performed simultaneously; and calculating phase individual rewards and duration individual rewards based on the first to third environmental reward values ​​using Shapley value theory.

[0105] The residual reward is defined as the increase in individual reward resulting from correction, i.e., the individual reward after correction minus the individual reward before correction. A larger residual reward indicates more effective correction; a small or even negative residual reward indicates limited performance improvement from correction, or even damage to basic control performance. To ensure correction performance and the effectiveness of co-correction, the final individual residual reward is calculated as follows:

[0106]

[0107]

[0108]

[0109] in, Individual rewards before correction The values ​​represent the pre-correction phase and duration, as well as the global reward value, all derived from a reward model R implemented using a multilayer perceptron (MLP). p (·), R d (·), R pd (·) Predicted, The residual reward is denoted by α, which is a coefficient that adjusts the weight of the residual reward. Through this mechanism, the phase agent and the duration agent can effectively improve collaborative control capabilities while maintaining basic performance.

[0110] Incremental optimization strategy

[0111] To reduce the overall optimization difficulty of phase-duration joint control tasks and improve the stability of control performance, this disclosure proposes an incremental optimization strategy that decomposes the complex joint optimization process into two stages to gradually improve model performance.

[0112] In the first stage, the basic phase module is optimized during fixed-duration control, and the basic duration module is optimized during fixed-phase sequence control. This allows the two modules to interact independently with the environment, and the reward signal returned by the environment directly reflects their respective control contributions. This stage directly utilizes environmental rewards to optimize the basic control module, and its loss function can be expressed as:

[0113]

[0114] in, This refers to the dataset collected during the interaction process at this stage.

[0115] According to an exemplary embodiment, the residual reward is used to update the dual residual correction module using the following loss function:

[0116]

[0117] in, Q represents the residual reward. x (s t ,a x ) represents the final phase or duration action value vector, γ represents the discount factor, and D x This represents the dataset acquired during the phase of updating the dual residual correction module, and E represents the expectation for the samples.

[0118] According to an exemplary embodiment, the method further includes a training process that employs a progressive optimization strategy, including: a first stage, a fixed-duration control strategy for training the basic phase module separately; and a fixed-phase control strategy for training the basic duration module separately; and a second stage, after completing the first stage of training, introducing and jointly training a dual residual correction module.

[0119] After completing the first phase, three interaction settings were used to collect state-action-reward samples: interaction only between the basic phase module and the environment (fixed duration); interaction only between the basic duration module and the environment (fixed phase); and simultaneous interaction between the basic phase module and the basic duration module and the environment. Based on this data, three reward prediction models R were trained. p (·), R d (·), R pd (·), used to predict individual residual rewards in the second stage, without needing to be trained in conjunction with the dual residual correction module.

[0120] In the second stage, a dual residual correction module is introduced to further modify the basic actions based on the modeled phase-duration coordination relationship, thereby improving the joint control effect. In this stage, the basic phase module and the basic duration module simultaneously execute the coordinated and corrected actions and receive the global reward signal returned by the environment. This global reward is decomposed into individual residual rewards through an individual residual reward mechanism. When used to optimize the dual residual correction module, its loss function can be expressed as:

[0121]

[0122] Among them, D x To coordinate the dataset collected during the correction phase, this disclosure achieves a transition from basic control capabilities to collaborative correction capabilities through a two-stage progressive optimization strategy, effectively reducing training instability and significantly improving joint control performance.

[0123] This embodiment provides a traffic signal phase and duration collaborative control method based on dual residual reinforcement learning, used to optimize intersection traffic light control strategies. The entire process includes a basic decision-making module, a dual residual correction module, an individual residual reward mechanism, and a progressive optimization strategy, as detailed below:

[0124] In this step, a basic phase module and a basic duration module are constructed, each containing a feature extraction unit and an action value prediction unit, respectively. The feature extraction unit uses a multi-head self-attention mechanism to embed the intersection state (including the current signal phase, the number of vehicles in the approach lane, the number of vehicles in the lane segment, etc.) to capture the correlation between traffic states of different lanes; the action value prediction unit uses a linear layer to output the basic value of actions for each phase and each duration. and The two basic modules determine their respective initial actions based on the value of the basic actions. and As input for subsequent corrections.

[0125] In this step, a dual residual correction module is constructed, consisting of a phase-duration coordinated modeling unit and two sets of residual value units. The coordinated unit will convert the basic motion value... and The state features extracted by multi-head self-attention are input together into a multilayer perceptron to obtain the fused feature h. t The two sets of residual value units are mapped to the phase and duration action spaces, respectively, and output ΔQ. p With ΔQ d The final corrected value of the action is obtained. and Select the final action a based on the revised action value. p and a d .

[0126] In this step, to address the problem of difficulty in distinguishing individual contributions from reward signals in reinforcement learning, an individual residual reward mechanism is designed. First, the Shapley value theory is used to fairly distribute the global reward to the phase agent and the duration agent, obtaining their individual rewards φ. p and φ dSecondly, the individual residual reward mechanism divides the reward into two parts: a basic reward and a residual reward. The basic reward ensures that the control performance after coordination and correction is not lower than the basic control level, while the residual reward is used to measure the performance gain brought about by the correction. Finally, the individual residual reward serves as a training objective, guiding the two agents to continuously improve the coordinated control effect while maintaining basic performance.

[0127] In this step, a progressive optimization strategy is used to train the aforementioned modules. First, in the initial stage, the phase module and the base duration module are trained separately with the other side's parameters fixed, allowing them to interact independently with the environment, with the environment providing rewards. It is directly used to optimize the basic control module, and collects state, action, and reward samples to train three reward prediction models R. p (·), R d (·), R pd (·); In the second stage, a dual residual correction module is introduced to coordinate and correct the basic actions. After executing the corrected actions, the global reward is received, and then the global reward is decomposed into individual residual rewards through the individual residual reward mechanism. It is also used to optimize the dual residual correction module, achieving a stable improvement in the performance of combined phase and duration control.

[0128] The technical solution provided in this embodiment can effectively capture the coordination relationship between phase and duration while ensuring the stability of basic control, realize the gradual optimization and coordinated improvement of traffic signal control, and significantly improve the intersection operation efficiency and overall traffic flow capacity.

[0129] In the embodiments of this disclosure, a dual residual reinforcement learning model is employed to achieve phase and duration co-optimization control of traffic signals. Compared with traditional reinforcement learning control methods (which do not introduce a residual correction mechanism or only use a single control dimension mechanism), this method shows significant advantages in control stability, strategy convergence speed, and traffic flow coordination.

[0130] Specifically, the method disclosed herein can maintain high signal control stability in complex traffic environments with multiple intersections and avoid traffic conflicts caused by frequent phase switching; at the same time, the introduction of the dual residual module significantly improves the coordination consistency between phase and duration, making the overall signal control smoother.

[0131] In terms of optimization performance, this method effectively reduces model instability through a phased training strategy and strengthens the cooperation between the basic and residual modules, resulting in better signal control performance under various traffic flow patterns. In summary, the method disclosed herein demonstrates significant improvements over the comparative method in terms of collaborative modeling capability, learning stability, and overall control performance.

[0132] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles used, and is not intended to limit the scope of the claimed disclosure, but only to illustrate preferred embodiments of this disclosure. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalent features without departing from the inventive concept. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

Claims

1. A traffic signal phase and duration collaborative control method based on dual residual reinforcement learning, characterized in that, include: Obtain the traffic state vector of the intersection, including the current signal phase, the number of vehicles in each approach lane, and the number of vehicles in each lane segment; The basic phase module and the basic duration module generate the initial phase action value vector and the initial duration action value vector respectively based on the traffic state vector; The dual residual correction module generates a phase residual value vector and a duration residual value vector based on the initial phase action value vector, the initial duration action value vector, and the traffic state vector, respectively. Based on the initial phase action value vector, the initial duration action value vector, the phase residual value vector, and the duration residual value vector, a final phase action value vector and a final duration action value vector are generated; and Phase actions and duration actions are selected based on the final phase action value vector and the final duration action value vector to form traffic signal control instructions.

2. The method according to claim 1, characterized in that, The basic phase module and the basic duration module generate the initial phase action value vector and the initial duration action value vector respectively based on the traffic state vector: The traffic state vector is embedded with features based on a multi-head self-attention mechanism to generate an initial feature representation. The initial feature representation is nonlinearly transformed based on a multilayer perceptron to generate a high-dimensional feature representation. as well as The high-dimensional feature representation is processed using a phase linear mapping layer and a duration linear mapping layer to generate the initial phase action value vector and the initial duration action value vector, respectively.

3. The method according to claim 2, characterized in that, The dual residual correction module generates phase residual value vectors and duration residual value vectors based on the initial phase action value vector, the initial duration action value vector, and the traffic state vector, respectively, including: The feature vector obtained by performing feature mapping on the initial phase action value vector and the initial duration action value vector based on a multi-layer perceptron is concatenated with the feature vector obtained by embedding the traffic state vector based on a multi-head self-attention mechanism. Feature extraction is performed on the concatenated feature vector using a multilayer perceptron to generate a fused feature vector; and The fused feature vectors are respectively mapped to the action spaces of phase and duration through a multilayer perceptron, thereby generating the phase residual value vector and the duration residual value vector.

4. The method according to claim 3, characterized in that, In selecting phase actions and duration actions to form traffic signal control instructions, an ε-greedy strategy is adopted when selecting phase actions and duration actions based on the final phase action value vector and the final duration action value vector: phase actions and duration actions are uniformly and randomly selected from the action space with probability ε, and the phase action and duration action with the highest current value are selected with probability 1-ε.

5. The method according to claim 4, characterized in that, The method further includes: The individual phase reward and the individual duration reward before correction are obtained respectively. The individual phase reward and the individual duration reward before correction are the individual phase reward and the individual duration reward obtained based on Shapley value theory when the dual residual correction module is not introduced. The corrected phase individual reward and the corrected duration individual reward are obtained separately. These are the phase individual reward and duration individual reward obtained based on Shapley value theory after introducing the dual residual correction module. The difference between the corrected phase individual reward and the uncorrected phase individual reward, and the difference between the corrected duration individual reward and the uncorrected duration individual reward, are used as residual rewards to update the dual residual correction module.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the first environmental reward value when only phase actions are performed and the duration is fixed at a preset default value; Obtain the second environment reward value when only the duration action is executed and the phase is fixed at the preset default value; Obtain third-environment reward value while simultaneously performing phase actions and duration actions; and Using Shapley value theory, phase individual rewards and duration individual rewards are calculated based on the first environment reward value to the third environment reward value.

7. The method according to claim 6, characterized in that, The residual reward is used to update the dual residual correction module using the following loss function: in, Q represents the residual reward. x (s t ,a x ) represents the final phase or duration action value vector, γ represents the discount factor, and D x E represents the dataset acquired during the phase of updating the dual residual correction module, and E represents the expectation for the samples.

8. The method according to claim 7, characterized in that, The method further includes a training process, which employs a progressive optimization strategy, including: The first stage involves training the basic phase module separately using a fixed-duration control strategy; and training the basic duration module separately using a fixed-phase control strategy; and In the second stage, after completing the first stage of training, the dual residual correction module is introduced and trained in conjunction with the first stage.

Images