A regional signal macro-micro integrated simulation control method and system based on adversarial enhanced reinforcement learning
By employing an adversarial reinforcement learning-based integrated macro-micro simulation control method for regional traffic signals, combined with conditional WGAN-GP and QMIX multi-agent reinforcement learning, the shortcomings of traditional traffic signal control methods in state prediction and collaborative optimization are addressed. This approach achieves efficient and robust traffic signal management, reduces queue length and delays, and improves the traffic efficiency of the urban road network.
Patent Information
- Application Number
- CN202610839084.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-11
- Publication Date
- 2026-07-14
AI Technical Summary
Existing traffic signal control methods struggle to simultaneously meet the demands for high-precision state prediction, efficient collaborative optimization, and strong generalization robustness. Traditional methods are inefficient in scenarios with large fluctuations in traffic demand, reinforcement learning-based methods rely heavily on large-scale historical data and lack generalization ability, and multi-agent control is insufficient in terms of regional collaborative optimization.
A regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning is adopted. Future traffic state data is generated through conditional WGAN-GP and combined with the QMIX multi-agent reinforcement learning algorithm to achieve signal collaborative optimization of the regional road network, forming a 'perception-generation-control' closed loop, thereby reducing average queue length and vehicle delay.
It significantly reduces the average queue length and vehicle delays of the regional road network, improves the adaptability and robustness of traffic signal control, and enhances the traffic efficiency of the regional road network. It is suitable for intelligent signal control in urban road network areas with multiple intersections.
Smart Images
Figure CN122392335A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent traffic control technology, and relates to a regional signal macro-micro integrated simulation control method and system based on adversarial enhancement reinforcement learning. Background Technology
[0002] With the acceleration of urbanization and the continuous growth of motor vehicle ownership, traffic congestion has become a core issue restricting urban operational efficiency. Intersections, as key nodes for traffic convergence and distribution in road networks, directly impact the traffic efficiency of the regional road network due to the effectiveness of their signal control strategies. Existing traffic signal control methods mainly suffer from the following technical shortcomings:
[0003] Traditional fixed timing control relies on preset signal schemes, which cannot adapt to the dynamic time-varying characteristics of traffic flow. In scenarios with large fluctuations in traffic demand, it is easy to cause green lights to be wasted or queue overflow, resulting in low control efficiency.
[0004] While adaptive control methods based on reinforcement learning can dynamically adjust signal timing through the interaction between the agent and the environment, their performance is highly dependent on large-scale, high-quality historical interaction data. In real-world scenarios, data acquisition is costly and time-consuming. Furthermore, when encountering sudden congestion or extreme traffic patterns not covered in the training data, the model's generalization ability and robustness decrease significantly.
[0005] Current research on generative artificial intelligence and reinforcement learning shows a trend of "parallel but not deeply integrated". Generative models are mostly used as independent data preprocessing tools for data augmentation or state prediction. Their generation results are disconnected from the downstream adaptive signal control decision-making process, failing to form a closed loop of "state generation-decision optimization", which limits the adaptive capabilities of complex traffic systems.
[0006] In terms of regional multi-intersection collaborative control, traditional control methods optimize each intersection independently, making it difficult to achieve optimal global traffic efficiency. Existing multi-agent reinforcement learning methods still have shortcomings in coordinating the joint actions of multiple agents and ensuring the stability of centralized training and distributed execution. There is a lack of a complete closed-loop framework that can efficiently generate future states and collaboratively optimize the timing of signals at multiple intersections.
[0007] In summary, existing control methods cannot simultaneously meet the requirements of high-precision state prediction, efficient collaborative optimization, and strong generalization robustness. Summary of the Invention
[0008] The purpose of this invention is to provide a regional signal macro-micro integrated simulation control method and system based on adversarial reinforcement learning, which can reduce the average queue length and vehicle delay of regional road networks and improve the adaptability and robustness of traffic signal control.
[0009] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.
[0010] In a first aspect, this invention proposes a regional signal macro-micro integrated simulation control method based on adversarial enhancement reinforcement learning, comprising:
[0011] Lane-level traffic state data is collected, and the collected traffic state data is preprocessed to obtain preprocessed traffic state data. Based on the preprocessed traffic state data and the simulated road network of the area to be studied, a historical traffic state sequence is generated. The traffic state data includes the flow rate of the approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate.
[0012] The obtained historical traffic state sequence is input into the pre-trained conditional WGAN-GP model to predict the traffic state data for the next time step, that is, the flow rate of the approach lane for the next time step.
[0013] Based on the QMIX multi-agent reinforcement learning algorithm, each intersection in the regional road network is modeled as an independent agent, where a single intersection agent is used to implement lane-level traffic light control. The traffic state data predicted for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output.
[0014] In conjunction with the first aspect, after outputting the signal timing of each intersection in a future time step, the simulation control method of the present invention further includes the following steps:
[0015] The system executes the signal timing for each intersection based on the output in the next future time step, collects traffic state data after execution, and updates the historical traffic state data based on the executed traffic state data.
[0016] In conjunction with the first aspect, further, the collected lane-level traffic state data is preprocessed to obtain preprocessed traffic state data; based on the preprocessed traffic state data and the simulated road network of the area under study, a historical traffic state sequence is generated, including:
[0017] Traffic status data is collected using lane detectors;
[0018] Outliers were removed from the collected traffic status data using certain criteria, and missing data were filled in using spatiotemporal interpolation methods to obtain preprocessed traffic status data.
[0019] Based on the simulated road network of the area under study, simulated road network information is extracted; the preprocessed traffic state data and the extracted simulated road network information are fused to obtain the historical traffic state sequence.
[0020] The simulated road network information includes the center coordinates of intersections, lane geometric attributes, entrance and exit attributes, and topological connections between lanes. The lane geometric attributes include lane length, lane width, and lane orientation.
[0021] The step of fusing the preprocessed traffic state data and the extracted simulated road network information to obtain a historical traffic state sequence includes:
[0022] Based on the lane number, the states of each entrance lane within the same intersection are spliced together to form a local state vector. Then, based on the topological connection relationship between lanes, the states of adjacent lanes are mapped or weighted and fused to obtain a historical traffic state sequence.
[0023] In conjunction with the first aspect, the training method for the conditional WGAN-GP model is as follows: using historical traffic state sequences as conditions, traffic state data for the next time step is generated, including:
[0024] Step 2.0: Obtain the historical traffic state sequence for N consecutive time steps;
[0025] Step 2.1: Use the historical traffic state sequence of N consecutive time steps as the condition vector and the random noise vector following a Gaussian distribution as the latent variable. Concatenate the condition vector and the latent variable and input them into the generator.
[0026] Step 2.2: The generator uses an M-layer fully connected network, which maps layer by layer through linear transformation and ReLU activation function to output the traffic state sequence of the next time step. The output value is constrained to the interval [-1,1] by Tanh activation function to output the traffic state sequence of the next time step.
[0027] Step 2.3: Construct a discriminator. The discriminator adopts a two-branch parallel structure, where the global branch is used to evaluate the overall authenticity of the traffic state sequence in the next time step, and the local branch is used to evaluate the accuracy of the traffic state sequence at the last time step in the next time step.
[0028] Step 2.4: Using Wasserstein distance as the optimization objective of the generative adversarial network, a gradient penalty term is introduced to ensure training stability, and a terminal temporal consistency constraint term is added to the discriminator loss. The discriminator loss function is expressed as:
[0029] ;
[0030] in, For discriminator loss, This is a sequence of actual traffic conditions. It is random noise. Input the historical traffic state sequence conditions. For generator, For discriminator, This is a linear interpolation between real traffic state data and generated traffic state data. For gradient penalty weights, For end-time consistency weights, and These represent the states of the actual traffic state sequence and the generated traffic state sequence at the last time step, respectively. This represents the distribution of real traffic condition data; The noise prior distribution; To indicate about The gradient; This represents the expected output of the discriminator's score on the real traffic state sequence; This represents the expected output score of the discriminator when sampling from a noisy prior distribution and generating fake samples by the generator; Represents the gradient penalty term; This represents the end-point timing consistency constraint term;
[0031] Step 2.5: The sum of the discriminator output expectation (with a negative generator loss function) and the terminal temporal consistency constraint term is used to minimize the generator loss, enabling the generator to learn to generate sequences that conform to the true distribution and have accurate terminals. The generator loss function is expressed as:
[0032] ;
[0033] in, The generator loss is minimized to enable the generation module to generate a traffic state sequence for the next time step that conforms to the true distribution and has high accuracy at the final time step.
[0034] The traffic state sequence for the next time step output by the generator has the same state dimension as the historical traffic state sequence, and the traffic state sequence for the next time step is used to characterize the lane-level traffic operation status for the next one or N time steps.
[0035] In conjunction with the first aspect, the method for obtaining the historical traffic state sequence over N consecutive time steps is as follows:
[0036] A simulated road network for the area under study is established in a microscopic traffic simulation platform, and lane detectors are deployed on the entrance and exit lanes of each intersection to collect traffic status data.
[0037] The collected traffic status data is preprocessed by using criteria to remove outliers and spatiotemporal interpolation to fill in missing data, resulting in preprocessed traffic status data.
[0038] Based on the preprocessed traffic state data, the simulation road network file is parsed to extract the intersection center coordinates, lane geometric attributes, entrance and exit attributes and the topological connection relationship between lanes to obtain time series data.
[0039] The preprocessed traffic state data of each lane within the same time step are correlated and fused according to the lane spatial location. Specifically, the states of each entrance lane in the same intersection are spliced together to form a local state vector according to the lane number, and the states of adjacent lanes are mapped or weighted and fused according to the topological connection relationship between lanes to obtain time series data.
[0040] Arrange the time-series data within N consecutive time steps in chronological order to form a historical traffic state sequence; the historical traffic state sequence includes historical traffic flow sequence, historical average vehicle delay sequence, historical maximum queue length sequence, historical average vehicle parking time sequence, and historical vehicle occupancy rate sequence.
[0041] In conjunction with the first aspect, further, the QMIX multi-agent reinforcement learning algorithm models each intersection in the regional road network as an independent agent, wherein a single intersection agent is used to implement lane-level traffic light control and perform control optimization; the predicted traffic state data for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output, including:
[0042] Each intersection in the regional road network is modeled as an independent intelligent agent. The local state of each intelligent agent includes the traffic flow of each approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate. The global state is defined as the summary information of the local states of each intersection in the region, which is used to characterize the macro traffic situation in the region.
[0043] Each agent uses a deep Q-network to fit a local Q-value function. The input of the deep Q-network is the local state, and the output is the Q value corresponding to each action. The action space includes two types of actions: the first type is to extend the duration of the current signal phase, and the second type is to switch to the next signal phase.
[0044] The reward function is designed as a weighted sum of average queue length, average vehicle parking time, and regional congestion level indicators. Minimizing this weighted sum is the optimization objective. The reward function is expressed as:
[0045] ;
[0046] in, The average queue length, This represents the average vehicle parking time. As an indicator of regional congestion level, These are the weighting coefficients. Represents the reward function;
[0047] A hybrid network is used to combine the local Q-values of each agent into a global Q-value. The hybrid network generates the weight matrix and bias vector from the supernetwork, and achieves a monotonic mapping from the local Q-value to the global Q-value through two layers of nonlinear transformation. Its expression is:
[0048] ;
[0049] in, The global action value function; For the first The local Q-value function of an agent; For the first The observation history of the first agent, i.e., the first... Traffic state sequence of an intersection; For the first The corresponding action of the first agent, i.e. the first... Signal phase control actions at each intersection; This is the global state. The monotonic mixture function generated by the hypernetwork, This represents the joint observation history of all agents. This represents the joint action of all intelligent agents. This represents the total number of intelligent agents, which is the total number of intersections in the regional road network;
[0050] A centralized training and distributed execution framework is adopted. Training is performed through experience replay, where an experience replay buffer stores the joint experience of all agents. The loss function is constructed using the TD error, and the parameters of each agent network and the hybrid network are jointly updated through the Adam optimizer. Represented as:
[0051]
[0052] In the formula, To replay the experience pool The mathematical expectation of the joint experience of random sampling in the middle, The time difference target value, This is the global action value function output by the current hybrid network; These are the current network parameters, including the parameters of each agent network and the hybrid network;
[0053] The time difference objective value is:
[0054] ;
[0055] In the formula, The global action value function output by the target hybrid network; This is the candidate joint action vector for all agents in the next time step; This will serve as the joint observation history of all agents for the next time step; This represents the global state at the next time step. For global, instant rewards; Discount factor;
[0056] The current network parameters are softly updated to the target network every fixed number of steps to execute the update strategy and ensure training stability. During execution, each agent independently selects the action that maximizes the local Q value based on the local state to obtain the optimal action combination and achieve distributed real-time control.
[0057] Secondly, this invention proposes a regional signal macro-micro integrated simulation control system based on adversarial reinforcement learning, used to implement the aforementioned regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning, comprising:
[0058] The perception module is configured to collect lane-level traffic state data, preprocess the collected traffic state data to obtain preprocessed traffic state data, and generate a historical traffic state sequence based on the preprocessed traffic state data and the simulated road network of the area under study. The traffic state data includes the flow rate of the approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate.
[0059] The generation module is configured to input the obtained traffic state sequence into a pre-trained conditional WGAN-GP model to predict the traffic state data for the next time step, i.e., the flow rate of the approach lanes for the next time step.
[0060] The control module is configured to model each intersection in the regional road network as an independent agent based on the QMIX multi-agent reinforcement learning algorithm, wherein a single intersection agent is used to implement lane-level traffic light control; the predicted traffic state data for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output.
[0061] In conjunction with the second aspect, it further includes a closed-loop update module, which is configured to execute the signal timing of each intersection in the next future time step, collect traffic state data after execution, and update historical traffic state data based on the traffic state data after execution, so as to achieve integrated macro and micro control.
[0062] In conjunction with the second aspect, further, the cascading of the sensing module, generation module, and control module into a closed-loop system, iteratively optimizing it in a simulation environment, and outputting a coordinated signal control strategy includes:
[0063] Step 4.1: The sensing module continuously collects and stores traffic status data for each lane at preset fixed time intervals, forming a sliding window historical traffic status sequence;
[0064] Step 4.2: When the length of the accumulated historical traffic state sequence reaches the preset window threshold, the generation module uses the historical traffic state sequence as a condition and combines it with random noise to generate the predicted traffic state for the next time step.
[0065] Step 4.3: The control module fuses the predicted traffic state for the next time step with the real-time collected traffic state as the input for the local and global states of each agent, and outputs and executes the signal timing actions of each intersection through the QMIX framework.
[0066] Step 4.4: Feed back the new traffic state data after the action is executed to the perception module, update the historical traffic state sequence, and form a closed-loop iteration of perception, generation, control and feedback;
[0067] Step 4.5: Repeat steps 4.1 to 4.4 until convergence or the preset simulation time is reached, and output the final cooperative signal control strategy.
[0068] Thirdly, the present invention proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-mentioned integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning.
[0069] Fourthly, the present invention provides a computer device comprising:
[0070] Memory, used to store computer programs;
[0071] A processor is used to execute the computer program to implement the steps of the above-described method for integrated macro-micro simulation control of regional signals based on adversarial reinforcement learning.
[0072] Fifthly, the present invention proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning.
[0073] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0074] (1) This invention deeply integrates the data generation capability of generative adversarial networks with multi-agent collaborative decision-making to form a "perception-generation-control" closed loop, which significantly reduces the average queuing length and vehicle delay of regional road networks and improves the adaptive capability and robustness of traffic signal control.
[0075] (2) This invention constructs a perception module based on microscopic traffic simulation, realizes high-fidelity future traffic state generation with the conditional WGAN-GP generation module as the core, and realizes regional multi-intersection signal collaborative optimization with QMIX multi-agent reinforcement learning as the engine, forming a "perception-generation-control" closed-loop system to improve the traffic efficiency of regional road network.
[0076] (3) This invention constructs a "perception-generation-control" closed-loop framework, deeply integrating the high-fidelity future traffic state generated by conditional WGAN-GP with QMIX multi-agent cooperative control, effectively solving the problems of poor adaptability, strong dependence on large-scale historical data, and insufficient regional coordination capability of traditional signal control methods. This invention uses a conditional Wasserstein generative adversarial network with gradient penalty and a terminal timing consistency constraint term to generate high-quality future traffic state data with a small amount of historical traffic state data, reducing the dependence on real historical traffic state data; based on QMIX multi-agent cooperative control, it realizes global optimization of regional multi-intersection signals. Simulation results show that compared with the fixed timing strategy, the control method of this invention reduces the average queue length by 46.8% and the average vehicle delay by 33.2%, significantly improving the traffic efficiency of the regional road network and the robustness of the control system, and can be widely applied to intelligent signal control in urban road network multi-intersection areas.
[0077] (4) This invention is applicable to adaptive and coordinated traffic signal control in urban road network areas with multiple intersections, and can provide dynamic, efficient and robust signal timing decision support for intelligent traffic management systems. Attached Figure Description
[0078] Figure 1 This is a flowchart illustrating the simulation control method in Embodiment 1 of the present invention;
[0079] Figure 2 This is a time-series comparison diagram of the traffic generated by the generation module and the actual traffic in Embodiment 2 of the present invention;
[0080] Figure 3 This is a performance comparison chart of the original traffic control system and the autonomous traffic control system (the traffic control system using the simulation control method of this invention) in Embodiment 2 of the present invention. Figure 3 Figure (a) shows a comparison of the average speed (m / s) of the control module and the original fixed timing strategy. Figure 3Figure (b) shows a comparison of the average maximum queue length (m) between the control module and the original fixed timing strategy. Figure 3 Figure (c) shows a comparison of the average waiting time (s) of the control module and the original fixed timing strategy. Figure 3 Figure (d) shows a comparison of the time loss (s) between the control module and the original fixed timing strategy;
[0081] Figure 4 This is a histogram showing the performance difference between the original traffic control system and the autonomous traffic control system (the traffic control system using the simulation control method of this invention) in Embodiment 2 of the present invention. Figure 4 Figure (a) is a histogram showing the distribution of the difference in average velocity (m / s). Figure 4 Figure (b) is a histogram showing the distribution of the difference in average maximum queue length (m). Figure 4 Figure (c) is a histogram showing the distribution of the difference in average waiting time (s). Figure 4 The middle (d) figure is a histogram of the time loss (s) difference distribution. Detailed Implementation
[0082] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0083] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0084] Example 1
[0085] like Figure 1 As shown, the steps of the region signal macro-micro integrated simulation control method based on adversarial enhancement reinforcement learning in this embodiment are as follows:
[0086] Step S1: Construct a perception module based on microscopic traffic simulation, collect and fuse multi-source lane-level traffic state data, and generate a historical traffic state sequence including the microscopic state of a single intersection and the macroscopic state of the regional road network.
[0087] In step S1, to achieve a unified representation of the micro-state of a single intersection (micro-state refers to the state of a single vehicle, a single lane, and a single traffic light, such as the speed and acceleration of a single vehicle; the traffic flow and vehicle occupancy rate in a single lane; and the state and duration of a single traffic light) and the macro-state of the regional road network (macro-state, in contrast to micro-state, refers to the state of all vehicles, all lanes, and all traffic lights in the entire road network), a perception module is constructed. The specific steps are as follows:
[0088] Step 1.1: Establish a simulated road network for the area to be studied in the micro traffic simulation platform, and deploy lane detectors on the entrance and exit lanes of each intersection. The lane detectors are used to detect traffic state data, including the flow rate of the entrance lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate.
[0089] It should be noted that: when applying this invention, real-time traffic status data is collected; when training the conditional WGAN-GP model in this invention, historical traffic status data is collected.
[0090] Step 1.2: Preprocess the collected traffic status data using... The criteria remove outliers and use spatiotemporal interpolation methods to fill in missing data, so as to ensure the continuity of traffic status data in the time dimension and the consistency in the spatial dimension.
[0091] Step 1.3: Based on the preprocessed traffic state data, parse the simulation road network file, extract the intersection center coordinates, lane geometric attributes, entrance and exit attributes and the topological connection relationship between lanes, and establish a spatial identifier with intersection number and lane number as index, thus obtaining the time series data;
[0092] Step 1.4: The traffic state data of each lane within the same time step are correlated and fused according to the spatial location of the lane. Specifically, the traffic flow, average vehicle stopping time, maximum queue length and detector occupancy of each entrance lane in the same intersection are spliced together to form a local state vector according to the lane number. The state parameters of adjacent lanes are directly correlated and mapped or fused to the current lane according to the spatial adjacency based on the topological connection relationship between lanes, so as to obtain time series data that integrates spatial data and temporal data.
[0093] Step 1.5: Arrange the time series data within N consecutive time steps in chronological order to form a historical traffic state sequence. The historical traffic state sequence includes the historical traffic flow sequence, the historical average vehicle delay sequence, the historical maximum queue length sequence, the historical average vehicle parking time sequence, and the historical vehicle occupancy rate sequence.
[0094] Historical traffic state sequences are used to simultaneously characterize the micro-state of a single intersection and the macro-state of the regional road network, thus providing a unified input for subsequent generation and control modules.
[0095] Step S2: Construct a generation module based on conditional WGAN-GP, using historical traffic state sequences as conditional vectors to generate traffic state data for the next time step;
[0096] In step S2, to achieve high-fidelity generation of traffic states for a future time step, a generation module based on conditional WGAN-GP is constructed. The specific steps are as follows:
[0097] Step 2.1: Use the historical traffic state sequence of several consecutive time steps as the condition vector, and the random noise vector that follows a Gaussian distribution as the latent variable. Then concatenate the condition vector and the latent variable and input them into the generator.
[0098] Step 2.2: The generator uses a multi-layer fully connected network to map the input condition vector. Through linear transformation and ReLU activation function, it maps layer by layer to output the traffic state sequence for the next time step. The Tanh activation function then constrains the output traffic state sequence for the next time step to the interval [-1,1]. The traffic state sequence for the next time step is consistent with the historical traffic state sequence in the state dimension, and is used to represent the lane-level traffic operation state for one or more future time steps.
[0099] Step 2.3: Construct a discriminator. The discriminator adopts a dual-branch parallel structure. The global branch is used to evaluate the overall authenticity of the traffic state sequence in the next time step, and the local branch is used to evaluate the accuracy of the traffic state sequence at the end time step in the next time step, so as to ensure that the generated results not only conform to the overall traffic evolution trend, but also maintain the consistency of the end time series characteristics.
[0100] Step 2.4: The Wasserstein distance is used as the optimization objective of the generative adversarial network. A gradient penalty term is introduced to ensure training stability. An end-time consistency constraint term is added to the discriminator loss to make the last time step of the generated sequence (traffic state sequence in the next time step) as close as possible to the corresponding time step of the real sequence (real traffic state sequence).
[0101] Step 2.5: Introduce the discriminator output term and the terminal temporal consistency constraint term into the generator loss function. By minimizing the generator loss, the generator learns to generate traffic state data for the next time step that conforms to the true distribution and has accurate terminal prediction under historical traffic state conditions.
[0102] Step 2.6: The generated traffic state data for the next time step is used for regional macro-situation prediction and single-intersection micro-control input, thereby providing the control module with forward-looking state information.
[0103] Step S3: Construct a control module based on QMIX multi-agent reinforcement learning, taking the traffic state data of the next time step as input, and output the signal timing actions of each intersection to achieve regional collaborative control optimization;
[0104] In step S3, to achieve the unification of regional multi-intersection collaborative control and single-intersection fine control, a control module based on QMIX multi-agent reinforcement learning is constructed. The specific steps are as follows:
[0105] Step 3.1: Model each intersection in the regional road network as an independent intelligent agent. Each intelligent agent corresponds to a micro control unit. The micro control unit uses the traffic flow, average vehicle delay, maximum queue length, average vehicle stopping time and vehicle occupancy rate of each entrance lane of the intersection as local states to achieve fine signal control of a single intersection.
[0106] Step 3.2: Construct a regional coordination layer, taking the joint state of all intersections within the region as the global state. The global state is used to characterize the macro-traffic situation of the regional road network, and then to coordinate and constrain the local control behavior of each intersection.
[0107] Step 3.3: Each agent uses a deep Q-network to fit the local Q-value function. The input of the deep Q-network is the local state, and the output is the Q value corresponding to each action. The action space includes two types of actions: the first type is to extend the duration of the current signal phase, and the second type is to switch to the next signal phase.
[0108] Step 3.4: The reward function is designed as a weighted sum of the average queue length, average vehicle parking time, and regional congestion level indicators. The optimization objective is to minimize the weighted sum so that the control strategy can simultaneously take into account the operating efficiency of a single intersection and the regional coordination effect.
[0109] Step 3.5: A hybrid network is used to combine the local Q-values of each agent into a global Q-value. The hybrid network generates a weight matrix and a bias vector from a super network. Through two layers of nonlinear transformation, a monotonic mapping from local Q-values to global Q-values is achieved, thereby constraining and guiding the micro-control actions of each intersection based on the macro-optimization objective of the region.
[0110] Step 3.6: A centralized training and distributed execution framework is adopted. Training is carried out through experience replay. During the training process, the experience replay buffer stores the joint experience of all agents. The current network parameters and target network parameters are updated through the loss function. The parameters of each agent network and the hybrid network are jointly updated through the Adam optimizer, and the update strategy is executed.
[0111] Step 3.7: Softly update the current network parameters to the target network every fixed number of steps to execute the update strategy and ensure training stability; during execution, each agent independently selects the action that maximizes the local Q value based on its local state to obtain the optimal action combination and achieve distributed real-time control;
[0112] Step 3.8: The control module achieves coordinated optimization of signal timing at multiple intersections at the regional level and fine adjustment of lane-level signal phase at the single intersection level, thereby forming a control mechanism that combines macro-regional coordination with micro-node control.
[0113] Step S4: The sensing module, generation module and control module are cascaded into a closed-loop system. The traffic status is updated based on the control results in the simulation environment, and the regional cooperative signal control strategy is iteratively optimized.
[0114] In step S4, to form a macro-micro integrated closed-loop control mechanism, the sensing module, generation module, and control module are cascaded into a closed-loop system. The specific steps are as follows:
[0115] Step 4.1: The sensing module continuously collects and stores traffic status data for each lane at preset fixed time intervals, updates the historical traffic status sequence, and forms a sliding window historical sequence.
[0116] Step 4.2: When the length of the accumulated historical sequence reaches the preset window threshold, the generation module uses the historical sequence as a condition and combines it with random noise to generate the predicted traffic state for the next time step or the next time window.
[0117] Step 4.3: The control module fuses the predicted traffic state for the next time step with the real-time collected traffic state as the input for the local and global states of each agent, and outputs the signal timing actions of each intersection through the QMIX framework and performs control optimization.
[0118] Step 4.4: After the signal timing action is executed, the road network traffic status is updated, and the updated traffic status is fed back to the sensing module to correct the historical traffic status sequence and form a new sliding window;
[0119] Step 4.5: Repeat steps 4.1 to 4.4 to continuously perform the "perception-generation-control-feedback" closed-loop iteration in the simulation environment until convergence or the preset simulation time is reached, and output the final regional cooperative signal control strategy.
[0120] Example 2
[0121] This embodiment further illustrates the macro-micro integrated simulation control method for regional signals based on adversarial enhancement reinforcement learning, using more specific data.
[0122] This invention relates to a region signal macro-micro integrated simulation control method based on adversarial enhancement reinforcement learning, the specific steps of which are as follows:
[0123] (a) Constructing a perception module.
[0124] A 5×5 grid road network with 25 intersections was constructed using the SUMO microsimulation platform. Lane detectors were installed at each approach lane to collect raw traffic state data, including lane flow, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy. The vehicle types were passenger cars and buses (9:1 ratio). The car-following model used the CACC (Cooperative Adaptive Cruise Control) model integrated into the SUMO microsimulation platform, and the lane-changing model used the LC2013 model integrated into the SUMO microsimulation platform. The road network was divided into 16 traffic zones, and OD matrices were randomly generated. The data acquisition time step was 300 seconds, the simulation duration was 3600 seconds, and the simulation was repeated 30 times. The collected traffic state data was analyzed using... The criteria are used to remove outliers, linear interpolation is used to fill in missing values, and historical traffic status sequences are formed by merging them according to lane spatial location.
[0125] (ii) Training and generation module.
[0126] Using the preprocessed historical traffic state sequence's approach lane flow sequence as the learning objective, the data is divided into continuous segments of length 6 using a sliding window. The first 5 time steps serve as conditional inputs, and the 6th time step serves as the prediction target. The generator takes the approach lane flow sequence of the first 5 time steps and random noise as input, and outputs the traffic state data for the next time step (i.e., the approach lane flow sequence for the next time step) through a 6-layer fully connected network. The discriminator adopts a two-branch structure: a global branch to judge the overall authenticity of the sequence, and a local branch to judge the accuracy of the final time step. The RMSprop optimizer is used for training; the discriminator is updated every 5 times, and the generator is updated once. The loss function is WGAN-GP loss with added L1 constraints at the end. After training, the MAE on the test set is 86.65, and the discriminator's F1 score is 0.736.
[0127] (III) Training control module.
[0128] Each intersection is represented by an agent. Its local state consists of traffic flow, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy data for each approach lane at that intersection. Its global state consists of average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy data for all intersections within the region. Each agent employs DQN, and its actions include extending the current phase duration or switching phases. The reward function is a weighted sum of queue length and stopping time, with a weight ratio of 6:4. The QMIX framework is used to map local Q-values to global Q-values. Training is performed through experience replay, with stable updates during training via experience replay and the target network. An Adam optimizer is used to jointly update parameters to execute the update strategy.
[0129] (iv) Closed-loop simulation verification.
[0130] The three modules are cascaded, with a simulation duration of 9600 seconds. The perception module collects historical traffic state sequences through a sliding window, the generation module predicts the traffic state data for the next time step (i.e., the flow rate of the approach lanes in the next time step), and the control module fuses the predicted values (traffic state data for the next time step) with real-time state output signals to form a closed loop. Compared with a fixed timing strategy, the simulation control method of this invention reduces the average queue length by 46.8% and the average delay by 33.2%, verifying the effectiveness of this invention.
[0131] like Figure 2 As shown, the generated traffic flow (referring to the traffic state data generated at the next future time step) and the actual traffic flow (referring to the actual traffic state data at the same time step as the traffic state data at the next future time step) generally follow the same trend. Both exhibit a fluctuating characteristic of first rising and then falling within the 300-3600 s range, reaching peaks around 600 s and 2400 s, with the actual traffic flow at approximately 445 veh / h and the generated traffic flow at approximately 318 veh / h. The traffic flow reaches a trough around 1800 s, with the actual traffic flow at approximately 330 veh / h and the generated traffic flow at approximately 260 veh / h. This indicates that the generation module can follow the actual traffic flow trend relatively well, but the overall amplitude is slightly lower.
[0132] like Figure 3 As shown, Figure 3 This chart compares the performance indicators of the original traffic control system and the autonomous traffic control system (the traffic control system using the simulation control method of this invention). Compared with the original traffic control system (i.e., the existing traffic control system), the autonomous traffic control system (i.e., the traffic control system using the simulation control method of this invention) achieves better performance in terms of average speed (e.g., ...). Figure 3 The overall average maximum queue length is higher in (a) of the figure, as shown in Figure (a). Figure 3 (b) Figure in the middle), average waiting time (e.g. Figure 3 (c) in the figure) and time loss (e.g. Figure 3 The overall lower value in Figure (d) indicates that the simulation control method of the present invention can effectively improve the traffic efficiency of intersections and reduce congestion.
[0133] like Figure 4 As shown, Figure 4 The histogram shows the performance difference distribution between the original traffic control system and the autonomous traffic control system (the traffic control system using the simulation control method of this invention). The overall performance difference distribution of the two traffic control systems favors the autonomous traffic control system, with the mean average speed difference being -0.44 m / s (e.g., ...). Figure 4 (Figure (a)) shows that the average difference in maximum queue length is 7.00 m (as in Figure (a)). Figure 4 (Figure (b) in the figure) shows that the average waiting time difference is 5.53 s (as shown in the figure). Figure 4 In Figure (c), the mean time loss difference is 10.25 s (as shown in Figure 1). Figure 4 Figure (d) illustrates the consistent advantages of autonomous traffic management systems in terms of speed improvement and congestion reduction.
[0134] Example 3
[0135] Based on the same inventive concept as Embodiment 1, this embodiment introduces a regional signal macro-micro integrated simulation control system based on adversarial enhancement reinforcement learning, comprising:
[0136] The perception module is configured to collect lane-level traffic state data, preprocess the collected traffic state data to obtain preprocessed traffic state data, and generate a historical traffic state sequence based on the preprocessed traffic state data and the simulated road network of the area under study. The traffic state data includes the flow rate of the approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate.
[0137] The generation module is configured to input the obtained traffic state sequence into a pre-trained conditional WGAN-GP model to predict the traffic state data for the next time step, i.e., the flow rate of the approach lanes for the next time step.
[0138] The control module is configured to model each intersection in the regional road network as an independent agent based on the QMIX multi-agent reinforcement learning algorithm, wherein a single intersection agent is used to implement lane-level traffic light control; the predicted traffic state data for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output.
[0139] The closed-loop update module is configured to execute the signal timing of each intersection in the next future time step, collect traffic state data after execution, and update the historical traffic state data based on the traffic state data after execution, so as to achieve integrated macro and micro control.
[0140] Example 4
[0141] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning.
[0142] Example 5
[0143] Based on the same inventive concept as other embodiments, this embodiment introduces a computer device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning.
[0144] Example 6
[0145] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning.
[0146] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0147] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.
[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0150] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention, and these modifications are all within the protection scope of the present invention.
Claims
1. A method for integrated macro- and micro-scale simulation control of regional signals based on adversarial reinforcement learning, characterized in that, include: Collect lane-level traffic status data, preprocess the collected traffic status data, and obtain preprocessed traffic status data. Based on the preprocessed traffic state data and the simulated road network of the area under study, a historical traffic state sequence is generated; the traffic state data includes the flow rate of the approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate. The obtained historical traffic state sequence is input into the pre-trained conditional WGAN-GP model to predict the traffic state data for the next time step, that is, the flow rate of the approach lane for the next time step. Based on the QMIX multi-agent reinforcement learning algorithm, each intersection in the regional road network is modeled as an independent agent, where a single intersection agent is used to implement lane-level traffic light control. The traffic state data predicted for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output.
2. The integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning according to claim 1, characterized in that, After outputting the signal timings for each intersection in the next time step, the following steps are also included: The system executes the signal timing for each intersection based on the output in the next future time step, collects traffic state data after execution, and updates the historical traffic state data based on the executed traffic state data.
3. The integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning according to claim 1, characterized in that, The traffic status data at the lane level is collected, and the collected traffic status data is preprocessed to obtain preprocessed traffic status data. Based on the preprocessed traffic state data and the simulated road network of the area under study, a historical traffic state sequence is generated, including: Traffic status data is collected using lane detectors; Outliers were removed from the collected traffic status data using certain criteria, and missing data were filled in using spatiotemporal interpolation methods to obtain preprocessed traffic status data. Based on the simulated road network of the area under study, simulated road network information is extracted; the preprocessed traffic state data and the extracted simulated road network information are fused to obtain a historical traffic state sequence. The simulated road network information includes the center coordinates of intersections, lane geometric attributes, entrance and exit attributes, and topological connections between lanes. The lane geometric attributes include lane length, lane width, and lane orientation. The step of fusing the preprocessed traffic state data and the extracted simulated road network information to obtain a historical traffic state sequence includes: Based on the lane number, the states of each entrance lane within the same intersection are spliced together to form a local state vector. Then, based on the topological connection relationship between lanes, the states of adjacent lanes are mapped or weighted and fused to obtain a historical traffic state sequence.
4. The integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning according to claim 1, characterized in that, The training method for the conditional WGAN-GP model is as follows: Obtain a historical traffic state sequence over N consecutive time steps; The historical traffic state sequence of N consecutive time steps is used as the condition vector, and a random noise vector following a Gaussian distribution is used as the latent variable. The condition vector and the latent variable are concatenated and then input into the generator. The generator uses a fully connected network to map the input condition vector; it outputs the traffic state sequence for the next time step. The discriminator adopts a dual-branch parallel structure, where the global branch is used to evaluate the overall authenticity of the traffic state sequence at the next time step, and the local branch is used to evaluate the accuracy of the traffic state sequence at the last time step at the next time step. Wasserstein distance is used as the optimization objective of generative adversarial networks. A gradient penalty term is introduced to ensure training stability, and a terminal temporal consistency constraint term is added to the discriminator loss. The sum of the discriminator output expectation with a negative generator loss function and the terminal time series consistency constraint term is minimized to generate traffic state data for the next time step.
5. The integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning according to claim 4, characterized in that, The method for obtaining the historical traffic state sequence over N consecutive time steps is as follows: A simulated road network for the area under study is established in a microscopic traffic simulation platform, and lane detectors are deployed on the entrance and exit lanes of each intersection to collect traffic status data. The collected traffic status data is preprocessed by using criteria to remove outliers and spatiotemporal interpolation to fill in missing data, resulting in preprocessed traffic status data. Based on the preprocessed traffic state data, the simulation road network file is parsed to extract the intersection center coordinates, lane geometric attributes, entrance and exit attributes and the topological connection relationship between lanes to obtain time series data. The preprocessed traffic state data of each lane within the same time step are correlated and fused according to the lane spatial location. Specifically, the states of each entrance lane in the same intersection are spliced together to form a local state vector according to the lane number, and the states of adjacent lanes are mapped or weighted and fused according to the topological connection relationship between lanes to obtain time series data. Arrange the time-series data within N consecutive time steps in chronological order to form a historical traffic state sequence; the historical traffic state sequence includes historical traffic flow sequence, historical average vehicle delay sequence, historical maximum queue length sequence, historical average vehicle parking time sequence, and historical vehicle occupancy rate sequence.
6. The integrated macro-micro simulation control method for regional signals based on adversarial reinforcement learning according to claim 1, characterized in that, The QMIX multi-agent reinforcement learning algorithm models each intersection in the regional road network as an independent agent, where a single intersection agent is used to implement lane-level traffic light control. The algorithm inputs the predicted traffic state data for the next time step into the QMIX multi-agent reinforcement learning algorithm and outputs the signal timing for each intersection in the next time step, including: Each intersection in the regional road network is modeled as an independent intelligent agent. The local state of each intelligent agent includes the traffic flow of each approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate of the intersection. The global state is defined as the summary information of the local states of each intersection in the region. Each agent uses a deep Q-network to fit a local Q-value function. The input of the deep Q-network is the local state, and the output is the Q value corresponding to each action. The action space includes two types of actions: the first type is to extend the duration of the current signal phase, and the second type is to switch to the next signal phase. The reward function is designed as a weighted sum of average queue length, average vehicle parking time, and regional congestion level indicators. Minimizing this weighted sum is the optimization objective. The reward function is expressed as: ； in, The average queue length, This represents the average vehicle parking time. As an indicator of regional congestion level, These are the weighting coefficients. Represents the reward function; A hybrid network is used to combine the local Q-values of each agent into a global Q-value. The hybrid network generates the weight matrix and bias vector from the supernetwork, and achieves a monotonic mapping from the local Q-value to the global Q-value through two layers of nonlinear transformation. Its expression is: ； in, The global action value function; For the first The local Q-value function of an agent; For the first The observation history of the first agent, i.e., the first... Traffic state sequence of an intersection; For the first The corresponding action of the first agent, i.e. the first... Signal phase control actions at each intersection; This is the global state. The monotonic mixture function generated by the hypernetwork, This represents the joint observation history of all agents. This represents the joint action of all intelligent agents. This represents the total number of intelligent agents, which is the total number of intersections in the regional road network; A centralized training and distributed execution framework is adopted. Training is performed through experience replay, where an experience replay buffer stores the joint experience of all agents. The loss function is constructed using the TD error, and the parameters of each agent network and the hybrid network are jointly updated through the Adam optimizer. Represented as: ； In the formula, To replay the experience pool The mathematical expectation of the joint experience of random sampling in the middle, The time difference target value, This is the global action value function output by the current hybrid network; These are the current network parameters; The time difference objective value is: ； In the formula, The global action value function output by the target hybrid network; This is the candidate joint action vector for all agents in the next time step; This will serve as the joint observation history of all agents for the next time step; This represents the global state at the next time step. For global, instant rewards; Discount factor; The current network parameters are softly updated to the target network every fixed number of steps to ensure training stability. During execution, each agent independently selects the action that maximizes the local Q value based on its local state to obtain the optimal action combination.
7. A regional signal macro-micro integrated simulation control system based on adversarial reinforcement learning, characterized in that, The method for implementing the integrated macro-micro simulation control of regional signals based on adversarial enhancement reinforcement learning as described in any one of claims 1 to 6 includes: The perception module is configured to collect lane-level traffic state data, preprocess the collected traffic state data to obtain preprocessed traffic state data, and generate a historical traffic state sequence based on the preprocessed traffic state data and the simulated road network of the area under study. The traffic state data includes the flow rate of the approach lane, average vehicle delay, maximum queue length, average vehicle stopping time, and vehicle occupancy rate. The generation module is configured to input the obtained traffic state sequence into a pre-trained conditional WGAN-GP model to predict the traffic state data for the next time step, i.e., the flow rate of the approach lanes for the next time step. The control module is configured to model each intersection in the regional road network as an independent agent based on the QMIX multi-agent reinforcement learning algorithm, wherein a single intersection agent is used to implement lane-level traffic light control; the predicted traffic state data for the next time step is input into the QMIX multi-agent reinforcement learning algorithm, and the signal timing of each intersection in the next time step is output.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning as described in any one of claims 1 to 6.
9. A computer device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the regional signal macro-micro integrated simulation control method based on adversarial reinforcement learning as described in any one of claims 1 to 6.