A signal timing optimization method and system based on reinforcement learning

Abstract

The application provides a signal timing optimization method and system based on reinforcement learning, which comprises the following steps: collecting intersection multi-modal traffic data through a radar and video integrated machine, performing abnormality inspection and interpolation repair on the data, and obtaining standardized traffic flow data; constructing an AI traffic flow prediction model with an encoder-decoder as the backbone, containing a multi-scale spatio-temporal graph convolution network and a double uncertainty quantification module, and outputting phase-level flow prediction values, queue length prediction values and their confidence intervals for future signal periods; designing a multi-objective oriented reinforcement learning agent, containing a scene judgment module, a hierarchical composite action space and a reward function with dynamic weight adjustment, and generating an optimal signal timing scheme under preset constraints. The application realizes the transformation of signal timing from passive response to active foresight, effectively deals with complex scenarios such as traffic flow mutation, oversaturation queuing and bad weather, and significantly improves the intersection passing efficiency and operation safety.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent traffic signal control and artificial intelligence, specifically relating to a signal timing optimization method and system based on reinforcement learning. Background Technology

[0002] Traffic signal timing optimization is a key technology for efficiently managing traffic flow at intersections or in areas by scientifically adjusting core parameters such as traffic signal phase division, green light duration, and red light interval. Existing technologies are mainly divided into three categories: multi-time period fixed timing control, inductive control, and adaptive control that integrates basic AI with radar-visual integrated machines.

[0003] Currently, adaptive control schemes still face the following technical bottlenecks: First, the advantages of radar-visual perception have not been fully realized, the spatiotemporal synchronization calibration of radar and video data is lacking, and data is prone to interruption or distortion under severe weather conditions, resulting in large errors in parameters such as queue length; Second, the collaboration between predictive AI and decision-making is disconnected, most schemes do not incorporate short-term traffic flow prediction, timing adjustments are lagging, and the prediction model does not quantify uncertainty, leading to a high risk of misjudgment in reinforcement learning decisions; Third, there is a lack of deep regional collaborative optimization, and the reward function only focuses on the efficiency of motor vehicle traffic, ignoring the needs of pedestrians, non-motorized vehicles, and special vehicles.

[0004] Therefore, how to achieve deep collaboration between high-precision perception, forward-looking prediction, and multi-objective decision-making is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a signal timing optimization method and system based on reinforcement learning. Through deep collaboration between multimodal data fusion from a radar-visual integrated machine, AI traffic flow prediction with uncertainty quantification, and multi-objective reinforcement learning decision-making, the method achieves a shift from passive response to proactive prediction in signal timing.

[0006] In a first aspect, the present invention provides the following technical solution: a signal timing optimization method based on reinforcement learning, comprising: Multimodal traffic data at intersections are collected using a radar-visual integrated machine. The multimodal traffic data includes lane-specific flow, speed, headway, queue length, and pedestrian and non-motorized vehicle crossing status. The multimodal traffic data is subjected to anomaly detection and interpolation repair to obtain standardized traffic flow data; An AI traffic flow prediction model is constructed, which outputs predicted data of future signal cycles based on the standardized traffic flow data. The AI ​​traffic flow prediction model has an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. Design a multi-objective-oriented reinforcement learning agent, based on the standardized traffic flow data and the predicted data, and generate an optimal signal timing scheme under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ; In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

[0007] Compared with existing technologies, the beneficial effects of this invention are as follows: First, by designing an anomaly real-time detection method for radar-visual integrated machines (covering situations such as missing data, continuous zero traffic flow, PCU traffic exceeding limits, and queue length exceeding limits) and an adaptive interpolation method, the problem of data interruption or distortion under severe weather conditions is effectively solved, significantly improving the accuracy and data continuity of core parameters such as queue length and lane-specific traffic flow. Second, an AI traffic flow prediction model integrating multi-scale spatiotemporal graph convolution and dual uncertainty quantization (deep integration and Monte Carlo Dropout) is constructed, which can output phase-level traffic flow and queue length for future signal cycles. The predicted length and its confidence interval enable a shift from passive response to proactive prediction in signal timing, significantly reducing the risk of decision-making errors. Finally, a multi-objective reinforcement learning agent based on dynamic scene determination (off-peak / peak / nighttime) is designed. Employing a hierarchical composite action space and a reward function with dynamic weight adjustment, it comprehensively optimizes multiple objectives such as queue reduction, traffic capacity, green light utilization, and energy consumption under safety, operational, and preventive constraints. This effectively balances the traffic demands of motor vehicles, pedestrians, non-motor vehicles, and special vehicles, overcoming the limitations of traditional methods in dealing with sudden changes in traffic flow, oversaturated queuing, and multi-user collaboration.

[0008] Preferably, the step of performing anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data includes: If the number of missing data per unit time exceeds the preset value, or if the single lane traffic flow remains at zero for an extended period of time... Exceeding the time period threshold or single-lane PCU flow Greater than the maximum allowed flow rate of the period or maximum queue length per lane Greater than the maximum allowable queue length at the entrance lane If the data is deemed abnormal, an interpolation algorithm based on adjacent period data is used to repair it in order to obtain standardized traffic flow data. The duration for: ; In the formula, This refers to the point in time when the current counted flow is all zero. The most recent valid data point that is not all zeros; The maximum allowable flow rate for the period for: ; In the formula, For lane saturation flow, The supersaturation coefficient. The duration of the green light within the cycle. The duration of the cycle; The maximum allowable queuing length at the entrance lane for: ; In the formula, Design length of the intersection approach lane A reasonable threshold; The single-lane PCU flow rate for: ; In the formula, This refers to the number of small vehicles counted within the period. This refers to the number of medium-sized vehicles counted within the period. The number of heavy-duty vehicles counted within the period; The maximum queue length of a single lane for: ; In the formula, The radar-visual integrated machine detected the length of a certain lane. This refers to the distance between the installation location of the radar-guided all-in-one machine and the parking line of a certain lane. This refers to the length of the blind zone of the integrated radar vision system.

[0009] Preferably, the multi-scale spatiotemporal graph convolutional network includes a spatiotemporal feature extraction module, a multi-scale fusion module, and a time series prediction module; The spatiotemporal feature extraction module uses a graph convolutional network to capture spatial dependencies in the road network topology. The graph convolution operation is defined as follows: ; In the formula, For the first The feature matrix of all nodes in the layer, For degree matrix, To add self-joins to the adjacency matrix, satisfying... ,in This is the original adjacency matrix. It is the identity matrix. For the first The feature matrix of the layer, For the first The trainable weight matrix of the layer, For activation functions; The multi-scale fusion module extracts features at different time scales through parallel one-dimensional dilated convolutional layers, and its fusion operation is defined as follows: ; In the formula, For a moment The output characteristics, The kernel size is [size]. For the first The convolution kernel weights at each position, The input feature sequence at time... The value, Void ratio; The time-series prediction module combines a gated recurrent unit with an attention mechanism to capture the temporal dependencies of the sequence and dynamically monitor key historical information. The update formula for the gated recurrent unit is as follows: ; In the formula, These are updating the gate vector and resetting the gate vector, respectively. In the candidate hidden state, The current hidden state. They are respectively The corresponding trainable weight matrix, Input the features at the current time. The Sigmoid activation function is used. For Hadamard product; The input feature dimension of the AI ​​traffic flow prediction model is: ,in, For batch size, Given the length of the input sequence, The feature dimension for each moment; the AI ​​traffic flow prediction model outputs the feature dimension as follows: ,in, To predict the step size, For the number of phases, Indicates output phase-level predicted flow rate With queue length Two characteristic indicators.

[0010] Preferably, the dual uncertainty quantification module employs a combination of deep integration and Monte Carlo Dropout. Prediction results for: ; In the formula, The number of independently trained models. For the first The weights of the prediction results of each independent model. For the first Individual model prediction results; Variance of prediction results for cognitive uncertainty through multiple forward propagations in Monte Carlo Dropout Quantification: ; In the formula, For the number of samples, For the first The model prediction result value of the second sampling. for The mean of the prediction results from each sample; Data uncertainty is modeled using heteroscedasticity regression, with the model additionally outputting a variance term. The overall variance is The confidence interval is ,in, This represents the quantiles corresponding to the confidence level.

[0011] Preferably, the scene determination module uses a single signal cycle as the analysis window and introduces phase saturation. As a core quantitative indicator: ; In the formula, For phase saturation, For the actual flow rate of the phase, Phase saturation flow rate; The scene determination module determines the phase saturation based on the... Time period characteristics and queue length The current intersection operation scenario is determined as off-peak, peak, or nighttime. The criteria for determining off-peak scenarios are: simultaneously satisfying the following conditions: the current time period is within the daytime range of 9:00-17:00 or the nighttime range of 19:30-21:30, and all phase saturation is... Furthermore, there was no sustained surge in traffic. The criteria for determining peak scenarios are: the current time period is within the morning peak (7:00-9:00) or evening peak (17:00-19:00), or any phase saturation. or the maximum queue length of any entrance lane Reaching or exceeding at least one of the set peak queue length thresholds; The criteria for determining a nighttime scene are: simultaneously satisfying the conditions that the current time period is within the time interval of 22:00 to 6:00 the next day, and that all phase saturation... Furthermore, there was no sustained surge in traffic.

[0012] Preferably, the reward function Rewards Including queue reduction rewards Traffic capacity rewards Green light utilization reward and energy consumption optimization rewards ; Queue reduction reward for: ; In the formula, For the first Within the cycle The maximum queue length of the real-time phase level for each phase. The total number of phases, The penalty coefficient is... The first output of the prediction model The first cycle The queue length of each phase, The confidence level for the queue length output by the prediction model; The traffic capacity reward for: ; In the formula, Rewards are based on actual traffic volume. To predict matching rewards, The weighting coefficient for the actual traffic reward item. To predict the weighting coefficients of the matching reward items; Among them, the actual traffic reward for: ; In the formula, For the first Within the cycle Real-time phase-level PCU traffic for each phase For the first Saturation flow rate of each phase, For the first Within the cycle The duration of the green light for each phase, For the first Total duration of the cycle; The predictive matching reward for: ; In the formula, To predict the confidence level of the flow output by the model, The first output of the prediction model The first cycle PCU flow rate for each phase; The green light utilizes rewards for: ; In the formula, For the first The idle time during which no vehicles pass through during the green light period of each phase. For the first Total green light duration for each phase; The energy consumption optimization reward The calculation formula is: ; In the formula, This represents the number of signal phase transitions within a period. The total duration is calculated for each period.

[0013] Preferably, the preset constraints include safety constraints, operational constraints, and prevention constraints; The safety constraints include at least: minimum green light time for the phase. Constraints are used to ensure that pedestrians can safely cross the street and that queuing vehicles can safely start and pass the stop line; minimum clearing time. Constraints are used to set a total duration of not less than [a certain value] between any two conflicting phase transitions. The interval time ensures that the last vehicle in the previous phase can safely leave the conflict zone; the priority passage response constraint is used to ensure that when a special event marker in the state space is activated, the agent must trigger the corresponding special priority mode phase sequence in the next signal cycle. The operational constraints include at least: maximum green light time for a given phase. Constraints are used to prevent a single phase from excessively occupying right-of-way; cycle duration range constraints are used to limit the total duration of the signal cycle to a reasonable range; cycle change smoothness constraints are used to limit the variation in the total duration of adjacent cycles; phase structure and sequence constraints are used to enforce the phase logic relationship that conforms to the pre-set principle of traffic conflict elimination. The preventative constraints include at least: queue overflow avoidance constraints and flow mutation adaptive adjustment constraints, wherein the queue overflow avoidance constraints are used to adjust the predicted queue length. and confidence level For queues whose predicted length exceeds the maximum allowable queue length at the entrance lane The action is pre-judged and avoided; the adaptive adjustment constraint for traffic mutation is used to adjust the predicted traffic flow. and confidence level The action and reward function are dynamically adjusted when the predicted traffic flow changes significantly.

[0014] Secondly, the present invention provides the following technical solution: a signal timing optimization system based on reinforcement learning, the system comprising: The data acquisition module is used to collect multimodal traffic data at intersections through the radar-visual integrated machine. The multimodal traffic data includes lane-specific flow, speed, headway, queue length, and pedestrian and non-motorized vehicle crossing status. The processing module is used to perform anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data; The construction module is used to build an AI traffic flow prediction model, which outputs prediction data of future signal cycles based on the standardized traffic flow data and according to the AI ​​traffic flow prediction model. The AI ​​traffic flow prediction model has an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. The reinforcement learning module is used to design a multi-objective-oriented reinforcement learning agent. Based on the standardized traffic flow data and the predicted data, the reinforcement learning agent generates an optimal signal timing scheme under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ; In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

[0015] Thirdly, the present invention provides the following technical solution: a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the signal timing optimization method based on reinforcement learning as described above.

[0016] Fourthly, the present invention provides the following technical solution: a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the signal timing optimization method based on reinforcement learning as described above. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the signal timing optimization method based on reinforcement learning provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the design process of a reinforcement learning agent as provided in Embodiment 1 of the present invention. Figure 3 This is a block diagram of the signal timing optimization system based on reinforcement learning provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the hardware structure of a computer provided for another embodiment of the present invention.

[0019] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the present invention, and should not be construed as limiting the present invention.

[0021] Example 1 In Embodiment 1 of the present invention, as Figure 1 As shown, a signal timing optimization method based on reinforcement learning includes: S1. Collect multimodal traffic data of the intersection through the radar-visual integrated machine, wherein the multimodal traffic data includes lane flow, speed, headway, queue length and pedestrian and non-motorized vehicle crossing status; Specifically, the integrated radar-visual unit is deployed on the existing traffic light poles at the intersection, covering both oncoming and outgoing traffic. Parameters are initialized via a web interface, including: the number of detection lanes (3-4 lanes per approach lane), the location of virtual detection lines (30m, 50m, and 80m from the stop line), and the data upload frequency (2 seconds / upload). The preview function verifies the completeness of the detection area coverage, ensuring that all lanes are effectively covered. The radar-view integrated machine outputs multiple traffic parameters every 2 seconds: lane flow (small cars, medium cars, and heavy cars are counted separately), speed (instantaneous speed and average speed), headway (time interval between adjacent vehicles passing through the same section), queue length (distance from the stop line to the last stationary vehicle), and pedestrian and non-motorized vehicle crossing status (presence, number, and waiting time). All data are associated with intersection ID, device number, lane number, and timestamp to form a standardized data table, which is transmitted to the edge computing unit through a switch. Specifically, after the raw data is entered into the database, a statistical data table is generated every 5 minutes. First, the PCU (standard passenger car equivalent) conversion is performed, followed by anomaly detection and interpolation repair.

[0022] S2. Perform anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data; Specifically, step S2 is as follows: If the number of missing data per unit time exceeds the preset value, or if the single lane traffic flow remains at zero for an extended period of time... Exceeding the time period threshold or single-lane PCU flow Greater than the maximum allowed flow rate of the period or maximum queue length per lane Greater than the maximum allowable queue length at the entrance lane If the data is deemed abnormal, an interpolation algorithm based on adjacent period data is used to repair it in order to obtain standardized traffic flow data. Specifically, if the number of missing data points exceeds 10 within a unit of time (1 minute) (i.e., no valid data for 20 consecutive seconds), it will be directly judged as a data anomaly, and the subsequent signal timing update task will be suspended. The duration for: ; In the formula, This refers to the point in time when the current counted flow is all zero. The most recent valid data point that is not all zeros; The maximum allowable flow rate for the period for: ; In the formula, For lane saturation flow, The supersaturation coefficient. The duration of the green light within the cycle. The duration of the cycle; The maximum allowable queuing length at the entrance lane for: ; In the formula, Design length of the intersection approach lane A reasonable threshold; The single-lane PCU flow rate for: ; In the formula, This refers to the number of small vehicles counted within the period. This refers to the number of medium-sized vehicles counted within the period. The number of heavy-duty vehicles counted within the period; The maximum queue length of a single lane for: ; In the formula, The radar-visual integrated machine detected the length of a certain lane. This refers to the distance between the installation location of the radar-guided all-in-one machine and the parking line of a certain lane. This refers to the length of the blind zone of the integrated radar vision system; Meanwhile, the data interpolation process is as follows: Traffic data interpolation When no more than 10 data points are missing within a unit of time (1 minute), handle them according to the degree of missingness: Scenario A: There are fewer than 2 valid data points during the current green light period, but the data during the green light periods of the preceding and following periods are complete and usable. Then the interpolated value is: ; In the formula, This is the flow interpolation value every 2 seconds during the current green light period, in units of pcu / 2 seconds; This refers to the traffic data for every 2 seconds during the green light period of the previous adjacent cycle; This represents the traffic data for every 2 seconds during the green light period of the next adjacent cycle. During the red light period, all data points are uniformly assigned a value of 0.

[0023] Scenario B: There are at least two valid data points during the current green light period. In this case, adjacent point data interpolation is used. ; In the formula, This represents the 2-second data points missing during the current green light period, expressed in pcu / 2 seconds. This refers to the previous valid 2-second data point; This is the next valid 2-second data point. Data points during the red light period are assigned a value of 0.

[0024] Queue length interpolation For the missing maximum queue length of a cycle, imputation is performed using the maximum queue length of the preceding and following valid cycles: ; In the formula, This represents the maximum missing queue length within the current period, in meters. The maximum queue length in the previous valid period, in meters; This represents the maximum queue length for the next valid cycle, expressed in meters.

[0025] S3. Construct an AI traffic flow prediction model, based on the standardized traffic flow data and outputting prediction data for future signal cycles according to the AI ​​traffic flow prediction model. The AI ​​traffic flow prediction model uses an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. Specifically, regarding the overall architecture of the AI ​​traffic flow prediction model, the encoder consists of a spatiotemporal feature extraction module and a multi-scale fusion module. The decoder consists of a time-series prediction module (using a gated recurrent unit (GRU) and an attention mechanism). An uncertainty quantification module is designed between the encoder and decoder. The model input is standardized traffic flow data for six historical cycles (30 minutes), and the output is the phase-level traffic flow prediction values ​​for the next two cycles. And queue length prediction It also outputs the corresponding confidence interval.

[0026] The multi-scale spatiotemporal graph convolutional network includes a spatiotemporal feature extraction module, a multi-scale fusion module, and a time series prediction module. The spatiotemporal feature extraction module uses a graph convolutional network to capture spatial dependencies in the road network topology. The graph convolution operation is defined as follows: ; In the formula, For the first The feature matrix of all nodes in the layer, For degree matrix, To add self-joins to the adjacency matrix, satisfying... ,in This is the original adjacency matrix. It is the identity matrix. For the first The feature matrix of the layer, For the first The trainable weight matrix of the layer, For activation functions; Specifically, the spatiotemporal feature extraction module models each approach lane of the intersection as a directed graph: nodes represent lanes, and edges represent traffic flow direction relationships (straight, left turn, right turn). This operation enables each lane node to aggregate traffic state information of its upstream neighbor lanes, thereby capturing spatial dependencies formed by traffic flow turning, queue propagation, etc. The multi-scale fusion module extracts features at different time scales through parallel one-dimensional dilated convolutional layers, and its fusion operation is defined as follows: ; In the formula, For a moment The output characteristics, The kernel size is [size]. For the first The convolution kernel weights at each position, The input feature sequence at time... The value, Void ratio; Specifically, to simultaneously capture short-term fluctuations, medium-term trends, and long-term periodicity of traffic flow, a parallel one-dimensional dilated convolutional layer is employed. This patent sets up three parallel branches with a dilation rate of [missing information]. Take values ​​of 1, 3, and 5 respectively. Capture subtle changes (short-term fluctuations) at nearby time points. Capture trends over a medium time span (such as changes within a complete signal cycle). It captures patterns with longer periods (such as cumulative effects across multiple periods). The outputs of the three branches are concatenated and linearly transformed to fuse into a multi-scale feature vector.

[0027] The time-series prediction module combines a gated recurrent unit with an attention mechanism to capture the temporal dependencies of the sequence and dynamically monitor key historical information. The update formula for the gated recurrent unit is as follows: ; In the formula, These are updating the gate vector and resetting the gate vector, respectively. In the candidate hidden state, The current hidden state. They are respectively The corresponding trainable weight matrix, Input the features at the current time. The Sigmoid activation function is used. For Hadamard product; Specifically, the time-series prediction module combines a gated recurrent unit (GRU) with an attention mechanism. The GRU unit controls the retention and forgetting of historical information by updating and resetting gates, effectively capturing the long-term and short-term dependencies of traffic flow sequences. The attention mechanism dynamically calculates the weights of each hidden state of the encoder during decoding at each time step, enabling the model to focus on the key segments in the history most relevant to the current prediction. The formula for calculating the attention weights is: ; In the formula, For decoding time For the encoder The attention weights of each hidden state are dimensionless. For the decoder at time The hidden state; For the encoder in the first The hidden state at each moment; The total time step of the input sequence for the encoder; An additive attention mechanism is used as the alignment function. The decoder generates the predicted sequences for the next two periods in an autoregressive manner.

[0028] The input feature dimension of the AI ​​traffic flow prediction model is: ,in, For batch size, Given the length of the input sequence, The feature dimension for each moment; the AI ​​traffic flow prediction model outputs the feature dimension as follows: ,in, To predict the step size, For the number of phases, Indicates output phase-level predicted flow rate With queue length Two characteristic indicators.

[0029] Regarding uncertainty, this patent introduces two types of uncertainty: cognitive uncertainty (model parameter uncertainty) and data uncertainty (uncertainty caused by noise). The dual uncertainty quantification module employs a combination of deep integration and Monte Carlo Dropout. Prediction results for: ; In the formula, The number of independently trained models. For the first The weights of the prediction results of each independent model. For the first Individual model prediction results; Specifically, deep ensemble: independent training Several models with the same structure are used, each with different random initialization parameters and training data sampling (Bootstrap). The final prediction result is a weighted average of the outputs of all models. ; Variance of prediction results for cognitive uncertainty through multiple forward propagations in Monte Carlo Dropout Quantification: ; In the formula, For the number of samples, For the first The model prediction result value of the second sampling. for The mean of the prediction results from each sample; Specifically, Monte Carlo Dropout: Within each independent model, the Dropout layer is kept active during the inference phase. After one random forward propagation, we obtain A prediction result; cognitive uncertainty through this The variance quantification of each outcome is... ; Data uncertainty is modeled using heteroscedasticity regression, with the model additionally outputting a variance term. The overall variance is The confidence interval is ,in, The quantiles corresponding to the confidence level; Specifically, when the confidence interval width exceeds a set threshold (such as 30% of the predicted value), the model automatically reduces the weight of the predicted result in subsequent reinforcement learning decisions.

[0030] like Figure 2 As shown, S4, design a multi-objective-oriented reinforcement learning agent, based on the standardized traffic flow data and the predicted data, and generate the optimal signal timing scheme according to the reinforcement learning agent under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. Specifically, this step involves constructing a reinforcement learning agent based on the aforementioned predicted and real-time data to achieve dynamic optimization of signal timing.

[0031] The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ; In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

[0032] For the state space, real-time traffic state characteristics include flow rate at each phase, queue length at each phase, and headway statistics (mean, variance); predicted traffic state characteristics include predicted flow rate. and its confidence level, predicted queue length The environmental state features include time period features (one-hot encoding), intersection type (encoding), date type (weekday / weekend / holiday), and special event markers (Boolean values); historical decision performance features include historical action sequences (the last 5 actions) and reward function change trends (reward values ​​for each item in the last 5 periods); after standardization (Z-score normalization) and serialization, all features are combined to form a unified high-dimensional feature vector, which is then input into the reinforcement learning policy network. This patent designs a hierarchical composite action space, enabling the agent to make decisions at different granularities.

[0033] Discrete Action Layer: For the current execution phase, within the minimum safe time ( Under the premise of (seconds), the agent can adjust the green light duration in real time. Action options are: shorten by 3 seconds, shorten by 1 second, keep unchanged, extend by 1 second, and extend by 3 seconds. The adjustment range is subject to... and (Maximum green light time, set to 60 seconds) constraint.

[0034] Continuous Action Layer: Within permissible conditions, the agent can continuously adjust the total duration of the signal cycle. The period duration adjustment range is... The range of change between adjacent cycles is limited to Within seconds to ensure smoothness.

[0035] Structured Action Layer: The agent can select a matching phase release pattern from a predefined optimal phase sequence library; the phase sequence library includes: Two-phase basic mode: suitable for small intersections with low traffic volume and simple conflicts in off-peak scenarios; Four-phase symmetrical release: suitable for intersections with balanced flow under normal off-peak or medium traffic scenarios; Four-phase single-entry release: suitable for intersections with significantly high traffic flow at a single approach lane in peak or oversaturated scenarios; Overlapping phase mode: Suitable for intersections with high turning traffic and complex demands, it efficiently handles conflicting traffic flows by overlapping green light times during peak or specific periods; Special Priority Mode: Automatically triggered when a special priority passage request is detected, such as for buses or emergency vehicles.

[0036] Wherein, the reward function Rewards Including queue reduction rewards Traffic capacity rewards Green light utilization reward and energy consumption optimization rewards ; Queue reduction reward for: ; In the formula, For the first Within the cycle The maximum queue length of the real-time phase level for each phase. The total number of phases, The penalty coefficient is... The first output of the prediction model The first cycle The queue length of each phase, The confidence level for the queue length output by the prediction model; Specifically, the reward not only penalizes the current queue, but also applies penalties in advance based on the predicted queue length, with the lower the confidence level, the smaller the impact of the penalty.

[0037] The traffic capacity reward for: ; In the formula, Rewards are based on actual traffic volume. To predict matching rewards, The weighting coefficient for the actual traffic reward item. To predict the weighting coefficients of the matching reward items; Among them, the actual traffic reward for: ; In the formula, For the first Within the cycle Real-time phase-level PCU traffic for each phase For the first Saturation flow rate of each phase, For the first Within the cycle The duration of the green light for each phase, For the first Total duration of the cycle; Specifically, the reward is used to assess the utilization saturation of the green light time in the current cycle.

[0038] The predictive matching reward for: ; In the formula, To predict the confidence level of the flow output by the model, The first output of the prediction model The first cycle PCU flow rate for each phase; Specifically, the reward is used to encourage intelligent agents to prepare right-of-way for upcoming traffic flows in advance.

[0039] The green light utilizes rewards for: ; In the formula, For the first The idle time during which no vehicles pass through during the green light period of each phase. For the first Total green light duration for each phase; Specifically, the reward incentivizes the algorithm to reduce the time the green light is left idle.

[0040] The energy consumption optimization reward The calculation formula is: ; In the formula, This represents the number of signal phase transitions within a period. The total duration is calculated for each period. Specifically, the reward incentivizes the algorithm to reduce energy consumption caused by frequent phase switching.

[0041] The scene determination module uses a single signal cycle as the analysis window and introduces phase saturation. As a core quantitative indicator: ; In the formula, For phase saturation, For the actual flow rate of the phase, Phase saturation flow rate; Meanwhile, dynamic weights The weighting is adjusted in real time based on the scenario assessment results: For off-peak scenarios, the weighting adjustment focuses on green light utilization rewards (the green light utilization rewards have a higher weight); for peak scenarios, the weighting adjustment focuses on queue reduction rewards and traffic capacity rewards (the queue reduction rewards and traffic capacity rewards have a higher weight); for nighttime scenarios, the weighting adjustment focuses on energy consumption optimization rewards (the energy consumption optimization rewards have a higher weight). The scene determination module determines the phase saturation based on the... Time period characteristics and queue length The current intersection operation scenario is determined as off-peak, peak, or nighttime. The criteria for determining off-peak scenarios are: simultaneously satisfying the following conditions: the current time period is within the daytime range of 9:00-17:00 or the nighttime range of 19:30-21:30, and all phase saturation is... Furthermore, there was no sustained surge in traffic. The criteria for determining peak scenarios are: the current time period is within the morning peak (7:00-9:00) or evening peak (17:00-19:00), or any phase saturation. or the maximum queue length of any entrance lane Reaching or exceeding at least one of the set peak queue length thresholds; The criteria for determining a nighttime scene are: simultaneously satisfying the conditions that the current time period is within the time interval of 22:00 to 6:00 the next day, and that all phase saturation... Furthermore, there was no sustained surge in traffic.

[0042] The preset constraints include safety constraints, operational constraints, and prevention constraints. The safety constraints include at least: minimum green light time for the phase. Constraints are used to ensure that pedestrians can safely cross the street and that queuing vehicles can safely start and pass the stop line; minimum clearing time. Constraints are used to set a total duration of not less than [a certain value] between any two conflicting phase transitions. The interval time ensures that the last vehicle in the previous phase can safely leave the conflict zone; the priority passage response constraint is used to ensure that when a special event marker in the state space is activated, the agent must trigger the corresponding special priority mode phase sequence in the next signal cycle. The operational constraints include at least: maximum green light time for a given phase. Constraints are used to prevent a single phase from excessively occupying right-of-way; cycle duration range constraints are used to limit the total duration of the signal cycle to a reasonable range; cycle change smoothness constraints are used to limit the variation in the total duration of adjacent cycles; phase structure and sequence constraints are used to enforce the phase logic relationship that conforms to the pre-set principle of traffic conflict elimination. The preventative constraints include at least: queue overflow avoidance constraints and flow mutation adaptive adjustment constraints, wherein the queue overflow avoidance constraints are used to adjust the predicted queue length. and confidence level For queues whose predicted length exceeds the maximum allowable queue length at the entrance lane The action is pre-judged and avoided; the adaptive adjustment constraint for traffic mutation is used to adjust the predicted traffic flow. and confidence level The action and reward function are dynamically adjusted when the predicted traffic flow changes significantly. Specifically, any action that violates the hard constraints will be considered invalid, and the agent will automatically adopt the default safety policy or regenerate the action, thereby ensuring the reliability and security of the control agent.

[0043] The signal timing optimization method based on reinforcement learning provided in Embodiment 1 of this invention has the following advantages compared to existing technologies: First, by designing anomaly real-time detection methods for radar-visual integrated machines (covering situations such as missing data, continuous zero traffic flow, PCU traffic exceeding limits, and queue length exceeding limits) and adaptive interpolation methods, the problem of data interruption or distortion under severe weather conditions is effectively solved, significantly improving the accuracy and data continuity of core parameters such as queue length and lane-specific traffic flow. Second, an AI traffic flow prediction model integrating multi-scale spatiotemporal graph convolution and dual uncertainty quantization (deep integration and Monte Carlo Dropout) is constructed, capable of outputting future... The predicted values ​​and confidence intervals of phase-level traffic flow and queue length during the signal cycle realize the transformation of signal timing from passive response to proactive prediction, significantly reducing the risk of decision-making errors. Finally, a multi-objective reinforcement learning agent based on dynamic scene determination (off-peak / peak / nighttime) is designed. It adopts a hierarchical composite action space and a reward function with dynamic weight adjustment to comprehensively optimize multiple objectives such as queue reduction, traffic capacity, green light utilization, and energy consumption under safety, operation, and prevention constraints. It effectively balances the traffic demand of motor vehicles, pedestrians, non-motor vehicles, and special vehicles, and overcomes the limitations of traditional methods in dealing with sudden changes in traffic flow, oversaturated queues, and multi-user collaboration.

[0044] Example 2 like Figure 3 As shown, in Embodiment 2 of the present invention, a signal timing optimization system based on reinforcement learning is provided, the system comprising: The data acquisition module 1 is used to acquire multimodal traffic data of the intersection through the radar-visual integrated machine. The multimodal traffic data includes lane flow, speed, headway, queue length, and pedestrian and non-motorized vehicle crossing status. Processing module 2 is used to perform anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data; Module 3 is used to build an AI traffic flow prediction model. Based on the standardized traffic flow data, the AI ​​traffic flow prediction model outputs prediction data of future signal cycles. The AI ​​traffic flow prediction model has an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. Reinforcement learning module 4 is used to design a multi-objective-oriented reinforcement learning agent, which generates an optimal signal timing scheme based on the standardized traffic flow data and the predicted data and under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ; In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

[0045] Specifically, the processing module 2 is used for: If the number of missing data per unit time exceeds the preset value, or if the single lane traffic flow remains at zero for an extended period of time... Exceeding the time period threshold or single-lane PCU flow Greater than the maximum allowed flow rate of the period or maximum queue length per lane Greater than the maximum allowable queue length at the entrance lane If the data is deemed abnormal, an interpolation algorithm based on adjacent period data is used to repair it in order to obtain standardized traffic flow data. The duration for: ; In the formula, This refers to the point in time when the current counted flow is all zero. The most recent valid data point that is not all zeros; The maximum allowable flow rate for the period for: ; In the formula, For lane saturation flow, The supersaturation coefficient. The duration of the green light within the cycle. The duration of the cycle; The maximum allowable queuing length at the entrance lane for: ; In the formula, Design length of the intersection approach lane A reasonable threshold; The single-lane PCU flow rate for: ; In the formula, This refers to the number of small vehicles counted within the period. This refers to the number of medium-sized vehicles counted within the period. The number of heavy-duty vehicles counted within the period; The maximum queue length of a single lane for: ; In the formula, The radar-visual integrated machine detected the length of a certain lane. This refers to the distance between the installation location of the radar-guided all-in-one machine and the parking line of a certain lane. This refers to the length of the blind zone of the integrated radar vision system.

[0046] The multi-scale spatiotemporal graph convolutional network includes a spatiotemporal feature extraction module, a multi-scale fusion module, and a time series prediction module. The spatiotemporal feature extraction module uses a graph convolutional network to capture spatial dependencies in the road network topology. The graph convolution operation is defined as follows: ; In the formula, For the first The feature matrix of all nodes in the layer, For degree matrix, To add self-joins to the adjacency matrix, satisfying... ,in This is the original adjacency matrix. It is the identity matrix. For the first The feature matrix of the layer, For the first The trainable weight matrix of the layer, For activation functions; The multi-scale fusion module extracts features at different time scales through parallel one-dimensional dilated convolutional layers, and its fusion operation is defined as follows: ; In the formula, For a moment The output characteristics, The kernel size is [size]. For the first The convolution kernel weights at each position, The input feature sequence at time... The value, Void ratio; The time-series prediction module combines a gated recurrent unit with an attention mechanism to capture the temporal dependencies of the sequence and dynamically monitor key historical information. The update formula for the gated recurrent unit is as follows: ; In the formula, These are updating the gate vector and resetting the gate vector, respectively. In the candidate hidden state, The current hidden state. They are respectively The corresponding trainable weight matrix, Input the features at the current time. The Sigmoid activation function is used. For Hadamard product; The input feature dimension of the AI ​​traffic flow prediction model is: ,in, For batch size, Given the length of the input sequence, The feature dimension for each moment; the AI ​​traffic flow prediction model outputs the feature dimension as follows: ,in, To predict the step size, For the number of phases, Indicates output phase-level predicted flow rate With queue length Two characteristic indicators.

[0047] The dual uncertainty quantification module employs a combination of deep integration and Monte Carlo Dropout. Prediction results for: ; In the formula, The number of independently trained models. For the first The weights of the prediction results of each independent model. For the first Individual model prediction results; Variance of prediction results for cognitive uncertainty through multiple forward propagations in Monte Carlo Dropout Quantification: ; In the formula, For the number of samples, For the first The model prediction result value of the second sampling. for The mean of the prediction results from each sample; Data uncertainty is modeled using heteroscedasticity regression, with the model additionally outputting a variance term. The overall variance is The confidence interval is ,in, This represents the quantiles corresponding to the confidence level.

[0048] The scene determination module uses a single signal cycle as the analysis window and introduces phase saturation. As a core quantitative indicator: ; In the formula, For phase saturation, For the actual flow rate of the phase, Phase saturation flow rate; The scene determination module determines the phase saturation based on the... Time period characteristics and queue length The current intersection operation scenario is determined as off-peak, peak, or nighttime. The criteria for determining off-peak scenarios are: simultaneously satisfying the following conditions: the current time period is within the daytime range of 9:00-17:00 or the nighttime range of 19:30-21:30, and all phase saturation is... Furthermore, there was no sustained surge in traffic. The criteria for determining peak scenarios are: the current time period is within the morning peak (7:00-9:00) or evening peak (17:00-19:00), or any phase saturation. or the maximum queue length of any entrance lane Reaching or exceeding at least one of the set peak queue length thresholds; The criteria for determining a nighttime scene are: simultaneously satisfying the conditions that the current time period is within the time interval of 22:00 to 6:00 the next day, and that all phase saturation... Furthermore, there was no sustained surge in traffic.

[0049] Wherein, the reward function Rewards Including queue reduction rewards Traffic capacity rewards Green light utilization reward and energy consumption optimization rewards ; Queue reduction reward for: ; In the formula, For the first Within the cycle The maximum queue length of the real-time phase level for each phase. The total number of phases, The penalty coefficient is... The first output of the prediction model The first cycle The queue length of each phase, The confidence level for the queue length output by the prediction model; The traffic capacity reward for: ; In the formula, Rewards are based on actual traffic volume. To predict matching rewards, The weighting coefficient for the actual traffic reward item. To predict the weighting coefficients of the matching reward items; Among them, the actual traffic reward for: ; In the formula, For the first Within the cycle Real-time phase-level PCU traffic for each phase For the first Saturation flow rate of each phase, For the first Within the cycle The duration of the green light for each phase, For the first Total duration of the cycle; The predictive matching reward for: ; In the formula, To predict the confidence level of the flow output by the model, The first output of the prediction model The first cycle PCU flow rate for each phase; The green light utilizes rewards for: ; In the formula, For the first The idle time during which no vehicles pass through during the green light period of each phase. For the first Total green light duration for each phase; The energy consumption optimization reward The calculation formula is: ; In the formula, This represents the number of signal phase transitions within a period. The total duration is calculated for each period.

[0050] The preset constraints include safety constraints, operational constraints, and prevention constraints. The safety constraints include at least: minimum green light time for the phase. Constraints are used to ensure that pedestrians can safely cross the street and that queuing vehicles can safely start and pass the stop line; minimum clearing time. Constraints are used to set a total duration of not less than [a certain value] between any two conflicting phase transitions. The interval time ensures that the last vehicle in the previous phase can safely leave the conflict zone; the priority passage response constraint is used to ensure that when a special event marker in the state space is activated, the agent must trigger the corresponding special priority mode phase sequence in the next signal cycle. The operational constraints include at least: maximum green light time for a given phase. Constraints are used to prevent a single phase from excessively occupying right-of-way; cycle duration range constraints are used to limit the total duration of the signal cycle to a reasonable range; cycle change smoothness constraints are used to limit the variation in the total duration of adjacent cycles; phase structure and sequence constraints are used to enforce the phase logic relationship that conforms to the pre-set principle of traffic conflict elimination. The preventative constraints include at least: queue overflow avoidance constraints and flow mutation adaptive adjustment constraints, wherein the queue overflow avoidance constraints are used to adjust the predicted queue length. and confidence level For queues whose predicted length exceeds the maximum allowable queue length at the entrance lane The action is pre-judged and avoided; the adaptive adjustment constraint for traffic mutation is used to adjust the predicted traffic flow. and confidence level The action and reward function are dynamically adjusted when the predicted traffic flow changes significantly.

[0051] In other embodiments of the present invention, the present invention provides the following technical solution: a computer, including a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, wherein the processor 101 executes the computer program to implement the signal timing optimization method based on reinforcement learning as described above.

[0052] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0053] The memory 102 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include removable or non-removable (or fixed) media. Where appropriate, the memory 102 may be internal or external to a data processing device. In a particular embodiment, the memory 102 is non-volatile memory. In a particular embodiment, the memory 102 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random Access Memory (FPMDRAM), Extended Data Out Dynamic Random Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0054] The memory 102 can be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101.

[0055] The processor 101 implements the above-mentioned signal timing optimization method based on reinforcement learning by reading and executing computer program instructions stored in the memory 102.

[0056] In some embodiments, the computer may further include a communication interface 103 and a bus 100. For example, Figure 4 As shown, the processor 101, memory 102, and communication interface 103 are connected through bus 100 and complete communication with each other.

[0057] The communication interface 103 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of the present invention. The communication interface 103 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0058] Bus 100 includes hardware, software, or both, that couples components of a computer device together. Bus 100 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 100 may include one or more buses. Although specific buses are described and illustrated in the embodiments of the present invention, the present invention is contemplated by any suitable bus or interconnect.

[0059] The computer can acquire a signal timing optimization system based on reinforcement learning and execute the signal timing optimization method based on reinforcement learning of the present invention, thereby realizing signal timing optimization based on reinforcement learning.

[0060] In some further embodiments of the present invention, in conjunction with the above-described signal timing optimization method based on reinforcement learning, the present invention provides the following technical solution: a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-described signal timing optimization method based on reinforcement learning.

[0061] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0062] More specific examples of readable media (a non-exhaustive list) include: electrical connections (electronic devices) with one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0063] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0064] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A signal timing optimization method based on reinforcement learning, characterized in that, include: Multimodal traffic data at intersections are collected using a radar-visual integrated machine. The multimodal traffic data includes lane-specific flow, speed, headway, queue length, and pedestrian and non-motorized vehicle crossing status. The multimodal traffic data is subjected to anomaly detection and interpolation repair to obtain standardized traffic flow data; An AI traffic flow prediction model is constructed, which outputs predicted data of future signal cycles based on the standardized traffic flow data. The AI ​​traffic flow prediction model has an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. Design a multi-objective-oriented reinforcement learning agent, based on the standardized traffic flow data and the predicted data, and generate an optimal signal timing scheme under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ； In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

2. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The steps of performing anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data include: If the number of missing data per unit time exceeds the preset value, or if the single lane traffic flow remains at zero for an extended period of time... Exceeding the time period threshold or single-lane PCU flow Greater than the maximum allowed flow rate of the period or maximum queue length per lane Greater than the maximum allowable queue length at the entrance lane If the data is deemed abnormal, an interpolation algorithm based on adjacent period data is used to repair it in order to obtain standardized traffic flow data. The duration for: ； In the formula, This refers to the point in time when the current counted flow is all zero. The most recent valid data point that is not all zeros; The maximum allowable flow rate for the period for: ； In the formula, For lane saturation flow, The supersaturation coefficient. The duration of the green light within the cycle. The duration of the cycle; The maximum allowable queuing length at the entrance lane for: ； In the formula, Design length of the intersection approach lane A reasonable threshold; The single-lane PCU flow rate for: ； In the formula, This refers to the number of small vehicles counted within the period. This refers to the number of medium-sized vehicles counted within the period. The number of heavy-duty vehicles counted within the period; The maximum queue length of a single lane for: ； In the formula, The radar-visual integrated machine detected the length of a certain lane. This refers to the distance between the installation location of the radar-guided all-in-one machine and the parking line of a certain lane. This refers to the length of the blind zone of the integrated radar vision system.

3. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The multi-scale spatiotemporal graph convolutional network includes a spatiotemporal feature extraction module, a multi-scale fusion module, and a time series prediction module. The spatiotemporal feature extraction module uses a graph convolutional network to capture spatial dependencies in the road network topology. The graph convolution operation is defined as follows: ； In the formula, For the first The feature matrix of all nodes in the layer, For degree matrix, To add self-joins to the adjacency matrix, satisfying... ,in This is the original adjacency matrix. It is the identity matrix. For the first The feature matrix of the layer, For the first The trainable weight matrix of the layer, For activation functions; The multi-scale fusion module extracts features at different time scales through parallel one-dimensional dilated convolutional layers, and its fusion operation is defined as follows: ； In the formula, For a moment The output characteristics, The kernel size is [size]. For the first The convolution kernel weights at each position, The input feature sequence at time... The value, Void ratio; The time-series prediction module combines a gated recurrent unit with an attention mechanism to capture the temporal dependencies of the sequence and dynamically monitor key historical information. The update formula for the gated recurrent unit is as follows: ； In the formula, These are updating the gate vector and resetting the gate vector, respectively. In the candidate hidden state, The current hidden state. They are respectively The corresponding trainable weight matrix, Input the features at the current time. For the Sigmoid activation function, For Hadamard product; The input feature dimension of the AI ​​traffic flow prediction model is: ,in, For batch size, Given the length of the input sequence, The feature dimension for each moment; the AI ​​traffic flow prediction model outputs the feature dimension as follows: ,in, To predict the step size, For the number of phases, Indicates output phase-level predicted flow rate With queue length Two characteristic indicators.

4. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The dual uncertainty quantification module employs a combination of deep integration and Monte Carlo Dropout. Prediction results for: ； In the formula, The number of independently trained models. For the first The weights of the prediction results of each independent model. For the first Individual model prediction results; Variance of prediction results for cognitive uncertainty through multiple forward propagations in Monte Carlo Dropout Quantification: ； In the formula, For the number of samples, For the first The model prediction result value of the second sampling. for The mean of the prediction results from each sample; Data uncertainty is modeled using heteroscedasticity regression, with the model additionally outputting a variance term. The overall variance is The confidence interval is ,in, This represents the quantiles corresponding to the confidence level.

5. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The scene determination module uses a single signal cycle as the analysis window and introduces phase saturation. As a core quantitative indicator: ； In the formula, For phase saturation, For the actual flow rate of the phase, Phase saturation flow rate; The scene determination module determines the phase saturation based on the... Time period characteristics and queue length The current intersection operation scenario is determined as off-peak, peak, or nighttime. The criteria for determining off-peak scenarios are: simultaneously satisfying the following conditions: the current time period is within the daytime range of 9:00-17:00 or the nighttime range of 19:30-21:30, and all phase saturation is... Furthermore, there was no sustained surge in traffic. The criteria for determining peak scenarios are: the current time period is within the morning peak (7:00-9:00) or evening peak (17:00-19:00), or any phase saturation. or the maximum queue length of any entrance lane Reaching or exceeding at least one of the set peak queue length thresholds; The criteria for determining a nighttime scene are: simultaneously satisfying the conditions that the current time period is within the time interval of 22:00 to 6:00 the next day, and that all phase saturation... Furthermore, there was no sustained surge in traffic.

6. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The reward function Rewards Including queue reduction rewards Traffic capacity rewards Green light utilization reward and energy consumption optimization rewards ; Queue reduction reward for: ； In the formula, For the first Within the cycle The maximum queue length of the real-time phase level for each phase. The total number of phases, The penalty coefficient is... The first output of the prediction model The first cycle The queue length of each phase, The confidence level for the queue length output by the prediction model; The traffic capacity reward for: ； In the formula, Rewards are based on actual traffic volume. To predict matching rewards, The weighting coefficient for the actual traffic reward item. To predict the weighting coefficients of the matching reward items; Among them, the actual traffic reward for: ； In the formula, For the first Within the cycle Real-time phase-level PCU traffic for each phase For the first Saturation flow rate of each phase, For the first Within the cycle The green light duration for each phase, For the first Total duration of the cycle; The predictive matching reward for: ； In the formula, To predict the confidence level of the flow output by the model, The first output of the prediction model The first cycle PCU flow rate for each phase; The green light utilizes rewards for: ； In the formula, For the first The idle time during which no vehicles pass through during the green light period of each phase. For the first Total green light duration for each phase; The energy consumption optimization reward The calculation formula is: ； In the formula, This represents the number of signal phase transitions within a period. The total duration is calculated for each period.

7. The signal timing optimization method based on reinforcement learning according to claim 1, characterized in that, The preset constraints include safety constraints, operational constraints, and prevention constraints; The safety constraints include at least: minimum green light time for the phase. Constraints are used to ensure that pedestrians can safely cross the street and that queuing vehicles can safely start and pass the stop line; minimum clearing time. Constraints are used to set a total duration of not less than [a certain value] between any two conflicting phase transitions. The interval time ensures that the last vehicle in the previous phase can safely leave the conflict zone; the priority passage response constraint is used to ensure that when a special event marker in the state space is activated, the agent must trigger the corresponding special priority mode phase sequence in the next signal cycle. The operational constraints include at least: maximum green light time for a given phase. Constraints are used to prevent a single phase from excessively occupying right-of-way; cycle duration range constraints are used to limit the total duration of the signal cycle to a reasonable range; cycle change smoothness constraints are used to limit the variation in the total duration of adjacent cycles; phase structure and sequence constraints are used to enforce the phase logic relationship that conforms to the pre-set principle of traffic conflict elimination. The preventative constraints include at least: queue overflow avoidance constraints and flow mutation adaptive adjustment constraints, wherein the queue overflow avoidance constraints are used to adjust the predicted queue length. and confidence level For queues whose predicted length exceeds the maximum allowable queue length at the entrance lane The action is pre-judged and avoided; the adaptive adjustment constraint for traffic mutation is used to adjust the predicted traffic flow. and confidence level The action and reward function are dynamically adjusted when the predicted traffic flow changes significantly.

8. A signal timing optimization system based on reinforcement learning, characterized in that, The system includes: The data acquisition module is used to collect multimodal traffic data at intersections through the radar-visual integrated machine. The multimodal traffic data includes lane-specific flow, speed, headway, queue length, and pedestrian and non-motorized vehicle crossing status. The processing module is used to perform anomaly detection and interpolation repair on the multimodal traffic data to obtain standardized traffic flow data; The construction module is used to build an AI traffic flow prediction model, which outputs prediction data of future signal cycles based on the standardized traffic flow data and according to the AI ​​traffic flow prediction model. The AI ​​traffic flow prediction model has an encoder-decoder backbone network and includes a multi-scale spatiotemporal graph convolutional network and a dual uncertainty quantization module. The reinforcement learning module is used to design a multi-objective-oriented reinforcement learning agent. Based on the standardized traffic flow data and the predicted data, the reinforcement learning agent generates an optimal signal timing scheme under preset constraints. The agent includes a scene determination module, a hierarchical composite action space, and a reward function with dynamic weight adjustment. The state space of the reinforcement learning agent. Includes real-time traffic state characteristics, predicted traffic state characteristics, environmental state characteristics, and historical decision-making effect characteristics; action space It includes a discrete action layer for adjusting the green light duration, a continuous action layer for adjusting the signal period duration, and a structured action layer for switching phase sequence modes; reward function. Based on multi-objective optimization theory, its total reward value is constructed. The calculation formula is as follows: ； In the formula, For the first Total reward value for each cycle, For the first The first reward item in the Dynamic weights for each period, For the first The first reward item in the The value of each period, Set the total number for each reward item.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the signal timing optimization method based on reinforcement learning as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the signal timing optimization method based on reinforcement learning as described in any one of claims 1 to 7.

Images