Traffic signal lamp control method and system based on multi-view fusion and time sequence characteristics

Abstract

The application provides a traffic signal lamp control method and system based on multi-view fusion and time sequence characteristics, and relates to the technical field of intelligent traffic and computer vision. The method comprises the following steps: target detection, cross-frame tracking and spatial mapping are performed on multi-directional monitoring of an intersection to obtain three-dimensional position and motion information of participants under each view; then, feature extraction, bird's eye view projection and cross-view aggregation are performed according to the information to obtain current frame bird's eye view features; the features are subjected to long time sequence caching, history frame searching and time sequence fusion to generate spatio-temporal fusion features and a scene state vector; semantic analysis and event detection are performed on the fusion features to output an event type and a confidence; a normal or event response mode is selected according to the event and the confidence, a timing scheme is generated in the normal mode, a preset response strategy is triggered in the event mode, and a control instruction is generated to control the traffic signal lamp. The application realizes global situation awareness of the intersection, can capture evolution laws such as queue growth, and triggers corresponding control strategies in abnormal scenes.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and computer vision technology, and in particular to a traffic signal control method and system based on multi-view fusion and temporal features. Background Technology

[0002] With the accelerating pace of urbanization, urban road traffic congestion has become a prominent bottleneck restricting socio-economic development and the quality of residents' travel. Traffic lights, as the core control equipment for allocating right-of-way at intersections, directly determine the efficiency of road network traffic and the level of traffic safety through their timing schemes. Traditional fixed timing control cannot adapt to real-time fluctuations in traffic flow, and inductive control has a single perception dimension, making it difficult to comprehensively perceive the traffic situation at intersections. In recent years, intelligent signal control methods based on visual perception and reinforcement learning have received widespread attention due to their adaptive decision-making capabilities. Related research attempts to use surveillance cameras to acquire visual information at intersections, then use object detection algorithms to identify vehicles and pedestrians, converting the detection results into state features, and finally using reinforcement learning models to map the environmental state to signal timing strategies, aiming to achieve continuous optimization of intersection traffic efficiency.

[0003] Currently, intelligent control of traffic lights typically employs a combination of visual perception and reinforcement learning. This involves first extracting target detection results from a single-channel monitoring video feed, then discarding perceptual information such as the detection box coordinates, retaining only low-dimensional statistical values ​​like queue length and vehicle count as input to the reinforcement learning model for timing decisions. However, this approach suffers from several drawbacks. First, high-dimensional visual information is forcibly compressed, resulting in the permanent loss of numerous fine-grained spatial features. Second, single-channel cameras have blind spots due to field-of-view occlusion, leading to significant visual information loss and creating blind spots in single-view perception. Third, each frame is processed independently, with the decision model relying solely on current statistical values. Visual information from historical frames is discarded after aggregation, and the model lacks a memory mechanism, preventing the system from capturing the evolution of traffic conditions over time. Fourth, event detection and signal control are designed as two independently operating modules. Event detection only outputs alarm information and does not participate in signal timing decisions or strategy adjustments, resulting in an inability to adjust control strategies promptly in abnormal scenarios such as traffic accidents or special vehicle passage. Summary of the Invention

[0004] The purpose of this invention is to provide a traffic signal control method and system based on multi-view fusion and temporal features, which solves the problems of existing technologies that generate blind spots in single-view perception, cannot capture the evolution of traffic conditions in the time dimension, and cannot adjust control strategies in a timely manner in abnormal scenarios such as traffic accidents and special vehicle passage.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a traffic signal control method based on multi-view fusion and temporal features, the method comprising: Multi-target detection, cross-frame tracking, and 3D spatial mapping are performed sequentially on the surveillance video sequences from multiple directions at the intersection to obtain the 3D spatial position and motion state information of traffic participants from each perspective. Based on the three-dimensional spatial position and motion state information of traffic participants from various perspectives, feature extraction, bird's-eye view feature spatial projection, and cross-perspective feature aggregation are performed sequentially to obtain the bird's-eye view features of the current frame. The bird's-eye view features of the current frame are sequentially cached for long-term features, retrieved for historical frame features related to the current scene, and fused temporally to generate spatiotemporal fusion features and scene state feature vectors. Perform semantic parsing of traffic scenes and traffic event detection on spatiotemporal fusion features, and output event type and confidence level; Based on the event and confidence level, select to enter normal control mode or event response mode. In normal control mode, generate phase timing scheme based on scene state feature vector. In event response mode, trigger the corresponding preset response strategy based on event type. Control commands are generated based on the phase timing scheme or preset response strategy, and traffic lights are controlled according to the control commands.

[0006] A second aspect of the present invention provides a traffic signal control system based on multi-view fusion and temporal characteristics, the system comprising: The multi-view visual perception module is used to perform multi-target detection, cross-frame tracking and three-dimensional spatial mapping on the monitoring video sequence from multiple directions at the intersection, so as to obtain the three-dimensional spatial position and motion state information of traffic participants from each viewpoint. The projection and feature aggregation module is used to extract features, project bird's-eye view features into space, and aggregate cross-view features in sequence based on the three-dimensional spatial position and motion state information of traffic participants from various perspectives, so as to obtain the bird's-eye view features of the current frame. The retrieval and temporal fusion module is used to sequentially perform long temporal feature caching, historical frame feature retrieval related to the current scene, and temporal fusion on the bird's-eye view features of the current frame, generating spatiotemporal fusion features and scene state feature vectors. The scene parsing and detection module is used to perform semantic parsing of traffic scenes and traffic event detection on spatiotemporal fusion features, and outputs event type and confidence score. The signal decision module is used to select between normal control mode and event response mode based on the event and confidence level. In normal control mode, it generates a phase timing scheme based on the scene state feature vector. In event response mode, it triggers the corresponding preset response strategy based on the event type. The control module is used to generate control commands based on the phase timing scheme or preset response strategy, and to control traffic lights according to the control commands.

[0007] Compared to existing technologies, the traffic signal control method and system based on multi-view fusion and temporal features provided by this invention sequentially performs multi-target detection, cross-frame tracking, and 3D spatial mapping on monitoring video sequences from multiple directions at an intersection to obtain the 3D spatial position and motion state information of traffic participants from each viewpoint. Based on the 3D spatial position and motion state information of traffic participants from each viewpoint, feature extraction, bird's-eye view feature spatial projection, and cross-view feature aggregation are performed sequentially to obtain the bird's-eye view features of the current frame. The bird's-eye view features of the current frame are sequentially subjected to long-term temporal feature caching, historical frame feature retrieval related to the current scene, and temporal fusion to generate spatiotemporal fusion features and scene state feature vectors. Traffic scene semantic parsing and traffic event detection are performed on the spatiotemporal fusion features to output event type and confidence level. Based on the event and confidence level, the system selects to enter normal control mode or event response mode. In normal control mode, a phase timing scheme is generated based on the scene state feature vector. In event response mode, a corresponding preset response strategy is triggered based on the event type. Control commands are generated based on the phase timing scheme or preset response strategy, and traffic signals are controlled according to the control commands. This invention achieves comprehensive perception of the overall traffic situation at intersections by performing multi-target detection, cross-frame tracking, and 3D spatial mapping on surveillance video sequences from multiple directions. This aligns and fuses visual information from various perspectives within a unified bird's-eye view feature space. By performing long-term feature caching, historical frame feature retrieval, and temporal fusion on the bird's-eye view features of the current frame, the system can utilize historical information to understand the evolution of traffic conditions and capture slow-moving patterns such as queue length increases. Furthermore, by detecting traffic events using spatiotemporal fusion features, the system selects between normal mode and event response mode based on event type and confidence level. In event response mode, it automatically triggers corresponding preset response strategies, establishing a direct linkage mechanism between event detection and signal control. This enables the automatic triggering of appropriate control strategies in abnormal scenarios such as traffic accidents and special vehicle passage. Attached Figure Description

[0008] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 A flowchart illustrating a traffic signal control method based on multi-view fusion and temporal characteristics is shown schematically. Figure 2 A flowchart illustrating the determination of bird's-eye view features for the current frame is shown schematically. Figure 3 A flowchart illustrating the generation of spatiotemporal fusion features and scene state feature vectors is shown schematically. Figure 4 A schematic diagram of a traffic signal control system based on multi-view fusion and temporal characteristics is shown. Detailed Implementation

[0009] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0010] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0011] The methods described in the embodiments of the present invention will be explained in detail below.

[0012] Figure 1 The flowchart of a traffic signal control method based on multi-view fusion and temporal features according to an embodiment of the present invention is illustrated schematically. See [link / reference]. Figure 1 As shown, the traffic signal control method based on multi-view fusion and temporal features may include: S101. Perform multi-target detection, cross-frame tracking, and three-dimensional spatial mapping on the monitoring video sequences from multiple directions at the intersection to obtain the three-dimensional spatial position and motion state information of traffic participants from each perspective.

[0013] Surveillance cameras are deployed in four directions at the intersection to acquire multiple synchronous video streams covering the entire intersection area; the video frames are synchronized in time through pulse signals or network clock protocols to ensure that each perspective captures the intersection scene at the same moment.

[0014] Among them, traffic participants can be regarded as targets captured by surveillance cameras. Their three-dimensional spatial position includes position coordinates, and their motion state information includes orientation angle and velocity vector.

[0015] Specifically, step S101 includes: Step A1: Perform multi-target detection on each video frame in the surveillance video sequence from multiple directions at the intersection, and output the bounding box and category label of each target.

[0016] The targets can include vehicles, pedestrians, non-motorized vehicles, and special vehicles. The system performs multi-target detection on video frames from multiple directions of a traffic intersection and can also output confidence scores.

[0017] A unified, lightweight detection network is employed to perform end-to-end multi-object detection across various video frames. This unified, lightweight detection network, based on pre-trained weights, is fine-tuned using an intersection monitoring dataset, enabling simultaneous detection of vehicles, pedestrians, non-motorized vehicles, and special vehicles. For each frame, the detection network outputs the bounding box coordinates, class label, and confidence score for each object.

[0018] Step A2: Based on the bounding box and category label of each target, perform cross-frame association matching to generate the continuous motion trajectory of each target.

[0019] The continuous motion trajectory of each target includes the position sequence, velocity sequence and acceleration sequence of each target in time.

[0020] Based on appearance feature similarity and motion feature consistency, cross-frame association matching is performed on each target in step A1. Specifically, the appearance feature vector of each detected target is extracted, and the target position predicted by the Kalman filter is combined with the detection-trajectory association matching performed by the Hungarian algorithm. A unique tracking identifier is assigned to each target, and a continuous motion trajectory is generated.

[0021] Step A3: Based on the continuous motion trajectory of each target, estimate the depth value of each target, and based on the depth value and the camera's intrinsic and extrinsic parameters, map each target from the image coordinates to the three-dimensional world coordinate system with the center of the intersection as the origin, so as to obtain the position coordinates, orientation angle and velocity vector of each target on the intersection plane.

[0022] Depth values ​​of each target are estimated using a monocular depth estimation network. This network is pre-trained on intersection scene data using a self-supervised approach, eliminating the need for LiDAR point cloud annotation. By combining the camera's intrinsic and extrinsic parameter matrices, the center points of each target are derived from the 2D image coordinates. Mapped to a 3D world coordinate system with the center of the intersection as the origin. ,in The height of the target on the road surface is set to a fixed value. For each target, its two-dimensional position coordinates on the intersection plane are finally obtained. Orientation angle and velocity vector .in, In the horizontal direction, Vertical direction Let x be the x-coordinate of the target on the horizontal plane of the intersection in a three-dimensional world coordinate system with the center of the intersection as the origin. The target is the vertical coordinate of the intersection plane (horizontal plane) in a three-dimensional world coordinate system with the center of the intersection as the origin.

[0023] S102. Based on the three-dimensional spatial position and motion state information of traffic participants from each perspective, feature extraction, bird's-eye view feature spatial projection, and cross-view feature aggregation are performed sequentially to obtain the bird's-eye view features of the current frame.

[0024] Step S102 mainly involves fusing the independent visual information from the four directions at the feature level to construct a bird's-eye view feature representation covering the entire intersection.

[0025] Figure 2 A flowchart illustrating the determination of bird's-eye view features for the current frame is shown schematically. See [link / reference] Figure 2 As shown, step S102 includes: S1021: Input the video frames from each stream into a hierarchical visual encoder with shared weights to extract multi-scale feature maps from each viewpoint.

[0026] The shared-weight hierarchical visual encoder employs a hierarchical structure, comprising a low-level high-resolution branch and a top-level semantic branch. The low-level high-resolution branch preserves detailed information such as object edges, textures, and lane lines through a shallow convolutional network. The top-level semantic branch extracts the object's category semantics and scene context information through a deep convolutional network. The outputs of the two branches are concatenated along the channel dimension to obtain a multi-scale feature map. , ,in, Indicates the first One perspective, For the first Multi-scale feature maps from multiple perspectives For the first The height of the multi-scale feature map from each perspective. For the first Width of multi-scale feature maps from multiple perspectives For the first The number of channels in a multi-scale feature map from multiple perspectives. For dimension The three-dimensional real tensor space.

[0027] S1022: Based on the three-dimensional spatial position and motion state information, the multi-scale feature maps of each viewpoint are projected onto the bird's-eye view feature space using a deformable attention mechanism to obtain multi-view information. The multi-view information is then aggregated through multi-head attention to obtain the initial bird's-eye view features of the current frame.

[0028] Specifically, based on the three-dimensional spatial position and motion state information, a deformable attention mechanism is used to project multi-scale feature maps from various perspectives onto the bird's-eye view feature space to obtain multi-view information, including: Step B1: Set up a two-dimensional grid covering the intersection area in the feature space of the bird's-eye view.

[0029] The two-dimensional grid covering the intersection area is represented as , , The height of the two-dimensional grid in the feature space of the bird's-eye view. The width of the two-dimensional grid in the feature space of the bird's-eye view. For dimension A two-dimensional real matrix space.

[0030] Step B2: Based on the three-dimensional spatial position and motion state information, predict the sampling offset of each grid position in the two-dimensional grid under each viewpoint.

[0031] Each grid location corresponds to a physical area on the intersection plane. Each grid location is represented as follows: , The horizontal coordinate value for each grid location, The vertical coordinate value for each grid location.

[0032] For each grid location, a lightweight subnetwork is used. Predict the corresponding sampling offset at each viewpoint The sampling offset represents the grid position. In the Feature locations that require additional attention near the projection position on the feature map of each view.

[0033] Step B3: Based on the sampling offset, sample on the multi-scale feature map of each viewpoint to obtain the feature vector of each viewpoint at the grid position.

[0034] Sampling can be bilinear interpolation sampling, based on the sampling offset in the multi-scale feature map. Bilinear interpolation sampling is performed to obtain the grid position of each viewpoint. eigenvectors at location .

[0035] Step B4: Traverse all grid positions and determine the multi-view information of each view at all grid positions based on the feature vectors of each view at each grid position.

[0036] After step B4, multi-head attention is used to process the feature vectors at different viewpoints at the grid locations. That is, to aggregate information from multiple perspectives to generate grid positions. fusion feature vector Repeat the above process for all grid locations to obtain the initial bird's-eye view features of the current frame. .in, The first viewpoint is located at the grid position. The feature vector at that location, For the second perspective at the grid position The feature vector at that location, For the first Each perspective is located in the grid position. The feature vector at that location, The total number of viewpoints. This represents the number of channels in the two-dimensional grid within the feature space of the bird's-eye view.

[0037] S1023: Temporally align the initial bird's-eye view features of the current frame with the bird's-eye view features of the previous frame to obtain the bird's-eye view features of the current frame.

[0038] The initial bird's-eye view features of the current frame are temporally aligned with the bird's-eye view features of the previous frame through feature matching to obtain aligned bird's-eye view features. The aligned bird's-eye view features are then used as the bird's-eye view features of the current frame and output.

[0039] Predicting the first... using a lightweight alignment network. Initial bird's-eye view features of the frame Compared to the first Bird's-eye view features of the frame The offset field, using the offset field to the first Bird's-eye view features of the frame Perform resampling to obtain the aligned first... Historical bird's-eye view features of frames , will the Initial bird's-eye view features of the frame With the aligned first Historical bird's-eye view features of frames Element-wise weighted fusion is used to obtain the final bird's-eye view features of the aligned current frame, i.e., the first... Bird's-eye view features of the current frame .

[0040] S103. The bird's-eye view features of the current frame are sequentially cached for long-term features, retrieved for historical frame features related to the current scene, and fused temporally to generate spatiotemporal fusion features and scene state feature vectors.

[0041] Specifically, step S103 includes: S1031: Perform linear compression on the bird's-eye view features of the current frame to obtain a low-dimensional feature vector, and write the low-dimensional feature vector sequentially into a circular feature queue to build a long-term feature cache.

[0042] The circular feature queue retains the most recent preset time window length. The historical feature vectors are used to create a circular feature queue as a long-term feature cache.

[0043] The bird's-eye view features of each frame are linearly compressed into low-dimensional feature vectors. Then, they are sequentially written into a fixed-length circular feature queue. When a new vector is written, the oldest low-dimensional feature vector in the circular feature queue is removed. The preset time window length, For the first The low-dimensional feature vector of the frame, for 3D real vector space, For the circular feature queue, the first The low-dimensional feature vector of the frame, For the circular feature queue, the first The low-dimensional feature vector of the frame. The preset time window length can range from 10 frames to 100 frames.

[0044] S1032: Through the cross-attention mechanism, retrieve historical frame features related to the current scene from the long-term feature cache and determine the retrieved historical frame features.

[0045] Bird's-eye view features of the current frame Obtain the first by linear projection Frame query vector , with the first Frame query vector As a query, a fixed-length circular feature queue from a long-time feature cache is used via a cross-attention mechanism. In the process, retrieve the historical frame features most relevant to the current scene. (Based on the first...) Frame query vector For queries, use a fixed-length circular feature queue. For keys and values, attention weights are calculated and weighted aggregation is performed to obtain the retrieved historical frame features. : ; in, Represents the multi-head cross-attention function, the first... Frame query vector From the bird's-eye view features of the current frame Obtained by linear projection, For dimension A three-dimensional real tensor.

[0046] S1033: Perform temporal fusion of the bird's-eye view features of the current frame with the retrieved historical frame features to generate spatiotemporal fusion features and scene state feature vectors.

[0047] Figure 3 The flowchart illustrating the generation of spatiotemporal fusion features and scene state feature vectors is shown in the attached diagram. Figure 3 As shown, S1033 includes: S10331: The bird's-eye view features of the current frame are concatenated with the retrieved historical frame features in the channel dimension to obtain the concatenated features.

[0048] The spliced ​​features are represented as ,in, For the first Bird's-eye view features of the frame For the first Features of historical frames retrieved from frames. For dimension A three-dimensional real tensor.

[0049] S10332: Perform 3D convolution processing on the stitched features to obtain motion-enhanced features.

[0050] Along the time dimension, the concatenated features of the current frame are stacked with the concatenated features of the preceding and following frames to form a short temporal sequence of 5 frames. This sequence is then processed by two 3×3×3 three-dimensional convolutional kernels to output motion-enhanced features. In actual inference, the concatenated features of the preceding and following frames are read from a long-term temporal feature cache.

[0051] S10333: The motion enhancement features are sequentially processed by self-attention, global average pooling, and linear projection to generate spatiotemporal fusion features and scene state feature vectors.

[0052] Motion enhancement features are flattened into a sequence in the spatial dimension and fed into a Transformer encoder layer containing position encoding. This utilizes a self-attention mechanism to ensure that each spatial location pays attention to relevant locations at all historical moments, thereby capturing the evolutionary patterns of slow changes such as queue growth and pedestrian crossings. A global average pooling layer compresses the feature map into a global feature vector to generate spatiotemporal fusion features. Finally, a linear projection layer maps these spatiotemporal fusion features into a compact scene state feature vector. , It is a 256-dimensional real vector.

[0053] S104. Perform traffic scene semantic parsing and traffic event detection on the spatiotemporal fusion features, and output the event type and confidence level.

[0054] Scene semantic parsing includes estimating at least one of lane occupancy rate, queue length, and average speed. Traffic event detection includes identifying at least one event type, such as traffic accidents, pedestrians running red lights, emergency vehicles passing through intersections, intersection overflow, and vehicles driving in the wrong direction, and outputting the event type, location, confidence level, and duration.

[0055] Traffic incident detection of special vehicle passage includes: visually identifying at least one of police cars, ambulances, and fire trucks; traffic accident incidents include detecting at least one abnormal pattern among abnormal vehicle stops, vehicle trajectory conflicts, and pedestrian falls.

[0056] Specifically, step S104 includes: The spatiotemporal fusion features are input into multiple parallel task detection heads, and the scene semantic parsing detection head in the task detection head is used to process the spatiotemporal fusion features to estimate at least one of lane occupancy rate, queue length and average speed; and multiple event detection heads in the task detection head are used to process the spatiotemporal fusion features to identify event types and corresponding confidence levels.

[0057] The task detection head includes a scene semantic parsing detection head and multiple event detection heads.

[0058] Each task detection head consists of one or more convolutional layers and fully connected layers. The scene semantic parsing detection head is used to regress the vehicle occupancy percentage, queue length, and average speed of each lane based on spatiotemporal fusion features, and to predict the pedestrian crossing probability and crossing initiation time. Multiple event detection heads are used to identify the type and location of traffic events based on spatiotemporal fusion features. The event detection head outputs a confidence score for each candidate event type. When the confidence score exceeds a preset confidence threshold, the corresponding event is determined to have occurred, and the event type, its location coordinates in the bird's-eye view space, confidence score, and event duration in frames are output. The preset confidence threshold can be set to 0.7, and there are various preset confidence thresholds, which are not specifically limited here.

[0059] For example, three parallel scene semantic parsing detection heads are set up: a lane occupancy detection head, a queue length detection head, and an average speed detection head. The lane occupancy detection head consists of two convolutional layers and one fully connected layer, outputting a vector of the percentage of vehicles occupied in each lane. ,in, The total number of lanes at the intersection. For length is The queue length detection head consists of two convolutional layers and one fully connected layer, outputting a vector of queue length values ​​for each lane. The average speed detector consists of two convolutional layers and one fully connected layer, outputting the average speed vector of vehicles in each lane. .

[0060] By processing spatiotemporal fusion features using multiple event detection heads, the location and duration of occurrence can also be identified.

[0061] For example, multiple event detection heads are set up. Each event detection head consists of a convolutional feature extraction layer and a binary classifier, used to determine whether a corresponding type of event has occurred. Each event detection head outputs an event type label, the event's location coordinates in the bird's-eye view space, the event confidence score, and the event duration in frames. When the confidence score is higher than a preset confidence threshold, the event is confirmed to have occurred.

[0062] S105. Based on the event and confidence level, select to enter the normal control mode or the event response mode. In the normal control mode, generate a phase timing scheme based on the scene state feature vector. In the event response mode, trigger the corresponding preset response strategy based on the event type.

[0063] Specifically, step S105 includes: Step C1: Determine whether no event was detected or whether the confidence level is lower than a preset threshold.

[0064] Determine whether no event was detected or whether the confidence level is lower than a preset threshold. If yes, proceed to step C2; otherwise, proceed to step C3.

[0065] If the confidence level of no current events exceeds a preset threshold If the confidence level of one or more events reaches or exceeds the preset threshold, then the system enters normal control mode; Then it enters event response mode, with a preset threshold. The value can range from 0.5 to 0.8.

[0066] Step C2: Enter normal control mode, input the scene state feature vector into the hybrid expert decision network, and use the gating unit in the hybrid expert decision network to calculate the weight of each expert feedforward network in the hybrid expert decision network according to the scene state feature vector. Using each expert feedforward network, output the timing strategy according to the scene state feature vector, and perform weighted combination of the timing strategy according to the weight to generate the phase timing scheme.

[0067] The phase timing scheme includes the phase sequence and the green light duration for each phase.

[0068] The hybrid expert decision network comprises a gating unit and K parallel expert feedforward networks. The gating unit maps the scene state feature vector to a K-dimensional routing score via a linear layer, and normalizes the weight distribution of each expert feedforward network using a Softmax function. Each expert feedforward network is a three-layer nonlinear mapping network that maps the scene state feature vector to a timing policy vector, which includes phase sequence encoding and the green light duration for each phase. The final output of the hybrid expert decision network is the weighted sum of the timing policy vectors output by each expert feedforward network according to their respective weight distributions.

[0069] scene state feature vector In the hybrid expert decision network, the gating unit inputs the scene state feature vector. via linear layer Mapped to K-dimensional routing scores The weights of each expert feedforward network are obtained by normalization using the Softmax function. ,in For shape A two-dimensional real matrix. Each expert feedforward network is a three-layer nonlinear mapping network that maps the scene state feature vector. Mapped to a vector corresponding to a timing strategy ,in, The dimension for encoding the phase timing scheme. Score the K-dimensional route. The Softmax activation function is used. For the first The vector corresponding to the timing strategy for each viewpoint for A real vector space. The timing strategy vector contains the phase sequence encoding and the green light duration for each phase. The final output of the hybrid expert decision network is the vector corresponding to the timing strategy output by each expert feedforward network. A weighted summation according to the weight distribution yields the vector corresponding to the weighted combined timing strategy: ; in, For the first The weights of each expert feedforward network from each perspective, For the first The vector corresponding to the weighted combination timing strategy of the frame is decoded to obtain a phase timing scheme with phase sequence and green light duration for each phase.

[0070] Step C3: Enter event response mode and trigger the corresponding preset response strategy according to the event type.

[0071] Specifically, step C3 includes: Step C31a: When the event type is a special vehicle passage event, switch the traffic light in the direction of the special vehicle to green, and keep the traffic lights in other directions red until the special vehicle passes through the intersection, so as to complete the execution of the priority release plan.

[0072] Until the special vehicle passes through the intersection, the system gradually switches back to normal control mode within 2-3 cycles to complete the priority passage plan.

[0073] Step C31b: When the event type is a traffic accident, lock the traffic light in the direction corresponding to the lane where the accident is located to red, and keep the phase that conflicts with the lane where the accident is located red, so as to complete the execution of the accident isolation plan.

[0074] Setting the traffic light in the lane corresponding to the accident location to red, and keeping the phase of the lane conflicting with the accident lane red, can prevent secondary accidents. The traffic lights will be gradually restored after visual inspection confirms the accident has been cleared.

[0075] Step C31c: When the event type is pedestrian running a red light, extend the green light time in the direction of the pedestrian to a preset safe duration to complete the execution of the safety extension plan.

[0076] The preset safe duration can be set between 3 and 10 seconds.

[0077] Step C31d: When the event type is an intersection overflow event, increase the proportion of green light time for vehicles coming from the upstream direction of the overflow direction and reduce the release phase time for the corresponding overflow direction to complete the execution of the congestion relief plan.

[0078] Increasing the green light time ratio for vehicles coming from the overflow direction upstream and reducing the corresponding release phase time for the overflow direction can prevent more vehicles from flooding into already saturated intersections.

[0079] Step C32: After the event ends, the scheme executed in event response mode will be gradually transitioned back to the phase timing scheme of normal control mode within a preset number of signal cycles.

[0080] The preset number of signal cycles can be 2-3 cycles. After the event ends, the phase timing scheme implemented in the event response mode, namely at least one of the priority release scheme, accident isolation scheme, safety extension scheme and congestion relief scheme, will be gradually transitioned back to the phase timing scheme of the normal control mode within 2 to 3 signal cycles in a gradual manner, so as to avoid the sudden change of signal state causing disturbance to traffic flow.

[0081] S106. Generate control commands based on the phase timing scheme or preset response strategy, and control traffic lights according to the control commands.

[0082] The phase timing scheme or preset response strategy is converted into control commands for the traffic lights, and the traffic lights are controlled according to the control commands. A multi-objective reward function containing average vehicle delay, maximum queue length, average pedestrian waiting time and event response timeliness score is calculated based on the visual detection results of subsequent video frames. The quadruple of state, action, reward and next state is stored in the experience replay pool, and the policy gradient is updated periodically for the hybrid expert decision network.

[0083] Control commands are generated based on the phase timing scheme or preset response strategy. Based on the control commands, the traffic signal controller is used to control the switching of red, yellow and green lights in each direction at the intersection.

[0084] Following control, i.e., after executing the control commands generated by the phase timing scheme or preset response strategy, the actual traffic state after control is obtained based on the visual detection results of subsequent video frames generated by the control, and a multi-objective reward function is calculated based on the visual detection results. The expression for the multi-objective reward function is: ; in, Indicates the reward value. This represents the normalized value of the average vehicle delay, calculated by detecting the waiting time of vehicles at intersections; This represents the normalized value of the maximum queue length, obtained through the queue length detection head; This represents the normalized value of the average pedestrian waiting time, calculated by detecting the time pedestrians spend at the intersection. This represents the normalized value of the event response timeliness score. This is the weighting coefficient for average vehicle delay. The weighting coefficient for the maximum queue length. This is a weighting factor for the average pedestrian waiting time. This is a weighting coefficient for the timeliness of event response scoring. The specific normalization method is as follows: based on the waiting time of vehicles at intersections obtained from multi-target detection and cross-frame tracking of the surveillance video sequence, the average vehicle delay is calculated. This average vehicle delay is then divided by a preset delay reference value and limited to between 0 and 1 to obtain the normalized value of the average vehicle delay. The preset delay reference value is set based on the historical average vehicle delay or the acceptable delay limit for that intersection. The maximum queue length is determined based on the queue length output by the scene semantic parsing detection head in the task detection head. The maximum queue length is then divided by the preset queue capacity reference value and limited to between 0 and 1 to obtain the normalized value of the maximum queue length. The preset queuing capacity reference value is set based on the maximum number of vehicles that can be accommodated corresponding to the length of the approach lane and the number of lanes. The average pedestrian waiting time is calculated based on the pedestrian dwell time obtained from multi-target detection and cross-frame tracking of the surveillance video sequence. This average pedestrian waiting time is then divided by the preset pedestrian waiting time reference value and limited to between 0 and 1 to obtain the normalized value of the average pedestrian waiting time. The preset pedestrian waiting time reference value is set based on the acceptable waiting time for pedestrians to cross the street. The preset allowable response delay is subtracted from the response delay from event detection and confirmation to the execution of the corresponding preset response strategy. This result is then divided by the allowable response delay and limited to a range of 0 to 1 to obtain the normalized event response timeliness score. The preset allowable response delay is the time between the moment when the confidence level output by the event detection head reaches or exceeds the preset confidence threshold and the event is confirmed, and the moment when the signal decision module triggers the corresponding preset response strategy according to the event type and generates and issues the control command. The preset allowable response delay is set according to at least one of traffic management requirements, signal response capability, and event urgency.

[0085] The first Scene state feature vector of a frame , No. The vector corresponding to the weighted combination timing strategy of the frame , No. Frame reward value , and the Scene state feature vector of each frame As a quadruple The data is stored in the experience replay pool. When the number of quadruplets in the experience replay pool reaches the preset threshold, batch data is randomly sampled from it each time, and the hybrid expert decision network is updated using the deep deterministic policy gradient method to optimize the hybrid expert decision network in the direction of maximizing cumulative reward. The preset number of tuples can be 10,000 to 50,000.

[0086] This invention realizes an end-to-end processing flow from multi-view video input to the output of traffic light control commands. Each step is operated in the same visual feature space, avoiding the problem of progressive loss of visual information in traditional methods.

[0087] This invention introduces a cross-view deformable attention fusion mechanism. By aligning and aggregating camera images from multiple directions within a unified bird's-eye view feature space, it overcomes the blind spot problem of single-view perception and achieves comprehensive perception of the overall traffic situation at intersections. Simultaneously, it introduces long-term feature caching, using a cross-attention mechanism to extract and fuse long-term features, enabling the system to capture the evolution of traffic conditions over time and resolving the problem of fragmented temporal information caused by independent frame-by-frame processing in existing methods. Furthermore, this invention performs scene semantic parsing, event detection, and signal decision-making within a unified visual feature space, establishing a direct linkage mechanism between event detection and signal control. This allows for the automatic triggering of appropriate control strategies in abnormal scenarios such as traffic accidents and special vehicle passage. Compared to existing methods, this invention significantly improves the perception and adaptive capabilities of traffic light control in complex traffic scenarios, thus exhibiting superior control performance and robustness in complex intersection scenarios such as mixed traffic flow and abnormal event interference. This invention is applicable to the fields of intelligent traffic management and intersection signal optimization.

[0088] Based on the above Figure 1As can be seen from the implementation method, the embodiments of the present invention sequentially perform multi-target detection, cross-frame tracking, and three-dimensional spatial mapping on the monitoring video sequences of multiple directions at the intersection to obtain the three-dimensional spatial position and motion state information of traffic participants from each perspective; based on the three-dimensional spatial position and motion state information of traffic participants from each perspective, feature extraction, bird's-eye view feature spatial projection, and cross-view feature aggregation are sequentially performed to obtain the bird's-eye view features of the current frame; the bird's-eye view features of the current frame are sequentially subjected to long-term feature caching, historical frame feature retrieval related to the current scene, and temporal fusion to generate spatiotemporal fusion features and scene state feature vectors; traffic scene semantic parsing and traffic event detection are performed on the spatiotemporal fusion features to output event type and confidence level; based on the event and confidence level, the normal control mode or event response mode is selected; in the normal control mode, a phase timing scheme is generated based on the scene state feature vector; in the event response mode, the corresponding preset response strategy is triggered based on the event type; control commands are generated based on the phase timing scheme or preset response strategy, and traffic lights are controlled based on the control commands. This invention achieves comprehensive perception of the overall traffic situation at intersections by performing multi-target detection, cross-frame tracking, and 3D spatial mapping on surveillance video sequences from multiple directions. This aligns and fuses visual information from various perspectives within a unified bird's-eye view feature space. By performing long-term feature caching, historical frame feature retrieval, and temporal fusion on the bird's-eye view features of the current frame, the system can utilize historical information to understand the evolution of traffic conditions and capture slow-moving patterns such as queue length increases. Furthermore, by detecting traffic events using spatiotemporal fusion features, the system selects between normal mode and event response mode based on event type and confidence level. In event response mode, it automatically triggers corresponding preset response strategies, establishing a direct linkage mechanism between event detection and signal control. This enables the automatic triggering of appropriate control strategies in abnormal scenarios such as traffic accidents and special vehicle passage.

[0089] Based on the same inventive concept, as an implementation of the above-mentioned traffic signal control method based on multi-view fusion and temporal characteristics, this embodiment of the invention also provides a traffic signal control system based on multi-view fusion and temporal characteristics. Figure 4 This is a structural diagram of the traffic signal control system based on multi-view fusion and temporal characteristics in an embodiment of the present invention. See also... Figure 4 As shown, the system may include: The multi-view visual perception module 401 is used to perform multi-target detection, cross-frame tracking and three-dimensional spatial mapping on the monitoring video sequence of multiple directions at the intersection in sequence, so as to obtain the three-dimensional spatial position and motion state information of traffic participants from each viewpoint. The projection and feature aggregation module 402 is used to extract features, project bird's-eye view features into space, and aggregate cross-view features in sequence based on the three-dimensional spatial position and motion state information of traffic participants from each viewpoint, so as to obtain the bird's-eye view features of the current frame. The retrieval and temporal fusion module 403 is used to sequentially perform long temporal feature caching, historical frame feature retrieval related to the current scene, and temporal fusion on the bird's-eye view features of the current frame to generate spatiotemporal fusion features and scene state feature vectors. The scene parsing and detection module 404 is used to perform traffic scene semantic parsing and traffic event detection on spatiotemporal fusion features, and output the event type and confidence level. The signal decision module 405 is used to select to enter the normal control mode or the event response mode based on the event and the confidence level. In the normal control mode, it generates a phase timing scheme based on the scene state feature vector. In the event response mode, it triggers the corresponding preset response strategy based on the event type. The control module 406 is used to generate control commands based on the phase timing scheme or preset response strategy, and to control traffic lights according to the control commands.

[0090] The multi-view visual perception module 401 is specifically used to perform multi-target detection on each video frame in the surveillance video sequence of multiple directions at the intersection, and output the bounding box and category label of each target; based on the bounding box and category label of each target, it performs cross-frame association matching on each target to generate the continuous motion trajectory of each target. The continuous motion trajectory of each target includes the position sequence, velocity sequence and acceleration sequence of each target in time; based on the continuous motion trajectory of each target, it estimates the depth value of each target, and based on the depth value and camera intrinsic and extrinsic parameters, it maps each target from image coordinates to a three-dimensional world coordinate system with the center of the intersection as the origin, so as to obtain the position coordinates, orientation angle and velocity vector of each target on the intersection plane.

[0091] The projection and feature aggregation module 402 is specifically used to input the video frames from each channel into a hierarchical visual encoder with shared weights to extract multi-scale feature maps from each viewpoint; based on the three-dimensional spatial position and motion state information, the multi-scale feature maps from each viewpoint are projected onto the bird's-eye view feature space using a deformable attention mechanism to obtain multi-view information, and the multi-view information is aggregated through multi-head attention to obtain the initial bird's-eye view features of the current frame; the initial bird's-eye view features of the current frame are temporally aligned with the bird's-eye view features of the previous frame to obtain the bird's-eye view features of the current frame.

[0092] In the projection and feature aggregation module 402, based on the three-dimensional spatial position and motion state information, a deformable attention mechanism is used to project multi-scale feature maps of each viewpoint onto the bird's-eye view feature space to obtain multi-view information, including: setting a two-dimensional grid covering the intersection area in the bird's-eye view feature space; predicting the sampling offset corresponding to each grid position under each viewpoint for each grid position in the two-dimensional grid based on the three-dimensional spatial position and motion state information; sampling on the corresponding multi-scale feature maps of each viewpoint based on the sampling offset to obtain the feature vector of each viewpoint at the grid position; traversing all grid positions, and determining the multi-view information of each viewpoint at all grid positions based on the feature vector of each viewpoint at the grid position corresponding to all grid positions.

[0093] The retrieval and temporal fusion module 403 is specifically used to linearly compress the bird's-eye view features of the current frame to obtain a low-dimensional feature vector, and sequentially write the low-dimensional feature vector into a circular feature queue to construct a long temporal feature cache; through a cross-attention mechanism, it retrieves historical frame features related to the current scene from the long temporal feature cache to determine the retrieved historical frame features; and temporally fuses the bird's-eye view features of the current frame with the retrieved historical frame features to generate spatiotemporal fusion features and scene state feature vectors.

[0094] In the retrieval and temporal fusion module 403, the bird's-eye view features of the current frame are temporally fused with the retrieved historical frame features to generate spatiotemporal fusion features and scene state feature vectors. This includes: concatenating the bird's-eye view features of the current frame with the retrieved historical frame features in the channel dimension to obtain concatenated features; performing three-dimensional convolution processing on the concatenated features to obtain motion enhancement features; and sequentially performing self-attention processing, global average pooling, and linear projection on the motion enhancement features to generate spatiotemporal fusion features and scene state feature vectors.

[0095] The scene parsing and detection module 404 is specifically used to input spatiotemporal fusion features into multiple parallel task detection heads, so as to use the scene semantic parsing detection head in the task detection head to process the spatiotemporal fusion features and estimate at least one of lane occupancy rate, queue length and average speed; and use multiple event detection heads in the task detection head to process the spatiotemporal fusion features and identify event types and corresponding confidence levels.

[0096] The signal decision module 405 is specifically used to determine whether no event has been detected or whether the confidence level is lower than a preset threshold. If so, it enters the normal control mode, inputs the scene state feature vector into the hybrid expert decision network, and uses the gating unit in the hybrid expert decision network to calculate the weights of each expert feedforward network in the hybrid expert decision network based on the scene state feature vector. Using each expert feedforward network, it outputs a timing strategy based on the scene state feature vector, and performs a weighted combination of the timing strategy according to the weights to generate a phase timing scheme. The phase timing scheme includes the phase sequence and the green light duration of each phase. If not, it enters the event response mode and triggers the corresponding preset response strategy according to the event type.

[0097] In the signal decision module 405, when the event type is a special vehicle passage event, the signal light in the direction of the special vehicle is switched to green, and the signal lights in other directions are kept red until the special vehicle passes through the intersection, thus completing the priority release scheme. When the event type is a traffic accident event, the signal light in the direction corresponding to the lane where the accident occurred is locked to red, and the phase that conflicts with the lane where the accident occurred is kept red, thus completing the accident isolation scheme. When the event type is a pedestrian running a red light event, the green light time in the direction of the pedestrian is extended to a preset safe duration, thus completing the safety extension scheme. When the event type is an intersection overflow event, the proportion of green light time in the upstream direction of the overflow direction is increased, and the release phase time corresponding to the overflow direction is reduced, thus completing the congestion relief scheme. After the event ends, the scheme executed in the event response mode is gradually transitioned back to the phase timing scheme of the normal control mode within a preset number of signal cycles.

[0098] In the system of the present invention, each module is deployed on the roadside edge computing unit, and the modules interact with each other through shared memory or message queues.

[0099] It should be noted that the above description of the traffic signal control system embodiment based on multi-view fusion and temporal characteristics is similar to the description of the traffic signal control method embodiment based on multi-view fusion and temporal characteristics, and has similar beneficial effects. For technical details not disclosed in the embodiments of the traffic signal control system based on multi-view fusion and temporal characteristics of this invention, please refer to the description of the traffic signal control method embodiment based on multi-view fusion and temporal characteristics of this invention for understanding.

[0100] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A traffic signal control method based on multi-view fusion and temporal characteristics, characterized in that, include: Multi-target detection, cross-frame tracking, and 3D spatial mapping are performed sequentially on the surveillance video sequences from multiple directions at the intersection to obtain the 3D spatial position and motion state information of traffic participants from each perspective. Based on the three-dimensional spatial position and motion state information of traffic participants from each perspective, feature extraction, bird's-eye view feature spatial projection, and cross-perspective feature aggregation are performed sequentially to obtain the bird's-eye view features of the current frame. The bird's-eye view features of the current frame are sequentially cached for long-term features, retrieved for historical frame features related to the current scene, and fused temporally to generate spatiotemporal fusion features and scene state feature vectors. The spatiotemporal fusion features are used for traffic scene semantic parsing and traffic event detection, and the event type and confidence level are output. Based on the event and the confidence level, select to enter normal control mode or event response mode. In normal control mode, generate phase timing scheme based on scene state feature vector. In event response mode, trigger corresponding preset response strategy based on event type. Control commands are generated based on the phase timing scheme or the preset response strategy, and traffic lights are controlled according to the control commands.

2. The traffic signal control method based on multi-view fusion and temporal features according to claim 1, characterized in that, The monitoring video sequences from multiple directions at the intersection are sequentially subjected to multi-target detection, cross-frame tracking, and 3D spatial mapping to obtain the 3D spatial position and motion state information of traffic participants from each perspective, including: Perform multi-target detection on each video frame in the surveillance video sequence from multiple directions at the intersection, and output the bounding box and category label of each target; Based on the bounding boxes and category labels of each target, cross-frame association matching is performed on each target to generate continuous motion trajectories of each target. The continuous motion trajectories of each target include the position sequence, velocity sequence and acceleration sequence of each target in time. Based on the continuous motion trajectory of each target, the depth value of each target is estimated, and based on the depth value and the camera's intrinsic and extrinsic parameters, each target is mapped from the image coordinates to a three-dimensional world coordinate system with the intersection center as the origin, so as to obtain the position coordinates, orientation angle and velocity vector of each target on the intersection plane.

3. The traffic signal control method based on multi-view fusion and temporal features according to claim 1, characterized in that, The process involves extracting features, projecting bird's-eye view features into space, and aggregating cross-view features based on the three-dimensional spatial positions and motion states of traffic participants from each perspective, to obtain the bird's-eye view features of the current frame. This includes: Each video frame is input into a hierarchical visual encoder with shared weights to extract multi-scale feature maps from each viewpoint. Based on the three-dimensional spatial position and the motion state information, the multi-scale feature maps of each viewpoint are projected onto the bird's-eye view feature space using a deformable attention mechanism to obtain multi-view information. The multi-view information is then aggregated through multi-head attention to obtain the initial bird's-eye view features of the current frame. The initial bird's-eye view features of the current frame are temporally aligned with the bird's-eye view features of the previous frame to obtain the bird's-eye view features of the current frame.

4. The traffic signal control method based on multi-view fusion and temporal features according to claim 3, characterized in that, The step involves projecting multi-scale feature maps from each perspective onto the bird's-eye view feature space using a deformable attention mechanism based on the three-dimensional spatial position and the motion state information to obtain multi-view information, including: A two-dimensional grid covering the intersection area is set in the feature space of the bird's-eye view; Based on the three-dimensional spatial position and the motion state information, predict the sampling offset of each grid position in the two-dimensional grid under each viewpoint; Based on the sampling offset, sampling is performed on the multi-scale feature map of each viewpoint to obtain the feature vector of each viewpoint at the grid position. Traverse all grid positions and determine the multi-view information of each view at all grid positions based on the feature vectors of each view at each grid position.

5. The traffic signal control method based on multi-view fusion and temporal features according to claim 1, characterized in that, The process of sequentially performing long-term temporal feature caching, historical frame feature retrieval related to the current scene, and temporal fusion on the bird's-eye view features of the current frame to generate spatiotemporal fusion features and scene state feature vectors includes: The bird's-eye view features of the current frame are linearly compressed to obtain a low-dimensional feature vector, and the low-dimensional feature vector is sequentially written into a circular feature queue to construct a long-term feature cache. By using a cross-attention mechanism, historical frame features related to the current scene are retrieved from the long-term feature cache, and the retrieved historical frame features are determined. The bird's-eye view features of the current frame are fused with the retrieved historical frame features in a temporal sequence to generate the spatiotemporal fusion features and the scene state feature vector.

6. The traffic signal control method based on multi-view fusion and temporal features according to claim 5, characterized in that, The step of temporally fusing the bird's-eye view features of the current frame with the retrieved historical frame features to generate the spatiotemporal fusion features and the scene state feature vector includes: The bird's-eye view features of the current frame are concatenated with the retrieved historical frame features in the channel dimension to obtain the concatenated features; The spliced ​​features are subjected to three-dimensional convolution processing to obtain motion-enhanced features; The motion enhancement features are sequentially processed by self-attention, global average pooling, and linear projection to generate the spatiotemporal fusion features and the scene state feature vector.

7. The traffic signal control method based on multi-view fusion and temporal features according to claim 1, characterized in that, The process of performing traffic scene semantic parsing and traffic event detection on the spatiotemporal fusion features, and outputting event type and confidence level, includes: The spatiotemporal fusion features are input into multiple parallel task detection heads, and the scene semantic parsing detection head in the task detection head is used to process the spatiotemporal fusion features to estimate at least one of lane occupancy rate, queue length and average speed; and multiple event detection heads in the task detection head are used to process the spatiotemporal fusion features to identify the event type and corresponding confidence level.

8. The traffic signal control method based on multi-view fusion and temporal features according to claim 1, characterized in that, The process involves selecting between a normal control mode and an event response mode based on the event and the confidence level. In the normal control mode, a phase timing scheme is generated based on the scene state feature vector. In the event response mode, a corresponding preset response strategy is triggered based on the event type, including: Determine whether the event was not detected or whether the confidence level is lower than a preset threshold; If so, the system enters the normal control mode, inputs the scene state feature vector into the hybrid expert decision network, and uses the gating unit in the hybrid expert decision network to calculate the weights of each expert feedforward network in the hybrid expert decision network based on the scene state feature vector. Using each expert feedforward network, the system outputs a timing strategy based on the scene state feature vector, and performs a weighted combination of the timing strategy based on the weights to generate the phase timing scheme. The phase timing scheme includes a phase sequence and the green light duration for each phase. If not, then enter the event response mode and trigger the corresponding preset response strategy according to the event type.

9. The traffic signal control method based on multi-view fusion and temporal features according to claim 8, characterized in that, The step of triggering the corresponding preset response strategy based on the event type includes: When the event type is a special vehicle passage event, the traffic light in the direction of the special vehicle is switched to green, and the traffic lights in other directions are kept red until the special vehicle passes through the intersection, so as to complete the execution of the priority passage plan. When the event type is a traffic accident, the traffic light in the direction corresponding to the lane where the accident occurred will be locked to red, and the phase that conflicts with the lane where the accident occurred will be kept red, so as to complete the execution of the accident isolation plan. When the event type is a pedestrian running a red light, the green light time in the direction of the pedestrian is extended to a preset safe duration to complete the execution of the safety extension plan; When the event type is an intersection overflow event, the green light time ratio of the upstream oncoming traffic in the overflow direction is increased, and the release phase time corresponding to the overflow direction is reduced, so as to complete the execution of the congestion relief plan. After the event ends, the scheme executed in the event response mode will be gradually transitioned back to the phase timing scheme of the normal control mode within a preset number of signal cycles.

10. A traffic signal control system based on multi-view fusion and temporal characteristics, characterized in that, include: The multi-view visual perception module is used to perform multi-target detection, cross-frame tracking and three-dimensional spatial mapping on the monitoring video sequence from multiple directions at the intersection, so as to obtain the three-dimensional spatial position and motion state information of traffic participants from each viewpoint. The projection and feature aggregation module is used to extract features, project bird's-eye view features into space, and aggregate cross-view features in sequence based on the three-dimensional spatial position and motion state information of traffic participants from each viewpoint, so as to obtain the bird's-eye view features of the current frame. The retrieval and temporal fusion module is used to sequentially perform long temporal feature caching, historical frame feature retrieval related to the current scene, and temporal fusion on the bird's-eye view features of the current frame to generate spatiotemporal fusion features and scene state feature vectors. The scene parsing and detection module is used to perform traffic scene semantic parsing and traffic event detection on the spatiotemporal fusion features, and output the event type and confidence level. The signal decision module is used to select to enter the normal control mode or the event response mode based on the event and the confidence level. In the normal control mode, it generates a phase timing scheme based on the scene state feature vector. In the event response mode, it triggers the corresponding preset response strategy based on the event type. The control module is used to generate control commands based on the phase timing scheme or the preset response strategy, and to control traffic lights based on the control commands.

Images