Traffic behavior real-time identification method based on multi-modal spatial-temporal feature fusion

Abstract

The invention relates to the technical field of image processing, and discloses a traffic behavior real-time identification method based on multi-modal spatial-temporal feature fusion. Traffic scene videos, images and traffic signal lamp state information are collected through a road monitoring camera, and target detection and tracking are carried out on video frames to obtain a target spatial-temporal trajectory; respectively extracting a target visual feature sequence and a signal feature sequence after signal lamp state coding by using a time sequence-space perception module, inputting the two sequences into a multi-modal semantic coupling and fusion network, and generating a fusion feature vector through feature alignment; and based on the vector, target behaviors are discriminated in real time through a behavior classifier, and a normal or abnormal detection result is output and an alarm is given. According to the method, multi-modal information is fused, semantic understanding and dynamic expression are enhanced, recognition robustness is improved, real-time performance and flexibility are achieved, misjudgment and missed judgment can be effectively reduced, and the method has great significance in improvement of the efficiency and the safety level of an intelligent traffic system.

Description

Technical Field

[0001] The present invention relates to a real-time traffic behavior recognition method based on multimodal spatiotemporal feature fusion, and belongs to the technical field of image processing. Background Art

[0002] With the rapid development of urban transportation systems and the increasing volume of traffic, road safety issues are becoming increasingly prominent. This is particularly true in high-density areas such as urban intersections, school perimeters, and shopping districts, where numerous potential traffic safety hazards exist. The most common include irregular behaviors such as pedestrians running red lights, ignoring traffic signals, non-motorized vehicles crossing vehicles at random, and motor vehicles crossing lanes. These complex behaviors often occur suddenly, are hidden, and carry high risks, making them a major factor in traffic accidents. Therefore, efficient, accurate, and real-time detection and early warning of these abnormal behaviors has become a key issue in the research of intelligent transportation systems.

[0003] Currently, traditional methods for traffic behavior detection rely on single-modality visual information analysis. These methods primarily collect video data from surveillance cameras and combine it with computer vision techniques such as object detection and tracking to identify and analyze targets within a specific area. While these methods have some application value in static scenarios, they often suffer from low recognition accuracy, poor robustness to environmental changes, and a lack of semantic understanding in complex and dynamic traffic environments. For example, the system may be unable to distinguish between pedestrians waiting at a red light and those illegally crossing a zebra crossing, resulting in frequent false positives and false negatives.

[0004] On the other hand, traffic light status, as crucial semantic information for regulating traffic behavior, is underutilized in most existing methods. In real-world scenarios, the legality of the same behavior under different traffic light states can vary significantly. For example, a left turn is legal on a green light, but illegal on a red light. Behavior recognition methods that rely solely on image features are unable to make semantic judgments about the legality of behavior, severely limiting the system's practicality and reliability. Furthermore, due to the strong temporal nature and multi-source heterogeneity of traffic data, methods that rely solely on image processing struggle to establish effective spatiotemporal correlations and lack the ability to dynamically model the evolution of behavior.

[0005] In recent years, with the development of deep learning technology, deep learning models have made significant progress in image and video understanding tasks. At the same time, multimodal learning has gradually become an important research direction in the field of artificial intelligence, integrating information from different sensory sources to achieve more comprehensive and robust decision-making. However, in the field of traffic behavior recognition, deep multimodal recognition methods that integrate video information and traffic light status are still relatively scarce. In particular, there is a lack of a systematic, multi-module collaborative real-time recognition architecture to address key issues such as temporal alignment of traffic signals and visual information, multimodal fusion, and semantic discrimination that arise in actual deployments. Summary of the Invention

[0006] To address the existing issues of low accuracy and slow response in detecting abnormal behaviors in traffic scenarios, this paper proposes a real-time identification method for complex traffic behaviors based on multimodal fusion. This method utilizes road camera video information and traffic light status information, using a deep learning network to achieve real-time detection and identification of complex behaviors. The following describes the specific modules:

[0007] In a first aspect, the present invention provides a method for real-time identification of traffic behavior by fusion of multimodal spatiotemporal features, the method comprising:

[0008] Step 1: Obtain video sequences and static images of traffic scenes through road monitoring cameras, and simultaneously obtain the status information of current traffic lights;

[0009] Step 2: Perform target detection and multi-target tracking on each frame of the video sequence, locate pedestrians, motor vehicles, non-motor vehicles and other targets, and record the spatiotemporal trajectory of each target;

[0010] Step 3: For each target image sequence obtained by tracking, the temporal-spatial perception module is used to extract the visual feature sequence. At the same time, the traffic light status information at the synchronization moment is encoded to obtain the corresponding signal feature sequence.

[0011] Step 4: Input the above visual feature sequence and signal feature sequence into the multimodal semantic coupling and fusion network. During the network training process, a multimodal feature synchronization optimization function is used to align the visual modality features and the signal modality features at the same time step, and a fusion strategy is used to generate the target fusion feature vector.

[0012] Step 5: Based on the fused feature vector, a behavior classifier is used to perform real-time behavior discrimination, output normal behavior detection results of pedestrians, vehicles, non-motor vehicles and other targets, or abnormal behavior detection results such as illegal crossing of traffic lights, and trigger an alarm.

[0013] The data collection process includes: using cameras installed at intersections to capture real-time traffic video streams and scene images, while simultaneously obtaining synchronized traffic light status (red / yellow / green) through a signal control system; performing image enhancement and denoising on the video frames, and then using a deep object detection algorithm to detect pedestrians, vehicles, non-motor vehicles, and other targets in the video frames to obtain target bounding box information; and correlating the detection results based on a multi-target tracking algorithm to obtain the spatiotemporal trajectory of each target in consecutive frames. This process ensures accurate tracking of the target of interest and provides target alignment input for subsequent feature extraction.

[0014] The temporal-spatial perception module extracts visual spatiotemporal features from each preprocessed target image sequence. This involves using a deep convolutional neural network to extract spatial semantic features from single-frame images, while also applying a recurrent neural network to encode consecutive frames to capture the temporal dynamics of target motion.

[0015] Specifically, the features of each frame obtained by pre-training CNN Input LSTM to get the temporal hidden state The formula is:

[0016]

[0017] Finally, the target spatiotemporal feature vector sequence is obtained At the same time, the state information of the traffic light (red / yellow / green) is mapped into a signal modal feature vector sequence through a one-dimensional embedding layer. Each of these Aligned with the video frame features at the corresponding moment.

[0018] The multimodal semantic coupling and fusion network is responsible for aligning and fusing visual features and signal features. First, a multimodal feature synchronization optimization function L is designed. align , mandatory visual modality features and signal modal characteristics Expressing consistent semantic information at the same time step can be defined as:

[0019]

[0020] Where T is the sequence length and |·|2 is the Euclidean norm. By optimizing this loss function, the features of different modalities can be aligned during the training process. Then, the aligned multimodal features are input into the fusion network for comprehensive modeling. The fusion strategy uses a learnable linear transformation plus an activation function for fusion:

[0021]

[0022] Where σ is the activation function, Wv ,W s , b is the model parameter.

[0023] In one embodiment of the present invention, the fused features It is input into the classifier for behavior recognition; suppose there are K types of target behaviors (such as normal traffic, running a red light, right-turn violation, etc.), and a fully connected network is used to output the score s for each category k :

[0024]

[0025] where w k and b k are the weights and biases of the classifier; the probability of each category is calculated by Softmax:

[0026]

[0027] When the probability p of a specific category (such as "illegally crossing a red light") red_violation When the preset threshold is exceeded, it can be determined as abnormal behavior and an alarm is triggered. The classification result not only considers visual information but also refers to the status of traffic lights, achieving semantic-level behavior discrimination. During the training process, cross entropy loss is used:

[0028]

[0029] Optimize the classification results, where y k is the one-hot vector of the true label.

[0030] In a second aspect, the present invention provides a multimodal fusion complex behavior real-time recognition system, the system comprising:

[0031] The data acquisition and preprocessing module is mainly responsible for collecting multimodal data from traffic intersection scenes in real time and performing preliminary processing on it to build the input basis for subsequent recognition tasks;

[0032] The module continuously acquires video streams and image data through the camera equipment installed in the intersection area; at the same time, it accesses the traffic signal control system to obtain traffic light status information synchronized with the video frame, including signal changes such as red, yellow and green lights; the acquired video image data is first processed by image enhancement and noise suppression to improve image quality. Subsequently, a target detection method based on a deep neural network is used to accurately identify pedestrians, motor vehicles, non-motor vehicles and other targets in the image, and extract their boundary position information; further, combined with multi-target tracking technology, the detection results in consecutive frames are temporally correlated to obtain the continuous trajectory of each target in the time dimension, ensuring that subsequent modules can accurately focus on the same target object. The visual trajectory temporal analysis module is used to perform dynamic modeling and semantic feature extraction of the image sequence for each identified and tracked traffic participant to obtain a characteristic description of its behavioral performance;

[0033] In the spatial dimension, the module uses a deep convolutional network structure to extract the appearance and semantic information of the target in each frame of the image; in the temporal dimension, the image representation of the target in consecutive frames is input into a recurrent neural network or a temporal modeling network for encoding to capture its motion trajectory and behavioral evolution process;

[0034] The module not only establishes the temporal continuity of visual features, but also retains information about key action changes. In addition, the state information of traffic lights is also mapped into a vector representation that can be used for learning at this stage and aligned with the visual features in chronological order, thus laying the foundation for subsequent multimodal fusion.

[0035] The modality alignment and fusion module mainly solves the problem of collaborative modeling between visual modalities and signal modalities, and realizes complementary enhancement of multi-source information through feature alignment and deep fusion;

[0036] This module establishes a modal alignment mechanism to ensure consistency in semantic information expressed by different modalities at the same time point, thereby reducing representational differences between modalities. This process is constrained by optimizing an objective function to keep the features of each modality close in a specific metric space. The aligned visual features and traffic light features are then fed into a fusion network for joint modeling. The fusion method can use static weighting, dynamic attention mechanisms, or structured transformations to achieve deep coupling of modal information. Through the design of this module, the system can effectively integrate behavioral cues at the visual and signal levels, improving the expressiveness of complex traffic behavior recognition.

[0037] The behavior classification and decision module classifies and judges traffic behavior types based on the aforementioned fusion features, and realizes the identification and warning of abnormal behavior;

[0038] The fused features are input into the behavior recognition network and combined with the fully connected layer for high-dimensional space mapping, outputting the discrimination results of multiple types of traffic behaviors; the behavior categories may include common and abnormal actions such as normal passage, running a red light, and illegal crossing; the system determines whether there is an illegal behavior based on the output confidence score, and triggers an alarm signal when necessary to prompt the monitoring system to respond; during the training process, combined with the supervised learning strategy, the model is trained and performance optimized through pre-labeled data samples, thereby improving the classification accuracy and system robustness; this module comprehensively utilizes visual and signal modal information to achieve intelligent recognition of abnormal traffic behaviors while ensuring real-time performance.

[0039] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for real-time identification of traffic behaviors by fusion of multimodal spatiotemporal features are implemented.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for real-time identification of traffic behavior by fusion of multimodal spatiotemporal features.

[0041] The beneficial effects of the present invention are:

[0042] The present invention proposes a real-time traffic behavior recognition method based on the fusion of multimodal spatiotemporal features, which breaks through the limitations of traditional single-modal recognition, integrates visual and traffic light information, and improves the semantic understanding ability of behavior recognition; by introducing time series modeling and multimodal alignment mechanism, it enhances the dynamic expression ability of behavior evolution process and significantly improves recognition robustness; it has good real-time performance and deployment flexibility, and is suitable for various complex traffic environments such as urban roads, school surroundings, intersections, etc. It can effectively reduce the misjudgment rate and missed judgment rate, improve the overall efficiency and safety level of the intelligent transportation system, and has significant practical application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without inventive moves.

[0044] Figure 1 A flowchart of a method for real-time identification of traffic behavior based on multimodal spatiotemporal feature fusion provided by the present invention;

[0045] Figure 2The difference between the existing single-modal process provided by the present invention and the multimodal fusion framework of the present invention in the use of tags;

[0046] Figure 3 Schematic diagram of the zero-sample inference process provided by the present invention;

[0047] Figure 4 A schematic diagram of the overall architecture of the multimodal semantic coupling and fusion network provided by the present invention;

[0048] Figure 5 Detailed schematic diagram of the visual adapter provided by the present invention in front of and inside the network;

[0049] Figure 6 This is a structural diagram of the network post-vision adapter provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] Example 1:

[0052] This embodiment provides a method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion, the method comprising:

[0053] Step 1: Obtain video sequences and static images of traffic scenes through road monitoring cameras, and simultaneously obtain the status information of current traffic lights;

[0054] Step 2: Perform target detection and multi-target tracking on each frame of the video sequence, locate pedestrians, motor vehicles, non-motor vehicles and other targets, and record the spatiotemporal trajectory of each target;

[0055] Step 3: For each target image sequence obtained by tracking, the temporal-spatial perception module is used to extract the visual feature sequence. At the same time, the traffic light status information at the synchronization moment is encoded to obtain the corresponding signal feature sequence.

[0056] Step 4: Input the above visual feature sequence and signal feature sequence into the multimodal semantic coupling and fusion network. During the network training process, a multimodal feature synchronization optimization function is used to align the visual modality features and the signal modality features at the same time step, and a fusion strategy is used to generate the target fusion feature vector.

[0057] Step 5: Based on the fused feature vector, a behavior classifier is used to perform real-time behavior discrimination, output normal behavior detection results of pedestrians, vehicles, non-motor vehicles and other targets, or abnormal behavior detection results such as illegal crossing of traffic lights, and trigger an alarm.

[0058] Example 2:

[0059] See also Figure 1-Figure 5 This embodiment provides a method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion, the method comprising:

[0060] S1: Data Collection Process: This invention uses a multimodal perception system deployed at traffic intersections to collect multi-source heterogeneous data in real time from traffic scenes, including continuous video image streams and traffic light control status information. Video data is captured by high-definition surveillance cameras installed at the intersection, covering key areas such as pedestrian, vehicle, and non-motorized vehicle lanes. Traffic light status is captured by accessing the signal control system, recording the real-time phase status of red, yellow, and green lights.

[0061] Specifically, in order to improve the quality of video data, the collected image frames first undergo image enhancement and denoising preprocessing to enhance recognition capabilities in low-light or blurry scenes. Subsequently, the target detection model constructed using a deep neural network is used to identify and locate traffic participants (including pedestrians, motor vehicles, and non-motor vehicles) appearing in each frame of the image, and their bounding box information is extracted. Combined with the multi-target tracking algorithm, the target detection results between adjacent frames are temporally correlated to form a continuous motion trajectory of the target in the time dimension. At the same time, the status information of traffic lights is synchronized and aligned according to the frame-level timestamp to ensure that each target frame sequence matches the signal state at the corresponding moment, thereby providing a strictly aligned data foundation for subsequent multimodal feature extraction and fusion learning.

[0062] S2: Semantic enhancement using language modalities.

[0063] The present invention firstly takes language modality as a component of a multimodal system and performs semantic enhancement on traffic behavior labels.

[0064] Specifically, such as Figure 3 As shown, for concise traffic violation labels such as "running a red light" and "driving against traffic," their semantic information is enriched by adding contextual descriptions or synonymous phrases. By feeding the behavior label text into a language encoder and learning it alongside video features, this system strengthens the video feature representation based on a multimodal learning framework based on video-text matching. This strategy leverages textual semantic supervision during the training phase. By calculating the similarity between video features and corresponding text label features, the video and text representations of the same category are brought closer in the feature space, thereby improving the classifier's sensitivity to label semantics and recognition accuracy.

[0065] S3: “Pre-training-Adaptation-Fine-tuning” strategy

[0066] This paper adopts a new pre-training-adaptation-fine-tuning paradigm, reusing a large-scale pre-trained vision-language model as the base network. While retaining the original weights of the pre-trained model, it locally adjusts the model through the design of an adapter, thereby achieving improved transferability to new tasks and zero-shot recognition capabilities.

[0067] Specifically, such as Figure 4 As shown, text adapters, network intra-adapters, and network post-adapters are introduced.

[0068] Specifically, the text adapter inserts learnable parameters into the label text input, expanding and optimizing the text feature representation by adding trainable vectors or multi-semantic descriptions to the text prompt. This approach is similar to expanding the prompt, enhancing the label semantics with richer language expressions.

[0069] Specifically, such as Figure 5 As shown, the intra-network adapter inserts a temporal modeling module within the visual feature extraction network to enable spatiotemporal information exchange. A temporal shift module is inserted between every two adjacent network layers to enable inter-frame information exchange by shifting some channel features along the temporal dimension. This module does not increase trainable parameters and can be directly reused with pre-trained network weights to avoid catastrophic forgetting.

[0070] Specifically, the post-network adapter adopts a staged spatial-temporal encoding structure, first processing each frame using a pre-trained spatial encoder, and then processing the frame sequence relationship using a separate temporal encoder. Specifically, the spatial encoder extracts spatial features from each frame, then concatenates the frame-level features in temporal order and inputs them into the temporal encoder for temporal modeling. The temporal encoder applies mean pooling, one-dimensional convolution, and an LSTM recurrent network to the frame-level feature sequence. This structure enables the model to enhance its ability to capture temporal dynamics while retaining strong spatial representation capabilities.

[0071] S4: Spatiotemporal feature extraction

[0072] Specifically, the temporal-spatial perception module extracts visual spatiotemporal features from each pre-processed target image sequence. A deep convolutional neural network is used to extract spatial semantic features from single-frame images, while a recurrent neural network is applied to consecutive frame sequences for encoding to capture the temporal dynamics of target motion.

[0073] Specifically, if Figure 6As shown in the figure, in one implementation, a CNN is used to extract spatial features from each frame, and the frame sequence is then fed into a long short-term memory (LSTM) network for temporal modeling to capture motion information. In another implementation, a visual transformer (ViT) is used to extract frame features, and then a temporal encoder based on the Transformer structure performs multi-layer self-attention calculations on the frame sequence. Both schemes clearly distinguish between spatial feature extraction and temporal dependency modeling.

[0074] Specifically, the features of each frame obtained by pre-training CNN Input LSTM to get the temporal hidden state The formula is:

[0075]

[0076] Finally, the target spatiotemporal feature vector sequence is obtained At the same time, the state information of the traffic light (red / yellow / green) is mapped into a signal modal feature vector sequence through a one-dimensional embedding layer. Each of these Aligned with the video frame features at the corresponding moment.

[0077] The multimodal semantic coupling and fusion network is responsible for aligning and fusing visual features and signal features. First, a multimodal feature synchronization optimization function L is designed. align , mandatory visual modality features and signal modal characteristics Expressing consistent semantic information at the same time step can be defined as:

[0078]

[0079] Where T is the sequence length and |·|2 is the Euclidean norm. By optimizing this loss function, the features of different modalities can be aligned during training. The aligned multimodal features are then input into the fusion network for comprehensive modeling. The fusion strategy uses a learnable linear transformation plus an activation function for fusion:

[0080]

[0081] Where σ is the activation function, W v ,W s , b is the model parameter.

[0082] S5: Semantic classification. In the behavior classification module, the present invention introduces a semantically guided classifier. The classifier uses a video-text matching strategy for classification: during training, a contrastive learning objective is used to make the video features of the correct category and its text label feature pair as close as possible; during inference, the similarity between the feature vector of each test video and the text vector of each candidate behavior label (for example, cosine similarity) is calculated, and the label with the highest similarity is selected as the predicted category. This classification method is different from the traditional fully connected classifier. It can directly use the semantic information of the label text to guide decision-making, thereby having the ability to predict zero samples across categories.

[0083] Specifically, such as Figure 3 As shown, the fused features It is input into the classifier for behavior recognition. Assume that there are K types of target behaviors (such as normal traffic, running a red light, right-turn violation, etc.), and use a fully connected network to output the score s for each category. k :

[0084]

[0085] where w k and b k are the weights and biases of the classifier. The probability of each category is calculated by Softmax:

[0086]

[0087] When the probability p of a specific category (such as "illegally crossing a red light") red_violation When the preset threshold is exceeded, it can be identified as abnormal behavior and an alarm is triggered. The classification result not only considers visual information, but also refers to the status of traffic lights, achieving semantic-level behavior discrimination. During the training process, cross entropy loss is used:

[0088]

[0089] Optimize the classification results, where y k is the one-hot vector of the true label.

[0090] Example 3:

[0091] This embodiment provides a multimodal fusion complex behavior real-time recognition system, the system comprising:

[0092] The data acquisition and preprocessing module is mainly responsible for collecting multimodal data from traffic intersection scenes in real time and performing preliminary processing on it to build the input basis for subsequent recognition tasks;

[0093] The module continuously acquires video streams and image data through the camera equipment installed in the intersection area; at the same time, it accesses the traffic signal control system to obtain traffic light status information synchronized with the video frame, including signal changes such as red, yellow and green lights. The acquired video image data is first processed by image enhancement and noise suppression to improve the image quality. Subsequently, a target detection method based on a deep neural network is used to accurately identify pedestrians, motor vehicles, non-motor vehicles and other targets in the image, and extract their boundary position information. Furthermore, combined with multi-target tracking technology, the detection results in consecutive frames are temporally correlated to obtain the continuous trajectory of each target in the time dimension, ensuring that subsequent modules can accurately focus on the same target object. The visual trajectory temporal analysis module is used to perform dynamic modeling and semantic feature extraction of the image sequence for each identified and tracked traffic participant to obtain a characteristic description of its behavioral performance;

[0094] In the spatial dimension, this module uses a deep convolutional network structure to extract the appearance and semantic information of the target in each frame. In the temporal dimension, the image representation of the target in consecutive frames is input into a recurrent neural network or a temporal modeling network for encoding to capture its motion trajectory and behavioral evolution.

[0095] This module not only establishes the temporal continuity of visual features but also preserves information about key motion changes. Furthermore, traffic light state information is mapped into a learnable vector representation at this stage and aligned with visual features in temporal order, laying the foundation for subsequent multimodal fusion.

[0096] The modality alignment and fusion module mainly solves the problem of collaborative modeling between visual modalities and signal modalities, and realizes complementary enhancement of multi-source information through feature alignment and deep fusion;

[0097] The module constructs a modal alignment mechanism to ensure consistency in the semantic information expressed by different modalities at the same time point, thereby reducing representational differences between modalities. This process is constrained by optimizing the objective function to keep the features of each modality close in a specific metric space. Subsequently, the aligned visual features and traffic light features are input into the fusion network for joint modeling. The fusion method can adopt static weighting, dynamic attention mechanism or structured transformation to achieve deep coupling of modal information. Through the design of this module, the system can effectively integrate behavioral cues at the visual and signal levels, improving the expressive ability of complex traffic behavior recognition.

[0098] The behavior classification and decision module classifies and judges traffic behavior types based on the aforementioned fusion features, and realizes the identification and warning of abnormal behavior;

[0099] The fused features are input into the behavior recognition network and combined with the fully connected layer for high-dimensional space mapping, which outputs the discrimination results of multiple types of traffic behaviors. Behavior categories may include common and abnormal actions such as normal passage, running red lights, and illegal crossing. The system determines whether there is an illegal behavior based on the output confidence score, and triggers an alarm signal when necessary to prompt the monitoring system to respond. During the training process, combined with the supervised learning strategy, model training and performance optimization are carried out through pre-labeled data samples, thereby improving classification accuracy and system robustness. This module comprehensively utilizes visual and signal modal information to achieve intelligent recognition of abnormal traffic behaviors while ensuring real-time performance.

[0100] Example 4:

[0101] In order to verify the technical effect of the present invention, this embodiment selected the traditional real-time identification method of traffic behavior based on multimodal spatiotemporal feature fusion and the real-time identification method of traffic behavior based on multimodal spatiotemporal feature fusion of the present invention for comparative experiments to verify the actual effect of the present invention.

[0102] This embodiment will be experimentally verified on a collected multimodal dataset of traffic behavior. This dataset contains about 100 hours of traffic surveillance videos of typical urban intersections, with corresponding synchronous recordings of traffic light status and target behavior label information. The data covers the real behaviors of pedestrians, non-motor vehicles and motor vehicles under different signal states, including normal traffic, crossing red lights, non-motor vehicles waiting on the line and other behavior types. Among them, 80 representative video clips were selected for testing, each about 30 seconds long, and all of them have been manually labeled to complete the behavior classification labels and time alignment information. The dataset fully covers actual traffic scenarios such as morning and evening rush hours, complex road conditions, occlusion and interference, to ensure that the test results are broadly representative and have real reference value.

[0103] The real-time traffic behavior recognition method based on multimodal spatiotemporal feature fusion of the present invention is compared with the traditional solution. The comparison results are shown in Table 1 below:

[0104] The above comparison data demonstrates that the present invention surpasses traditional solutions and expert manual modeling methods in both accuracy and efficiency, and is also faster. This comparison demonstrates that the present invention's real-time traffic behavior recognition method, based on multimodal spatiotemporal feature fusion, can rapidly, efficiently, accurately, and automatically detect traffic anomalies, demonstrating its strong practicality and application value.

[0105] Table 1: Comparison results between the present invention and traditional solutions

[0106]

[0107] In addition, the present invention also provides a computer device, which may include a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of a method for real-time identification of traffic behavior using multimodal spatiotemporal feature fusion as described in any of the above embodiments.

[0108] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of a method for real-time identification of traffic behavior by fusion of multimodal spatiotemporal features, and will not be described in detail here.

[0109] The present invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of a method for real-time traffic behavior identification using multimodal spatiotemporal feature fusion, as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device.

[0110] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of a method for real-time identification of traffic behavior by fusion of multimodal spatiotemporal features, and will not be described in detail here.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time traffic behavior recognition method based on multimodal spatiotemporal feature fusion, characterized by: The method comprises: Step 1: Obtain video sequences and static images of traffic scenes through road monitoring cameras, and simultaneously obtain the status information of current traffic lights; Step 2: Perform target detection and multi-target tracking on each frame of the video sequence, locate pedestrians, motor vehicles, non-motor vehicles and other targets, and record the spatiotemporal trajectory of each target; Step 3: For each target image sequence obtained by tracking, the temporal-spatial perception module is used to extract the visual feature sequence. At the same time, the traffic light status information at the synchronization moment is encoded to obtain the corresponding signal feature sequence. Step 4: Input the above visual feature sequence and signal feature sequence into the multimodal semantic coupling and fusion network. During the network training process, a multimodal feature synchronization optimization function is used to align the visual modality features and the signal modality features at the same time step, and a fusion strategy is used to generate the target fusion feature vector. Step 5: Based on the fused feature vector, a behavior classifier is used to perform real-time behavior discrimination, output normal behavior detection results of pedestrians, vehicles, non-motor vehicles and other targets, or abnormal behavior detection results such as illegal crossing of traffic lights, and trigger an alarm.

2. The method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion according to claim 1 is characterized in that: The data collection process of step 2 includes: collecting traffic video streams and scene images in real time through cameras installed at intersections, and obtaining synchronized traffic light status (red / yellow / green) through a signal control system; performing image enhancement and denoising on the video frames, and then using a deep object detection algorithm to detect pedestrians, vehicles, non-motor vehicles and other targets in the video frames to obtain the target's bounding box information; and correlating the detection results based on a multi-target tracking algorithm to obtain the spatiotemporal trajectory of each target in consecutive frames.

3. The method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion according to claim 1 is characterized in that: The temporal-spatial perception module in step 3 extracts visual spatiotemporal features from the temporal image sequence of each pre-processed target; on the one hand, a deep convolutional neural network is used to extract spatial semantic features from a single frame image; On the other hand, recurrent neural networks are applied to continuous frame sequences for encoding to capture the temporal dynamics of target motion; Specifically, the features of each frame obtained by pre-training CNN Input LSTM to get the temporal hidden state The formula is: Finally, the target spatiotemporal feature vector sequence is obtained At the same time, the state information of the traffic light (red / yellow / green) is mapped into a signal modal feature vector sequence through a one-dimensional embedding layer. Each of these Aligned with the video frame features at the corresponding moment.

4. The method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion according to claim 1 is characterized in that: The multimodal semantic coupling and fusion network in step 4 is responsible for aligning and fusing visual features and signal features. First, a multimodal feature synchronization optimization function L is designed. align , mandatory visual modality features and signal modal characteristics Expressing consistent semantic information at the same time step can be defined as: Where T is the sequence length and |·|2 is the Euclidean norm. By optimizing this loss function, the features of different modalities can be aligned during training. Then, the aligned multimodal features are input into the fusion network for comprehensive modeling; the fusion strategy uses a learnable linear transformation plus an activation function for fusion: Where σ is the activation function, W v ,W s , b is the model parameter.

5. The method for real-time traffic behavior recognition based on multimodal spatiotemporal feature fusion according to claim 1 is characterized in that: The step 4 is to combine the fused feature vector It is input into the classifier for behavior recognition; Assume that there are K types of target behaviors (such as normal traffic, running a red light, right-turn violation, etc.), and use a fully connected network to output the score s for each type. k : where w k and b k are the weights and biases of the classifier; the probability of each category is calculated by Softmax: When the probability p of a specific category (such as "illegally crossing a red light") red_violation When the preset threshold is exceeded, it can be determined as abnormal behavior and an alarm is triggered. The classification result not only takes into account visual information, but also refers to the status of traffic lights, achieving semantic-level behavior discrimination; during the training process, cross entropy loss is used: Optimize the classification results, where y k is the one-hot vector of the true label.

6. A real-time traffic behavior recognition system based on multimodal spatiotemporal feature fusion, characterized by: The system comprises: Data acquisition and preprocessing module: Data acquisition: Video streams and image data are collected through intersection cameras, and then connected to the traffic signal control system to obtain synchronized traffic light status information (red / yellow / green light signal changes). Preprocessing: First, the video and image are enhanced and noise suppressed. Then, a deep neural network target detection method is used to identify targets such as pedestrians, motor vehicles, and non-motor vehicles, and their boundary location information is extracted. Furthermore, combined with multi-target tracking technology, the detection results in consecutive frames are temporally correlated to obtain the continuous trajectory of each target in the time dimension. Visual Trajectory Temporal Analysis Module: This module uses a deep convolutional network structure to extract the appearance and semantic information of the target in each frame in the spatial dimension. In the temporal dimension, the image representation of the target in consecutive frames is input into a recurrent neural network or a temporal modeling network for encoding. This module not only establishes the temporal continuity of visual features and retains key action change information, but also maps traffic light status information into a vector representation that can be used for learning and aligns it with the visual features in chronological order. Modal Alignment and Fusion Module: This module constructs a modal alignment mechanism to reduce representational differences between modalities. This process is constrained by optimizing an objective function to keep the features of each modality close in a specific metric space. The aligned visual features and traffic light features are then fed into a fusion network for joint modeling. Fusion methods can employ static weighting, dynamic attention mechanisms, or structured transformations to achieve deep coupling of modal information. This module enables the system to effectively integrate behavioral cues from both the visual and signal levels, improving the expressive power of complex traffic behavior recognition. Behavior classification decision module: Based on the aforementioned fusion features, traffic behavior types are classified and judged, and abnormal behaviors are identified and alerted. The fused features are input into the behavior recognition network and combined with the fully connected layer for high-dimensional spatial mapping, outputting the judgment results of multiple types of traffic behaviors. Behavior categories may include common and abnormal actions such as normal passage, running red lights, and illegal crossing. The system determines whether there is an illegal behavior based on the output confidence score and triggers an alarm signal when necessary, prompting the monitoring system to respond. During the training process, the supervised learning strategy is combined with pre-labeled data samples for model training and performance optimization, thereby improving classification accuracy and system robustness. This module comprehensively utilizes visual and signal modal information to achieve intelligent recognition of abnormal traffic behaviors while ensuring real-time performance.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of a method for real-time identification of traffic behavior based on multimodal spatiotemporal feature fusion as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for real-time identification of traffic behavior based on multimodal spatiotemporal feature fusion as described in any one of claims 1 to 6 are implemented.

Images