Traffic video question answering method based on inter-frame difference

Through inter-frame difference and hierarchical contrast learning methods, the problem that existing models fail to capture sudden dynamic changes in traffic event analysis is solved, and the accuracy and efficiency of traffic video question answering are improved, especially the ability to recognize drastic motion changes such as sudden stops, turns or collisions of vehicles.

CN120689821APending Publication Date: 2025-09-23BEIJING UNIV OF TECH
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202510662725.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-22
Publication Date
2025-09-23

AI Technical Summary

Technical Problem

Existing video question-answering models fail to effectively capture sudden dynamic changes in traffic scenes in traffic event analysis, especially drastic motion changes such as sudden vehicle stops, turns, or collisions, resulting in the inability to identify key events in a timely manner.

Method used

The inter-frame difference method is used to extract the difference information between video frames. The visual and text features are combined. The perception ability of dynamic events is enhanced through the difference-visual feature fusion module and hierarchical contrast learning, and the contribution of static and dynamic information is balanced through the gated fusion mechanism.

Benefits of technology

It significantly improves the model's ability to identify emergencies in traffic scenes, improves the accuracy and computational efficiency of traffic video question answering, and enhances the ability to capture mutation features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120689821A_ABST
    Figure CN120689821A_ABST
Patent Text Reader

Abstract

The invention discloses a traffic video question and answer method based on inter-frame difference, and the method comprises the steps: innovatively constructing an inter-frame difference extraction module on the basis of an image pre-training model based on the local abrupt change characteristic of a traffic video: effectively capturing the abrupt change characteristic of a moving target through the pixel-level difference calculation of adjacent frames; and designing a gating feature fusion mechanism, dynamically aligning spatial distribution of differential features and visual features on the basis of fusing problem features, and realizing collaborative characterization of static scenes and dynamic changes. In order to enhance cross-modal feature alignment, a hierarchical comparison learning strategy is provided: differential-visual feature comparison is implemented at a bottom layer to enhance local dynamic perception, and text-visual comparison is performed at a high layer to establish global semantic association. Experimental results show that the classification performance is improved on the basis of the SUTD-TrafficQA reference data set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a video question-answering method for traffic systems and proposes a traffic video question-answering method based on inter-frame difference, which realizes semantic understanding of traffic scenes and dynamic event analysis through "vision + language" multimodal reasoning. Background Art

[0002] Thanks to breakthroughs in the Transformer architecture, large-scale text pre-training models (such as BERT and GPT-3) have demonstrated the ability to deeply model complex semantic structures and grammatical relationships through a self-supervised learning paradigm. To break through the semantic boundaries of unimodality, researchers have further explored the space of cross-modal joint representations and proposed multimodal pre-training models such as CLIP and DALL-E. These models establish cross-modal semantic associations through techniques such as contrastive learning and autoregressive generation, laying an important foundation for multimodal understanding.

[0003] As a core task in multimodal understanding, video question answering aims to establish a mapping relationship between visual dynamics and language semantics through spatiotemporal reasoning. Existing research mainly achieves cross-modal alignment through visual-language pre-training architecture. Among them, VideoBERT adopts a two-stream Transformer architecture and designs a video-text hierarchical encoder. It realizes the joint representation learning of video clips and text descriptions through the mask modality prediction task; Flamingo introduces a cross-modal retrieval enhancement mechanism and uses large-scale heterogeneous data to construct open semantic mappings, which significantly improves the generalization ability for unknown scenes. This type of method captures the global temporal patterns of videos through long sequence modeling and has made significant progress in general fields.

[0004] With the development of intelligent transportation systems, video question answering (VQA) has been introduced into the field of traffic incident analysis, creating the emerging task of Traffic Video Question Answering (Traffic VQA). This task requires models to analyze the microscopic motion patterns of traffic participants using surveillance cameras or on-board cameras, thereby performing decision-making reasoning such as event attribution and risk prediction. Methods based on this task can successfully identify potential correlations in scenarios such as collisions and violations through causal reasoning and generate "what-if" preventative plans through counterfactual analysis, significantly improving the interpretability and operability of traffic decision-making systems. However, existing methods fail to fully consider the unique spatiotemporal characteristics of traffic incident question answering when applied to the transportation domain. First, the video background from a fixed surveillance perspective is highly static, and the core dynamic information only exists in the local motion of road participants. Second, while the background from a mobile vehicle perspective changes, the primary focus remains on the motion of surrounding vehicles and pedestrians. Most importantly, traffic accidents are often accompanied by dramatic motion changes, such as sudden stops, turns, or collisions. These key events exhibit dramatic instantaneous differences between consecutive video frames. Existing video models, due to their focus on long-term temporal dependencies, ignore short-term drastic changes and fail to capture these sudden changes. Therefore, in traffic video question answering, capturing mutation features is far more important than modeling long-sequence videos. Summary of the Invention

[0005] A hierarchical contrastive learning method based on inter-frame difference is proposed to improve the model's ability to perceive dynamic sudden information in traffic incident question-answering tasks, while taking into account computational efficiency and cross-modal reasoning capabilities. Considering that sudden incidents in traffic scenes are often accompanied by obvious inter-frame changes, we introduce inter-frame difference operations to capture the mutation areas in video sequences. In this way, in static monitoring scenes, the fixed camera perspective and relatively static background enable inter-frame difference to effectively eliminate redundant information and focus on the significant change areas when the incident occurs. In vehicle-mounted video scenes, although there are certain dynamic changes in the background due to the movement of the vehicle body, the motion amplitude caused by sudden incidents is often more intense, so inter-frame difference can still highlight the core dynamic areas related to the accident. Figure 1 The figure shows the original video frame and the frame after frame difference. The gray area represents the subtracted portion. You can see that the frame after frame difference better highlights the subject while preserving the core background information. Furthermore, to alleviate the high computational cost associated with long video sequences, we use an image pre-training model to extract features from the original frame image and the difference image, generating visual features and differential features. Using an image pre-training model can significantly reduce training costs and improve the model's generalization capabilities in low-resource scenarios.

[0006] To fully integrate differential and visual features while preventing dynamic information from being overwhelmed by background features, we propose a differential-visual feature fusion module. This module consists of three components: a self-attention submodule, a cross-attention submodule, and a gated fusion mechanism. The self-attention submodule first operates on visual features to capture temporal dependencies between frames and emphasize the temporal dynamics of events. The cross-attention submodule uses differential features as queries and introduces self-attention-processed visual features as keys and values, enabling cross-modal information exchange. This allows the differential information to guide the visual features to focus on dynamically changing regions. The gated fusion mechanism dynamically adjusts the fusion strength of differential and visual features. Specifically, we concatenate question features, visual features, and differential features and feed them into a sigmoid-activated gating unit to generate a set of learnable weights, thereby achieving an adaptive weighted fusion of static and dynamic information. This mechanism ensures that the model preserves the semantic information provided by the background while emphasizing the critical role of sudden events in the question-answering task. Furthermore, to promote collaborative understanding between modalities, we designed a hierarchical contrastive alignment strategy to achieve multimodal learning at both the feature level and the semantic level: on the one hand, the alignment of differential features and visual features aims to narrow the distribution differences between the two in the representation space, provide a semantic basis for the cross-attention in the differential-visual feature fusion module, and enhance the model's ability to identify motion-sensitive areas; on the other hand, the alignment of text features and visual / differential features, through two-way contrastive learning, narrows the semantic distance between the problem semantics and static visual information and dynamically changing information, thereby improving the model's performance in multimodal understanding and cross-modal reasoning tasks.

[0007] The main steps of the present invention include the following:

[0008] Step 1: Define the TrafficVQA task:

[0009] The goal of the TrafficVQA task is to achieve semantic understanding and dynamic event analysis of traffic scenes using multimodal reasoning of "vision + language". It requires the model to select a candidate answer from the candidate answer pool given a video clip V of a traffic scene and a natural language question Q. Choose the correct answer a * The process can be described as follows:

[0010]

[0011] Where P is the function of the multimodal model to calculate the score of the candidate answer; V = {v1,v2,…,v T}, v t represents the t-th frame image; Q={q1,q2,…,q M}, consisting of M words.

[0012] Step 2: Establish inter-frame difference mechanism:

[0013] We introduce an inter-frame difference extraction mechanism to enhance the model’s spatiotemporal sensitivity to dynamic traffic events by calculating the pixel-level differences between adjacent frames of a video. Specifically, given a video sequence The calculation of the difference sequence ΔV is as follows:

[0014]

[0015] Where T is the number of frames, C is the number of channels, and H×W is the spatial resolution.

[0016] Since the difference sequence In the time dimension, it is better than the original video One frame less. To keep the timing aligned, we fill a zero tensor at the starting position of ΔV to obtain the aligned difference sequence D:

[0017] D={0,Δv1,Δv2,…,Δv T-1}

[0018] Step 3: Implement image text feature extraction:

[0019] For V in step 1 and D in step 2 , First, pixel value scaling is used to standardize the value range. The original pixel value range [0,255] is linearly mapped to the interval [0,1] through this process, avoiding the deviation of the optimization direction caused by the excessive value range during the gradient descent process. Z-score normalization is further used to align the data distribution with the predefined mean and standard deviation parameters, and the results are respectively and

[0020] Then, and Input the frozen pre-trained CLIP visual encoder respectively to generate high-level semantic embedding:

[0021]

[0022]

[0023] Where T is the number of frames and F is the feature dimension output by the CLIP visual encoder.

[0024] At the same time, the pre-trained BERT model is used to generate preliminary text embeddings, which are then mapped to the visual feature space through a fully connected layer:

[0025] F Q =MLP(BERT(Q))

[0026] F A =MLP(BERT(A))

[0027] Among them, Q is the question and A is the candidate answer.

[0028] Step 4: Build a differential-visual feature fusion module

[0029] The module consists of three parts: self-attention submodule, cross-attention submodule and gated fusion mechanism.

[0030] The original visual features F are V Perform spatiotemporal context modeling to capture long-range dependencies between video frames and emphasize the temporal dynamics of events:

[0031]

[0032] The self-attention mechanism is used to capture the temporal dependencies and global semantics of question text features and candidate answer text features:

[0033]

[0034] The cross-attention mechanism is used to align the differential features with the original features and use the original features as a reference to locate the dynamic event-sensitive areas in the differential sequence:

[0035]

[0036] Based on the question semantic feature F obtained in step 3 Q and the visual feature F in step 4 V and F D The combined information of ,generates dynamic fusion gating:

[0037]

[0038] Among them, Sigmoid is the activation function, W g To obtain learnable weights, the concatenated multimodal features are mapped to gated scalars.

[0039] The contribution of the original and differential features is balanced by the gating coefficient to generate the final fused visual representation:

[0040] F M =G⊙F D +(1-G)⊙F V

[0041] In this way, the gated fusion mechanism can adaptively enhance the weights of differential features based on the question semantics, focusing on the mutation areas. This mechanism ensures that the model can both retain the semantic information provided by the background and strengthen the critical role of mutation events in the question answering task.

[0042] Step 5: Decoding multimodal features

[0043] The gated fusion visual feature F M and question text features F Q Splicing along the feature dimension to form a multimodal joint input:

[0044] F cat =Concat(F M ,F Q )

[0045] Then, through the self-attention mechanism, visual and textual information are integrated, allowing the model to capture long-range dependencies within multimodal features:

[0046]

[0047] Finally, the fused features are mapped to the answer space:

[0048]

[0049] Step 6: Constructing Hierarchical Contrastive Learning

[0050] First, the original visual features F are constrained by unsupervised contrastive learning. V and differential feature F D The local consistency of forces the model to focus on the co-occurrence areas of dynamic events and static scenes in the video and suppress background noise:

[0051]

[0052] Where τ is the temperature coefficient used to control the fraction of contrast loss; B is the number of batches.

[0053] Secondly, through two-way contrastive learning, the semantic distance between the question semantics and static visual information and dynamically changing information is shortened, achieving text-visual cross-modal alignment:

[0054]

[0055] Finally, the alignment of supervised multimodal features and answer features is achieved through space-token two-layer contrastive learning:

[0056]

[0057] Among them, f iis the embedding of the fusion feature F within the batch, a i + is the embedding of the positive sample answer in the batch, a j - Embedding for negative answers within a batch.

[0058] The token-layer contrast loss calculates the similarity for each token of the answer text to enhance local semantic alignment:

[0059]

[0060] Step 7: Loss Function and Score Function

[0061] The final loss function is expressed as the weighted sum of the above layer contrastive learning losses:

[0062]

[0063] When evaluating the classification accuracy of the TrafficVQA task, the model calculates the cosine similarity between the semantic-visual features and all candidate answers and represents this similarity value as a score assigned to each candidate answer. Therefore, the scoring function can be defined as:

[0064] P(a|V,Q)=F·F A,K K=1,2,…,k

[0065] Among them, F A,k is the answer feature of the kth candidate answer, and F is the multimodal fusion feature.

[0066] Finally, the ratio of correctly identified true labels is calculated to determine the classification accuracy and overall model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 For the visualization of inter-frame difference in the present invention

[0068] Figure 2 The overall framework diagram based on inter-frame difference and hierarchical contrast learning DETAILED DESCRIPTION

[0069] The performance of the proposed method is compared with the classification performance of previous methods on the public dataset SUTD-TrafficQA. Then, ablation experiments are conducted to evaluate the impact of each model component.

[0070] The SUTD-TrafficQA dataset was established in 2021 to advance research on traffic event perception and reasoning in videos, which is closely related to applications such as intelligent transportation and assisted or autonomous driving. The dataset contains a total of 10,080 videos shot from real-world scenes and 62,535 question-answering pairs (QAs). These data come from traffic accident scenes on publicly available video sharing platforms such as YouTube, LiveLeak, Twitter, and Bilibili. The dataset in this method is divided into two subsets (training set and test set), the training set contains 56,460 QAs, and the test set consists of the remaining 6,075 QAs, and SUTD-TrafficQA covers six different and challenging tasks.

[0071] The performance of the proposed method and the baseline model are compared on the SUTD-TrafficQA dataset, and the experimental results are briefly summarized. Based on the results, the following conclusions can be drawn:

[0072] For VQA methods that do not use CLIP pre-trained models. Without CLIP pre-trained models, traditional VQA methods that rely on CNN, LSTM, Attention, GNN, or BERT features perform worse than VQA methods that utilize CLIP. Among these methods, VIS+LSTM and I3D+LSTM achieve the lowest accuracy because they simply concatenate visual features from CNN with textual features from long short-term memory (LSTM) and word embeddings. In contrast, HCRN, VQAC, and DualVGR use cross-modal attention mechanisms and LSTM or GNN features, which can explore multimodal features guided by questions, thus achieving better results.

[0073] For VQA methods using CLIP pre-trained models. The results in Table 1 show that the CLIP model, which was not specifically trained on the SUTD-TrafficQA dataset, performed the worst. However, through fine-tuning, adapters, or the application of LORA training techniques, the CLIP model was able to learn the distribution information of the SUTD-TrafficQA dataset, surpassing the performance of HCRN and DualVGR, confirming the critical importance of aligning multimodal features for VQA models.

[0074] Table 1 Experimental results of classification accuracy of SUTD-TrafficQA dataset

[0075]

[0076] TrafficVQA-oriented methods. Eclipse uses a model architecture that includes CNN, LSTM, and attention mechanism, but exhibits the lowest accuracy in the TrafficVQA model; CMCIR utilizes the Swin-Transformer pre-trained model, GNN, and cross-entropy loss to capture semantic relationships across multimodal features; Tem-Adapter aligns visual and textual features through hinge loss and learns temporal relationships in video features through generative loss; CFMMC-Align aligns textual and visual features through contrastive loss, achieving state-of-the-art performance compared to all other baseline methods. Our method achieves a 1.8% accuracy improvement over CFMMC-Align-Swin and a 1.3% accuracy improvement over CFMMC-Align-CLIP.

Claims

1. Traffic video question answering method based on inter-frame difference, the steps are as follows: Step 1: Define the TrafficVQA task; The goal of the TrafficVQA task is to achieve semantic understanding and dynamic event analysis of traffic scenes with "vision + language" multimodal reasoning; given a video clip V of a traffic scene and a natural language question Q, the candidate answer pool is selected. Choose the correct answer a * ; This process is described as: in, P is the function of the multimodal model to calculate the score of the candidate answer; V={v1,v2,…,v T }, v t represents the t-th frame image; Q={q1,q2,…,q M }, consisting of M words; Step 2: Establish an inter-frame difference mechanism; The inter-frame difference extraction mechanism is introduced to enhance the temporal and spatial sensitivity of the model to dynamic traffic events by calculating the pixel-level differences between adjacent frames of the video. Specifically, given a video sequence The calculation of the difference sequence ΔV is as follows: Where T is the number of frames, C is the number of channels, and H×W is the spatial resolution; Since the difference sequence In the time dimension, it is better than the original video One frame is missing. To keep the timing aligned, a zero tensor is filled at the starting position of ΔV to obtain the aligned differential sequence D: D={0,Δv1,Δv2,…,Δv T-1 } Step 3: Implement image text feature extraction: For V in step 1 and D in step 2, the value range is first standardized by pixel value scaling. The original pixel value range [0, 255] is linearly mapped to the interval [0, 1] through this process to avoid the optimization direction deviation caused by the excessive value range during the gradient descent process. Z-score normalization is used to align the data distribution with the predefined mean and standard deviation parameters, and the values ​​are obtained respectively. and Then, and Input the frozen pre-trained CLIP visual encoder respectively to generate high-level semantic embedding: Where T is the number of frames, and F is the feature dimension output by the CLIP visual encoder; At the same time, the pre-trained BERT model is used to generate preliminary text embeddings, which are then mapped to the visual feature space through a fully connected layer: F Q =MLP(BERT(Q)) F A =MLP(BERT(A)) Among them, Q is the question and A is the candidate answer; Step 4: Construct a differential-visual feature fusion module; The module consists of three parts: self-attention submodule, cross-attention submodule and gated fusion mechanism; The original visual feature F is processed by the self-attention mechanism V Perform spatiotemporal context modeling to capture long-range dependencies between video frames and emphasize the temporal dynamics of events: The self-attention mechanism is used to capture the temporal dependencies and global semantics of question text features and candidate answer text features: The cross-attention mechanism is used to align the differential features with the original features and use the original features as a reference to locate the dynamic event-sensitive areas in the differential sequence: Based on the question semantic feature F obtained in step 3 Q and the visual feature F in step 4 V and F D The combined information of ,generates dynamic fusion gating: Among them, Sigmoid is the activation function, W g To achieve learnable weights, the concatenated multimodal features are mapped into gated scalars; The contribution of the original and differential features is balanced by the gating coefficient to generate the final fused visual representation: F M =G⊙F D +(1-G)⊙F V Step 5: Decode multimodal features; The gated fusion visual feature F M and question text features F Q Splicing along the feature dimension to form a multimodal joint input: F cat =Concat(F M ,F Q ) Then, through the self-attention mechanism, visual and textual information are integrated, allowing the model to capture long-range dependencies within multimodal features: Finally, the fused features are mapped to the answer space: Step 6: Construct hierarchical contrastive learning; First, the original visual features F are constrained by unsupervised contrastive learning. V and differential feature F D The local consistency of forces the model to focus on the co-occurrence areas of dynamic events and static scenes in the video and suppress background noise: Where τ represents the temperature coefficient, which is used to control the fraction of contrast loss; B represents the number of batches; Secondly, through two-way contrastive learning, the semantic distance between the question semantics and static visual information and dynamically changing information is shortened, achieving text-visual cross-modal alignment: Finally, the alignment of supervised multimodal features and answer features is achieved through space-token two-layer contrastive learning: Among them, f i is the embedding of the fusion feature F within the batch, a i + is the embedding of the positive sample answer in the batch, a j - Embedding for negative sample answers within the batch; Step 7: Loss function and score function; The final loss function is expressed as the weighted sum of the above layer contrastive learning losses: When evaluating the classification accuracy of the TrafficVQA task, the model calculates the cosine similarity between the semantic-visual features and all candidate answers and represents this similarity value as a score assigned to each candidate answer; The score function is defined as: P(a|V,Q)=F·F A,K K=1,2,…,k Among them, F A,k is the answer feature of the kth candidate answer, and F is the multimodal fusion feature; Finally, the ratio of correctly identified true labels is calculated to determine the classification accuracy and overall model performance.

Citation Information

Cited By

  • Question reply method, apparatus and device, computer readable medium and program product

    CN121524396A

  • Remote sensing visual question and answer method based on large language model and multi-level attention mechanism

    CN121542456A

  • Video map frame accurate identification method and system

    CN121789107A

  • Scene text video question answering method and system based on selection and focusing mechanism

    CN122157282A