Video understanding method based on multi-scene behavior analysis

Abstract

The invention discloses a triple scene graph video understanding method fused with a time sequence perception moving window, which comprises the following five core steps of: inputting a video clip, acquiring a frame sequence through an encoder, carrying out space division on each frame and adding time embedding; executing multi-head self-attention calculation of space-time offset enhancement in each sliding window, and integrating space structure modeling and time sequence processing; constructing a cross-space-time sliding window based on continuous frames to extract a token subset; compressing the window output into a fixed-length video token (triple scene graph) through a sliding query converter; and inputting the triple scene graph of the video content and a user question into the multi-modal large model to realize semantic analysis and content understanding. According to the method, the video understanding precision is remarkably improved while the space-time relation modeling capability is enhanced, the average recall rate of behavior recognition is improved by 10%, a high-precision video understanding benchmark covering 25 object classes and 26 relation classes is constructed, the labeling error rate is lower than 3%, and the shielding problem and the long tail deviation problem are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-modal learning, and particularly relates to a kind of triple scene graph video understanding method based on the principle of spatio-temporal relationship structured representation and multi-modal semantic collaborative optimization in video understanding based on the present application, which realizes multi-scene behavior analysis video understanding by fusing time sequence perception moving window. BACKGROUND

[0002] Current traditional multi-modal large models are relatively accurate in overall video understanding when processing multi-scene video content understanding tasks. However, videos contain multi-modal signals such as vision, audio, text (subtitle / OCR), and motion information. The fusion of large models often stops at shallow feature splicing, which cannot be deeply aligned. At the same time, due to the problem of not being precise enough in processing video features, the details are not satisfactory, which directly affects the accurate extraction and generation of deep semantic information of the video (especially content description). In contrast, these large models show more detailed and accurate capabilities when processing text information features.

[0003] In the field of video content understanding, especially in the core tasks of action classification and temporal action localization, existing technical solutions generally focus on the two basic levels of 'which action occurs' (WhichAction) and 'when does the action occur' (When). Although this focus mode can effectively identify the basic action categories in the video segment and label the time interval of their occurrence, it has significant limitations in deeply analyzing the complete semantic information contained in the video content. Specifically, current mainstream methods often ignore the identification and modeling of the action performer (subject, Subject / Agent) and the action object (object, Object / Patient) and their specific behavior relationships (Predicate / Relation). This systematic lack of subject-object and their interaction relationships directly leads to the generation of video content descriptions (such as automatic video captions or structured labels) based on existing technologies, which can summarize events and their time sequences but are severely deficient in describing entity interaction details, with semantic expression showing surface and fuzzy characteristics. The surface and detail lack of description information further brings key bottlenecks in practical applications. On the one hand, in scenarios with similar semantics but different subjects and objects (e.g., "doctor diagnoses patient" and "patient operates device"), or in actions that need to distinguish subtle interaction relationships (such as "pick up object" and "put down object"), existing technologies often struggle to make accurate distinctions, leading to ambiguous results and limiting the system's generalization ability and recognition accuracy. On the other hand, numerous valuable deep downstream application scenarios, such as fine-grained video retrieval based on semantics, behavior intent understanding, embodied intelligent interaction, complex abnormal behavior analysis in security monitoring, and operation step compliance checking in medical assistance, all highly depend on the understanding of the core semantic triplet structure of 'who did what to whom'. Therefore, the triplet structure of [subject (Subject), relationship (Predicate), object (Object)] is chosen as the basic unit to represent the core semantic relationship of video content, and by capturing relevant information such as scene changes and behavior interactions during the construction of the video scene graph, the ability to understand the details of multi-scene video content is improved. SUMMARY

[0004] Based on the principles of structured representation of spatio-temporal relationships and multi-modal semantic collaborative optimization in video understanding, a triplet scene graph video understanding method is proposed, which realizes multi-scene behavior analysis video understanding. This method extracts structured semantic triplets of subject-relation-object (S-P-O) from continuous video frames, constructs a large model-resolvable temporal scene graph, and drives multi-modal large models for cross-modal semantic alignment and relationship reasoning, achieving multi-scene behavior analysis video understanding.

[0005] The principle of the triplet scene graph video understanding method of the fusion time sequence perception moving window in the application includes the following:

[0006] (1) The application designs a time-aware moving window module Shifted Time-aware Block (STAB), aiming to construct an attention module that jointly models time and space, and improves the understanding and time positioning ability of the model for key events in long videos. The Shifted Time-aware Block is based on the Shifted Window Attention of the Swin Transformer, and adds a time embedding mechanism and a sliding Q-Former compression strategy. It can simultaneously perform spatial attention within each window to identify the geometric relationship between objects in a single frame, and time attention to associate adjacent frames. It can also achieve the following three modeling objectives: The following three modeling objectives are achieved:

[0007] Spatial structure modeling: In each sliding time window, window-level multi-head self-attention (W-MSA) is performed on local spatial patches to maintain spatial relationship modeling capability, and the calculation formula is as follows: Where Q, K, and V are linear transformations of tokens within the window, B space is a spatial relative position bias that ensures the model's robust modeling ability for spatial displacement invariance, and B time is a time position encoding bias that constrains the attention weight distribution of adjacent frames through time bias, ensuring smooth modeling of continuous actions.

[0008] Local temporal modeling: a cross-frame sliding window W k covers L w consecutive frames, and the attention at the lower layer uses a shifted window to cover different spatio-temporal regions. For the construction of Shifted Time-aware Attention, a sliding window is formed by selecting consecutive L w frames from the time dimension, and a subset of tokens is extracted across time and space: Where k is the starting time index of the window, i.e., the frame number; L w is the window length, representing the number of consecutive frames covered; and the window sliding step is S, i.e., the frame interval between adjacent windows.

[0009] Time series processing: explicit time cues or time embeddings TimeEmbed(t) are added to patch tokens, and a time bias Btime Enhance the time series processing capability in attention calculation.

[0010] Video content extraction is performed based on a time-aware moving window, and a loss function is constructed as follows: L total = L p + L o + L intra Wherein, L p is a GMM-based predicate classification loss, L o is an object classification cross-entropy loss, and L intra is a time sequence consistency comparison loss.

[0011] The structure takes into account local time sequence modeling and spatial perception ability, has good space-time expansion, and significantly improves the recognition and positioning ability of the model to key actions and events in the video.

[0012] (2) The application designs a multi-scene behavior analysis video understanding framework, including a video content extraction stage and a multi-modal large model integration and reasoning stage. In the video content extraction stage, a triple (subject-predicate-object) structured description is introduced. The video content extraction stage includes an object level entity extraction, a predicate relationship modeling and a time sequence triple fusion stage. The object level entity is detected and tracked, and the spatio-temporal features output by the STAB module are used as the input of the object level entity extraction. A lightweight detection head is used to generate an object candidate box: The detection boxes of adjacent frames are associated through IoU to construct an object track chain: In the predicate relationship modeling stage, the input is the interaction feature f s,o =[f s ;f o ;f s-o ] of the subject s and the object o in the same frame, and a Gaussian mixture model GMM is used to model the probability distribution of the set 26 relationship predicates: Wherein, π k is the mixing weight, μ k ,∑k is the Gaussian component parameter, and the most likely predicate is output. In the time sequence triple fusion stage, the input is a single frame triple Triplet t =(s t ,p t ,o t ), and a sliding window [t,t+L w-1] Cohesive triplets: At this time, the timing consistency loss is calculated, and the formula is: Where sim is the cosine similarity, and tau = 1 is the temperature coefficient. In the multi-modal large model integration and reasoning stage, first, the timing triplet sequence is converted into a structured natural language description: Where phi is a regularized conversion function that preserves the S-P-O structure. Then, the video token is projected into the large model embedding space: Where the linear layer W v In the training, fine-tuning is performed to align the visual-text semantic space. The triplet text and video token are spliced at the LLM input layer, and the position encoding is added to the video token to maintain the time order. On this basis, the overall video content understanding is completed.

[0013] (3) The present application provides an innovative video understanding benchmark construction method for objectively evaluating the accuracy of video content understanding. In view of the technical problem that the existing single-frame-based evaluation index Recall@K ignores the timing consistency of video content, the present application designs a complete video understanding benchmark construction process. The present application selects multiple public video data sets as the basic data source, including but not limited to Action Genome (AG), ActivityNet, Kinetics-400, EPIC-KITCHENS, etc. data sets, ensuring the diversity and representativeness of the data source. Among them, Action Genome is the largest video scene graph generation benchmark data set, ActivityNet covers a wide range of human activity scenes, Kinetics-400 provides large-scale action recognition data, and EPIC-KITCHENS focuses on first-person perspective daily activities, providing comprehensive support for structured scene understanding tasks through multi-source data fusion. On this basis, the present application establishes a unified semantic classification system containing 25 object classes and 26 relationship classes. Specifically, the present application innovatively uses a large language model (LLM) as a "judge large model" for intelligent preliminary screening, automatically identifying and selecting video segments rich in complex scene transitions from multiple video data sets. Through the multi-source data acquisition strategy, the scene limitations and labeling bias problems that may exist in a single data set are effectively avoided. Subsequently, professional annotators strictly annotate the selected video segments according to the pre-defined semantic classification system in a hierarchical and fine-grained manner, and correct the quality. The core technical requirements of the quality correction include: 1) ensuring that all annotation information reaches 100% accuracy; 2) the annotation content completely covers all category labels in the semantic classification system, including objects, relationships, actions and the complete mapping relationship of their subjects / objects; 3) through multiple rounds of annotation and expert review mechanism, consistency verification between annotators is established, and the Kappa coefficient is required to be above 0.8; 4) a unified annotation standard mechanism across data sets is established to ensure the annotation consistency of data from different sources. Through the above technical solutions, the application finally constructs a set of video understanding benchmark data sets covering multiple scene types, high precision and high quality, and provides a more comprehensive and reliable real annotation benchmark (GroundTruth) for video understanding algorithm, effectively solving the technical defects of the existing video understanding evaluation method in the aspects of time sequence consistency and scene diversity.

[0014] The triple scene graph video understanding method fusing the time sequence perception moving window in the application includes the following steps:

[0015] Step one: input a video segment, and obtain a video frame sequence through an encoder Each frame is spatially divided (Patch Embedding) using a pre-trained encoder, each frame is divided into N P patches with a dimension of Meanwhile, an absolute time prompt is added to each frame:

[0016] Step two: perform time-enhanced multi-head self-attention in each window:

[0017] Step three: select continuous L w frames from the time dimension to form a sliding window, and extract a token subset across time-space:

[0018] Step four: for the token sequence output by each window, a sliding Query Transformer (Q-Former) is used to generate a fixed-length video token, that is, a structured natural language description of a single-frame triple, still retaining the S-P-O structure: The final output is:

[0019] Step five: input the obtained video token and the user's question about the video content into the multi-modal large model, and map the video token to the large model embedding space through a linear layer: With the help of the prompt, the auxiliary large model understands the video content and realizes efficient inference.

[0020] In the above scheme, during the training process, the progressive memory calculation is used for the memory library, that is, the memory library of the a-th round of training is calculated using the model weight of the (a-1)-th round of training. By iteratively refining the memory prototype in this way, the representation quality of the data-poor class can be improved.

[0021] In the above scheme, the number of components K of the Gaussian mixture model GMM is set to 4, which is used to model the uncertainty of this classification. If K is too high, the calculation burden will increase and the performance improvement will be saturated.

[0022] In the above scheme, the gradient scaling factor λ plays a core role in dynamically balancing the strength of knowledge transfer in the model, where the gradient scaling factor is defined as: wherein, is the original predicate embedding, is the memory diffusion feature, and λ∈(0,1] is a learnable gradient scaling factor. When λ→0, emphasizes domination, that is, forced knowledge transfer; when λ→1, emphasizes domination, that is, preserving the original bias.

[0023] When backpropagation, λ differentiates the gradient of and to avoid compensation that only overwhelms the original feature space. The formula is defined as:

[0024] Further, the gradient scaling factor λ is set to 0.5 in the task. When λ is too high, the data-rich class will overfit; when λ is too low, the original features will be weakened too much, reducing the performance of the data-poor class. The optimal λ can balance the recall rate (R@K) and the average recall rate (mR@K).

[0025] The beneficial effects of the present application are:

[0026] This invention utilizes a time-aware, shifted-time-block (STAB) based moving window, inheriting the local window attention advantage of the Swing Transformer. It accurately captures the geometric relationships between objects within a video frame, and reduces background interference by focusing on local areas through self-attention within the window. Furthermore, it achieves behavior continuity modeling through cross-frame sliding windows, compensating for detection errors in single frames and solving the problem of occlusion of the same object in consecutive frames. This can improve the behavior recognition mR@K by 10%.

[0027] The video understanding method based on multi-scene behavior analysis designed in this invention can leverage the significant advantages of multimodal large models in text feature processing. By capturing relevant information such as scene changes and behavioral interactions during the construction of video scene graphs, it can improve the ability to understand the details of multi-scene video content.

[0028] The video understanding benchmark designed in this invention uses a collaborative paradigm of initial screening of video content through a large referee model followed by precise manual correction. This approach achieves 100% coverage of video content in terms of name and predicate categories, avoiding long-tail bias. At the same time, it suppresses the influence of noise, achieving an annotation error rate of less than 3%, and improving the reliability of experimental comparison indicators (such as mR@K). Attached Figure Description

[0029] Figure 1 A flowchart for video understanding based on multi-scenario behavior analysis

[0030] Figure 2 Design diagram of time-aware mobile window module architecture

[0031] Figure 3 A diagram illustrating the benchmark construction process for video understanding. Detailed Implementation

[0032] Example 1: Time-Aware Moving Window Module Taking a specific video clip as an example, the video parameters are: total number of frames T = 128 (duration 8 seconds), resolution H × W = 224 × 224, and sliding window length L. w =8, sliding step size S=4. Step 1: Spatial Structure Modeling. First, input processing is performed on window W. k (For example, k=1, covering frames 1-8) Each frame of the image is processed in blocks: Each frame is divided into 14×14 image blocks (N P =196), each block is 16×16 pixels. Perform multi-head self-attention within the window: For spatial relative position bias, create spatial displacement encoding matrix Generate bias through learnable parameters: B space (i,j) = Linear(M(x i -x j ,y i -y j )) Step two: local temporal modeling. First, build a cross-frame sliding window, select 8 consecutive frames to form a window from the time dimension: In the shiftedwindow mechanism, for the standard window W1, divide each frame into 4x4 local windows, 7x7 image blocks; for the offset window W2, offset the window position to the lower right corner by 50%, covering a new spatiotemporal region. Step three: time series processing. First, add absolute time encoding to each image block to achieve time embedding mechanism: Then, build the time bias. First, generate the time difference matrix ΔT ij = |t i -t j | Then map the time bias through the MLP network:

[0033] Example 2: Video understanding based on multi-scene behavior analysis Step one: video content extraction stage. Receive the spatiotemporal features output by the STAB module There are 31 windows, each containing 32 tokens. Use a lightweight detection head to generate candidate boxes for each frame: Associate adjacent frame detection boxes through IoU to build a track chain: Then, for the interaction features f s,o = [f s ; f o ; f s-o ] of the subject s and the object o in the same frame, use a Gaussian mixture model GMM to model the probability distribution of the set of 26 relationship predicates: Get the most likely predicate. Then aggregate the triplets in the sliding window [t, t+7] and calculate the temporal consistency loss L intra (Temperature coefficient τ = 1): Step two: multimodal large model integration and reasoning phase. Convert the triple sequence into natural language state: Text triplet = φ({(s t , p t , o t}) ) = "train driving on railroad..." Then project the video token T video to the LLM embedding space: After concatenating the input, the video content reasoning is performed according to the prompt guidance.

[0034] Example 3: Video understanding benchmark construction Step one: multi-source video data collection. Collect original videos from public datasets such as Action Genome, ActivityNet, Kinetics-400, EPIC-KITCHENS, etc., to ensure scene diversity; set screening criteria: duration 5-30 seconds, resolution no less than 720p, complex scene transition defined as containing at least 3 object entities and their relationship changes; use an automatic screening algorithm based on content complexity evaluation to identify video segments rich in multi-object interaction and spatio-temporal relationship evolution. Step two: multi-model cross-screening. Select multiple different large language models as "judge large model group" for parallel reasoning, and the comprehensive output is used to establish preliminary content annotation for the video; including 25 object categories and 26 relationship categories. Step three: dataset construction and verification. Use stratified sampling method to divide into 60% training set, 20% verification set, and 20% test set, to ensure uniform distribution of scene and relationship in each subset; establish a multi-dimensional quality evaluation system, including annotation integrity evaluation, consistency verification, and bias analysis; implement class balance check to ensure similar proportions of object classes and relationship classes in the subsets. Step four: benchmark performance verification. Select CNN-RNN hybrid architecture, Transformer architecture, and multi-modal fusion model as baseline models; design comprehensive evaluation indicators: object recognition accuracy, relationship classification F1 score, temporal reasoning accuracy, and content similarity.

[0035] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for understanding scene graph videos using triples with time-aware moving windows includes the following steps: Step 1: Input a video clip and obtain the video frame sequence through an encoder. Each frame of the image is encoded using pre-training, and each frame is spatially partitioned (Patch Embedding) into N parts. P There are 1 patch, with a dimension of 1 At the same time, add an absolute time hint to each frame: Step 2: Perform time-enhanced multi-head self-attention in each window: Step 3: Select continuous L from the time dimension w Frames form a sliding window to extract a subset of tokens across time and space: Step 4: For the token sequence output by each window, a fixed-length video token is generated using a sliding Query Transformer (Q-Former), which is a structured natural language description of a single-frame triple, while still retaining the SPO structure. The final output is: Step 5: Input the obtained video token and the user's question about the video content into the multimodal large model, and map the video token to the large model's embedding space through a linear layer: With the help of prompts, large models can be assisted in understanding video content and achieving efficient inference.

2. The method for understanding video using triplet scene graphs with fused temporally aware moving windows according to claim 1, characterized in that: The aforementioned time bias mechanism introduces B into attention calculation. time This constrains the attention weight allocation between adjacent frames, ensuring smooth modeling of continuous actions.

3. The method for understanding video using triplet scene graphs with fused temporally aware moving windows according to claim 1, characterized in that: The loss function design employs a ternary loss combination L... total =L p +L o +L intra , where L p This refers to GMM-based predicate classification loss, L o It is the cross-entropy loss for object classification, L intra It is a time-series consistency comparison loss.

4. The method for understanding video using triplet scene graphs with fused time-aware moving windows according to claim 1, characterized in that: The gradient scaling factor (λ = 0.5) is determined by dynamically balancing the primitive predicate embeddings. With memory diffusion characteristics Optimize the intensity of knowledge transfer: Differentiated scaling of gradients during backpropagation avoids feature overload.

5. The method for understanding video using triplet scene graphs with fused temporally aware moving windows according to claim 1, characterized in that: Progressive in-memory computation refers to the process in which the memory library in round a uses the model weights from round a-1 to iteratively optimize the representation quality of data-poor classes.

6. Video understanding method based on multi-scene behavior analysis: Video content extraction stage: Introducing a triple (Subject-Predicate-Object) structured description. The video content extraction stage includes object-level entity extraction, predicate relation modeling, and temporal triple fusion to obtain a triple structured description. Multimodal large model integration and inference stage: First, the temporal triple sequence is transformed into a structured natural language description. Then, the video token is projected into the large model embedding space. At the LLM input layer, the triple text and video token are concatenated. Positional encoding is used to add learnable temporal positional codes to the video token to maintain the temporal order. Based on this, the overall video content understanding is completed.

7. The video understanding method based on multi-scene behavior analysis as described in claim 6, characterized in that... The video content is described by introducing a triplet structure, and the video content is obtained by integrating the triplet description with the video in a multimodal large model.

8. A method for constructing a video understanding benchmark, characterized in that... include: Intelligent Initial Screening: This innovative approach employs a large-scale language model (LLM) as the "judgment model" for intelligent initial screening, automatically identifying and selecting video clips rich in complex scene transitions from multiple video datasets. Through a multi-source data acquisition strategy, it effectively avoids the scene limitations and annotation biases that may exist with a single dataset. Manual correction: Fine-grained annotation based on a predefined semantic system (25 object classes + 26 relation classes) to ensure 100% coverage of all category labels, while the annotation error rate is less than 3%.

9. The video understanding benchmark construction method as described in claim 8, characterized in that... Complete annotation means mandatory coverage of all objects, relationships, actions, and subject / object labels to eliminate long-tail data bias. Through multiple rounds of annotation and expert review, consistency verification among annotators is established, and the Kappa coefficient is required to reach 0.8 or higher.

Images