An apparatus for long video understanding based on diffusion model

Abstract

The application provides a device for long video understanding based on a diffusion model, comprising a diffusion model supervision module and a multi-level alignment module; the diffusion model supervision module is used for extracting fine-grained visual features of a pre-trained diffusion model as an intermediate supervision signal; the multi-level alignment module adjusts the visual features of the diffusion model and a visual encoder through adaptive weight adjustment, and the multi-level alignment module is used for enhancing the capturing capability of the diffusion model for spatiotemporal dynamics. The diffusion model extracts fine-grained visual features as an intermediate supervision signal, and the visual features of the diffusion model and the visual encoder are adjusted through adaptive weight adjustment, so that the purpose of improving the video understanding performance in a complex dynamic scene is achieved.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, and in particular relates to a device for long video understanding based on a diffusion model. Background Technology

[0002] Multimodal video understanding based on diffusion models is an important technology at the intersection of computer vision and natural language processing. Its core lies in combining diffusion models and multimodal large language models to achieve efficient understanding and analysis of fine-grained spatiotemporal dynamics in long video content. This technology has shown broad application potential in multiple fields such as video surveillance, video retrieval, video captioning generation, and cross-modal reasoning. However, existing multimodal video understanding methods face many challenges when processing long video sequences, especially in capturing fine-grained spatiotemporal relationships and complex dynamic scenes. Long video understanding not only requires models to capture fine-grained temporal dependencies in the video but also to maintain high accuracy under complex conditions such as lengthy video content, rapid target movement, and frequent scene changes. Furthermore, given the massive nature of video data, models must balance efficiency and scalability to meet the stringent requirements of real-time applications.

[0003] For long video understanding tasks, existing methods include CLIP-based methods and self-supervised backbone network-based methods. CLIP-based methods extract features through semantic understanding but lack spatial contextual information; while self-supervised backbone network-based methods can capture more visual cues, they are computationally expensive and do not fully utilize the potential of large-scale generative models. Furthermore, existing methods often struggle to capture fine-grained spatiotemporal relationships when processing long videos, and their computational cost is high.

[0004] However, in complex video environments used in real-world applications, these methods still face the following challenges:

[0005] 1. Lack of fine-grained spatiotemporal modeling: Existing methods often struggle to capture fine-grained spatiotemporal relationships when processing long videos, resulting in poor performance in complex dynamic scenes.

[0006] 2. High computational cost: Although the self-supervised backbone network-based method can capture more visual cues, it has a high computational cost and is difficult to adapt to the needs of long video processing.

[0007] 3. Underutilization of the potential of generative models: Existing methods do not fully utilize the potential of large-scale generative models (such as diffusion models), which limits their performance in complex dynamic scenarios. Summary of the Invention

[0008] To address the problems existing in the prior art, the present invention provides a device for long video understanding based on a diffusion model.

[0009] This disclosure provides an apparatus for long video understanding based on a diffusion model, including: a diffusion model supervision module and a multi-level alignment module;

[0010] The diffusion model supervision module is used to extract fine-grained visual features from the pre-trained diffusion model as intermediate supervision signals.

[0011] The multi-level alignment module adjusts the visual features of the alignment diffusion model and the visual encoder through adaptive weight adjustment. The multi-level alignment module is used to enhance the diffusion model's ability to capture spatiotemporal dynamics.

[0012] Optionally, the diffusion model supervision module utilizes the diffusion model to extract fine-grained representations of video frames and aligns the output features of the Q-Former using a knowledge distillation loss function; the diffusion model reconstructs the signal from noisy data through a denoising process, extracting visual features at different levels, including noise data. By using Gaussian distribution Add noise to clean data Generates in the following formula:

[0013] ;

[0014] in, This indicates the time step in the diffusion process. For noise dispatcher parameters, Noise representing actual cleanliness data.

[0015] Optionally, the multi-level alignment module dynamically adjusts the alignment weights of different levels through an adaptive weight prediction network to optimize the feature alignment effect; the adaptive weight prediction network calculates the alignment weights based on the output features of Q-Former and the representation of the diffusion model.

[0016] Optionally, the multi-level alignment module aligns different levels of features in the Q-Former using a multi-level alignment loss function. The formula for calculating the alignment loss function is as follows:

[0017] ;

[0018] in, Indicates the first Layer alignment loss, Indicates the first Layer alignment weights.

[0019] Optionally, the diffusion model supervision module enhances the visual representation through the diffusion encoder in the intermediate layer. The diffusion encoder improves the ability to capture fine-grained spatiotemporal relationships through feature alignment and knowledge distillation loss.

[0020] Optionally, the multi-level alignment module projects the output of Q-Former to the same dimensional space as the output of the diffusion model through projection mapping, and aligns the projected output of Q-Former with the output of the diffusion model through the knowledge distillation loss function.

[0021] Optionally, the knowledge distillation loss function is:

[0022] ;

[0023] in, For projection mapping, parameterization is performed using an MLP layer; For the output of Q-Former, This is the output of the diffusion model.

[0024] Optionally, the multi-level alignment module uses a weight prediction mechanism to dynamically predict the loss weights for each layer, specifically:

[0025] ;

[0026] in, For the loss weight of each layer, It is the sigmoid activation function. For the first The output of the Q-Former layer, This is a weighted prediction mechanism.

[0027] Optionally, the diffusion model utilizes DiT to extract visual representations from each frame of video.

[0028] Optionally, the diffusion model supervision module extracts fine-grained visual features through a pre-trained diffusion model, including: using a positional encoding mechanism to combine temporal dynamics with spatial features, specifically:

[0029] ;

[0030] in, For the features of the nth frame, For location encoding, P is the number of image blocks, and t is the time step.

[0031] This invention provides a device for long video understanding based on a diffusion model. It extracts fine-grained visual features from the diffusion model as intermediate supervision signals and aligns the visual features of the diffusion model and the visual encoder through adaptive weight adjustment, thereby improving video understanding performance in complex dynamic scenes. It can effectively capture fine-grained spatiotemporal dynamics in videos and reduce computational overhead through an adaptive multi-level alignment mechanism, thus improving the model's performance in long video understanding tasks. Attached Figure Description

[0032] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0033] Figure 1 A comparative schematic diagram of a diffusion-based device for long video understanding provided in this disclosure and the prior art;

[0034] Figure 2 A schematic diagram of the overall architecture of Diff-LMM provided in the embodiments of this disclosure;

[0035] Figure 3 This is a diagram illustrating the impact of the hyperparameter λ on the LVU dataset relation recognition task, as provided in the embodiments of this disclosure.

[0036] Figure 4 This is a diagram illustrating the impact of different alignment levels λ of Q-Former on the LVU dataset relation recognition task, provided in this embodiment of the disclosure.

[0037] Figure 5 This is a schematic diagram illustrating the visualization results of a diffusion-based device for long video understanding provided in this embodiment of the present disclosure on a long-term video recognition task on the LVU dataset. Detailed Implementation

[0038] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0039] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0040] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0041] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0042] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0043] This embodiment provides a multimodal video understanding framework based on a diffusion model. By introducing supervision and a multi-level alignment mechanism within the diffusion model, this framework significantly enhances the model's ability to capture fine-grained spatiotemporal dynamics in long videos, thereby improving video understanding performance in complex dynamic scenes. Diff-LMM utilizes a pre-trained diffusion model to extract fine-grained visual features and aligns the output features of Q-Former through knowledge distillation loss, effectively compensating for the shortcomings of existing methods in capturing fine-grained spatiotemporal relationships. Simultaneously, an adaptive multi-level alignment mechanism is designed to optimize feature alignment by dynamically adjusting the alignment weights at different levels, ensuring a balance between capturing global and local features. By combining supervision and a multi-level alignment mechanism with the diffusion model, Diff-LMM not only effectively captures fine-grained spatiotemporal dynamics in videos but also reduces computational overhead through adaptive weight adjustment, improving model performance in long video understanding tasks. Furthermore, Diff-LMM performs autoregressive decoding using a large language model to generate natural language responses, further enhancing the model's cross-modal understanding capabilities. Diff-LMM has achieved state-of-the-art performance on multiple long video understanding benchmark datasets, significantly improving the accuracy and efficiency of video understanding in complex dynamic scenes.

[0044] like Figure 1As shown, existing methods connect the visual encoder to the large language model decoder via a query transformer and are guided by a language generation loss. In contrast, the method proposed in this invention introduces a diffusion-based supervision mechanism, enabling Q-Former to capture more refined cues and enhance spatiotemporal dynamics.

[0045] This embodiment provides a device for long video understanding based on a diffusion model, including a diffusion model supervision module and a multi-level alignment module. The diffusion model supervision module extracts fine-grained visual features through a pre-trained diffusion model as intermediate supervision signals. The multi-level alignment module adopts a multi-level alignment mechanism, which adjusts the weights adaptively to align the feature representations of the diffusion model and the visual encoder, thereby enhancing the model's ability to capture spatiotemporal dynamics.

[0046] Existing methods connect the visual encoder to the large language model decoder via a query converter and are guided by a language generation loss. In contrast, the device proposed in this embodiment introduces a diffusion-based supervision mechanism, enabling Q-Former to capture more refined cues and enhance spatiotemporal dynamics.

[0047] Experiments demonstrate that the Diff-LMM disclosed in this embodiment achieves superior performance on multiple long-video understanding benchmarks. Ablation analysis further confirms that the representations extracted from the pre-trained diffusion model have significant advantages in fine-grained tasks within long-video scenes. These findings highlight the immense value of diffusion models in enhancing the capabilities of multimodal video models, particularly in applications within complex environments.

[0048] Video-LLaMA directly models temporal relationships by adding a query transformer, although this increases computational complexity. MA-LMM, on the other hand, significantly reduces GPU consumption by introducing memory modules and memory compression mechanisms.

[0049] The diffusion model is a deep generative model that approximates the distribution of a natural image by utilizing the final state of a Markov chain starting from a standard Gaussian distribution. Although the diffusion model is primarily used for generative tasks, its denoising process can also be used to learn low-level and high-level features of the input data.

[0050] This embodiment adopts an adaptive multi-level alignment strategy, which flexibly adjusts the alignment granularity to bridge the representation gap between different models and make full use of the representational capabilities of the diffusion model.

[0051] To address the challenges of modeling fine-grained spatiotemporal relationships in videos, such as Figure 2As shown, a Diff-LMM framework is proposed. This framework integrates a diffusion-based supervision mechanism into a large video language model. Unlike traditional methods that directly use CLIP and Q-Former for visual feature extraction, this embodiment uses a diffusion encoder in the intermediate layer. This approach allows the diffusion features to enhance the CLIP representation through feature alignment and knowledge distillation loss.

[0052] Given a video containing N frames, use a pre-trained diffusion model to extract features from each frame to form a sequence. ,in Let P represent the features of the nth frame, where P is the number of image blocks and C is the feature dimension of each block. To enhance the spatiotemporal representation, a positional encoding (PE) mechanism is introduced, combining temporal dynamics with spatial features.

[0053] ;

[0054] in, Visual features are organized into a hierarchical memory using a dynamic compression strategy. This strategy preserves important temporal features while reducing redundancy, ensuring rich information retention and computational efficiency.

[0055] To improve the dynamic modeling of long-term temporal information, this embodiment employs an advanced long-term memory mechanism, combining a visual memory bank and a query memory bank. The visual memory bank stores feature representations of all historical frames extracted by the frozen encoder, capturing long-term contextual information. At each time step t, this bank aggregates features and serves as key input in the cross-attention mechanism. The query memory bank captures temporal dynamics by dynamically accumulating the query vector at each time step.

[0056] This embodiment introduces a diffusion-based supervision method, which allows for more detailed dynamic spatiotemporal modeling. Specifically, this method aligns the fine-grained representation extracted by the diffusion model with the CLIP model, enabling CLIP to focus more on local information during training, thereby enhancing the model's ability to capture fine-grained spatiotemporal dynamics.

[0057] The diffusion model leverages large-scale image data and visual generation priors to extract detailed structural object cues. By fusing feedback from these models, the CLIP model's fine-grained feature extraction capability is significantly enhanced. These probabilistic models learn the data distribution p(x) and are trained by generating images from random Gaussian variables. To capture more complex visual concepts, the diffusion model extracts images from noisy data at different noise levels. The signal is reconstructed from the data. The loss function for this process is defined as follows:

[0058] ;

[0059] in, Noise representing actual pollution clean data, This represents the noise predicted by the denoising model.

[0060] Noise data By using Gaussian distribution Add noise to clean data Generated in, following the noise scheduler The definition is as follows:

[0061] ;

[0062] in, Indicates the time step in the diffusion process; a larger one indicates a larger time step. This corresponds to stronger noise.

[0063] This embodiment introduces a supervisory signal from a diffusion model, which has been shown to effectively improve CLIP's performance on fine-grained tasks. Specifically, visual representations are extracted from each frame of video using DiT, and these representations are evaluated as targets. This alignment method aims to enhance Q-Former's ability to model fine-grained spatiotemporal dynamics.

[0064] Let f be a pre-trained diffusion model, considering a certain video frame. The output of the diffusion model is The output of Q-Former is , where H and D represent the number of hidden units and the embedding dimension, respectively.

[0065] This embodiment introduces a trainable projection mapping. This mapping will convert the output of Q-Former The projected Q-Former output is aligned to the same dimensional space as the diffusion model output. Finally, a knowledge distillation loss function is used to align the projected Q-Former output with the diffusion model output. The specific form of the loss function is shown below:

[0066] ;

[0067] in, Parameterization is performed using an MLP layer.

[0068] To optimize alignment and balance different semantic levels, an adaptive multi-level alignment module is proposed. This module performs dense alignment on each block and uses a learnable mechanism to evaluate the alignment difficulty, thereby improving learning efficiency and accuracy. Specifically, it leverages feature representations from a subset of Q-Former and integrates the multi-level alignment loss through an adaptive weighting mechanism. Given a Q-Former with L levels, M levels are selected for representation alignment. Let... is a set of indices for the selected Q-Former layers, where is the output of the th layer. For each selected block, the alignment loss is:

[0069] ;

[0070] To achieve effective alignment between different layers, a weight prediction mechanism is introduced, which dynamically predicts the loss weights for each layer. This prediction is based on the Q-Former output and the diffusion alignment features. Specifically, the alignment loss for the th layer can be calculated as:

[0071] ;

[0072] where is the sigmoid activation function, used to normalize the weights. Then, the total representation alignment loss can be calculated as:

[0073] ;

[0074] In this embodiment, a frozen large language model is used for autoregressive decoding to convert it into a discrete text sequence. The training objective of the framework includes two main parts:

[0075] (1) The cross-entropy loss for video-to-language autoregressive generation;

[0076] (2) The knowledge distillation loss for feature representation alignment.

[0077] The cross-entropy loss is defined as follows:

[0078]

[0079] where represents the i-th word generated, , V is the video information. represents the th token in the true sequence, represents the prediction of the model given the video context V and the previous token o<i.

[0080] To avoid introducing adverse effects on the model performance by the supervision signal, it is crucial to balance the contributions of these two parts. For this purpose, a hyperparameter λ is introduced in the final loss, which is used to balance the relative importance of the two parts.

[0081] The final loss is a weighted combination of the two parts:

[0082] ;

[0083] in, It is a hyperparameter used to adjust the representation of alignment loss. This contributes to the training process, thus balancing their relative importance in the training.

[0084] This embodiment combines quantitative analysis with qualitative evaluation to assess the performance of the proposed method in long video understanding, fine-grained video understanding, and video question-answering tasks, thereby validating its effectiveness.

[0085] The experiments in this embodiment are based on the LVU (Long Video Understanding) dataset, which contains approximately 30,000 video clips from 3,000 movies, each ranging from 1 to 3 minutes in length. This dataset covers a variety of real-world scenarios and is widely used for benchmarking long video recognition tasks. The experiments evaluated different sub-tasks, including Director Style Prediction and Movie Genre Classification, fully leveraging the diversity and complexity of the dataset to validate the effectiveness of this embodiment. The experimental results further demonstrate the significant advantages of the proposed device in long-term video content modeling, particularly in capturing temporal dependencies and contextual information. Furthermore, a comparison with existing methods on this dataset reveals the limitations of short-term video models in long-term video tasks.

[0086] To verify the capabilities of the proposed framework in complex spatiotemporal dynamic modeling, this embodiment further evaluates fine-grained tasks on the LVU dataset, including relation recognition.

[0087] In this embodiment, the relationship recognition task is to label the relationship category between two people in a video, such as... Figure 5As shown, relationships include husband-wife, friend, and other friendships. Because this task relies on only a limited number of video frames, even human observers struggle to accurately distinguish these relationships due to the involvement of fine-grained information such as facial expressions and behavioral features. To evaluate the effectiveness of our framework in fine-grained video understanding of MLLMs, the experiments primarily focus on predicting relationships between husband-wife and friends, excluding the "boyfriend-girlfriend" category due to data limitations. This experimental setup helps analyze the model's improvement in spatiotemporal fine-grained modeling.

[0088] To further verify the effectiveness of the proposed device on multimodal tasks, this embodiment also evaluated the MSVD-QA dataset. This dataset is a standard open-ended video question-answering dataset containing short videos (10-15 seconds in length), covering various scenes and including a rich variety of video question-answering tasks. It is widely used to evaluate the understanding capabilities of multimodal question-answering systems. Experiments on the MSVD-QA dataset not only verified the effectiveness of this embodiment in short video understanding tasks but also compared the performance of the Transformer structure of this embodiment with existing state-of-the-art (SOTA) methods. Experimental results show that this embodiment outperforms existing methods in short video question-answering tasks, efficiently fusing visual and linguistic information, thereby significantly improving the accuracy and robustness of video question-answering tasks.

[0089] In terms of implementation details, this embodiment uses Vicuna-7B as the large language model. The model is trained for 20 epochs, with a learning rate of [value missing]. The batch size is 64. The optimizer uses AdamW, where the hyperparameters are set to... =0.9, =0.999, weight decay is set to 0.05. The input image size is adjusted to 224×224 pixels. Based on the REPA method, the output of the eighth layer of the diffusion model is selected as the supervision signal. On the LVD dataset, the hyperparameter λ is set to 1, while on the MSVD-QA dataset, λ is set to 0.0005. In the decoding stage, a beam search with a width of 5 is used. The frame length is set to 100, and the memory length is set to 20. In the Language-Video Understanding (LVU) task, the prompt word format is: "What is the..." <task>"of the movie?", where "task" represents the task category, such as relationship, speaking style, scene, director, genre, writer, and release year. Top-1 accuracy is used as the primary performance indicator for evaluation.

[0090] Table 1: Comparison with state-of-the-art methods on the LVU [Wu and Krahenbuhl, 2021] dataset

[0091]

[0092] Compared to previous state-of-the-art (SOTA) methods and ViS4mer's performance on the LVD benchmark, this embodiment achieves significant breakthroughs in both content understanding and metadata prediction tasks, with Top-1 accuracy improving by 1.8% on most tasks. As shown in Table 1, although MA-LMM improves performance on long video tasks by introducing advanced memory mechanisms, its CLIP training process tends towards global feature alignment, limiting its performance on fine-grained tasks. In contrast, this embodiment enhances the model's ability to model fine-grained spatiotemporal dynamics through diffusion supervision and feature alignment strategies, enabling Diff-LMM to achieve SOTA performance on long video understanding tasks.

[0093] This embodiment demonstrates a relationship recognition task on the LVD dataset, requiring the identification of relationships between characters within a 2.5-minute video. The challenge lies in its fine-grained nature, necessitating the inference of spatiotemporal features over a long period, such as body movements and facial expressions. Distinguishing between spousal and friend relationships is particularly challenging, especially between opposite-sex friends and spouses, where behavioral similarities can lead to misjudgments. Table 2 shows that the current state-of-the-art method, MA-LMM, achieves 76.7% accuracy in identifying friend relationships, but only 27.8% in identifying spousal relationships, further validating the difficulty of this task.

[0094] Table 2: Comparison with benchmark methods on refined relation subsets of LVU [Wu and Krahenbuhl, 2021]. Top-1 results are reported.

[0095]

[0096] To verify the advantages of the proposed architecture, this embodiment compares it with current multimodal video understanding methods on the MSVD-QA dataset to evaluate its robustness to spatiotemporal representation. Although Diff-LMM is not specifically designed for short video question answering tasks, experimental results on the MSVD-QA dataset show that the proposed method still achieves state-of-the-art performance (SOTA) (see Table 3). Despite the relatively small improvement on the MSVD-QA dataset, this indirectly indicates that fine-grained spatiotemporal modeling is more important for long video understanding tasks than for short video tasks. Furthermore, the proposed method further enhances this capability by introducing diffusion supervision.

[0097] Table 3: Comparison with state-of-the-art methods on the MSVD dataset [Xu et al., 2017]. Top-1 and top-2 accuracies are reported.

[0098]

[0099] In terms of ablation experiments, this embodiment analyzes the impact of different λ values ​​on the relation recognition task in the LVU dataset, such as... Figure 3 As shown. Experimental results show that setting λ to 1 achieves the best performance for adaptive multi-LA alignment, thereby promoting representation alignment between the diffusion model and the CLIP model and overcoming the limitation of CLIP focusing only on global alignment. However, as Figure 3 As shown, maintaining a balance between global and fine-grained features is crucial. When λ is too high, the model over-relies on diffuse representations, causing Q-Former to tend to capture too much fine-grained information during visual-text alignment, resulting in CLIP's inability to effectively learn global features and thus reducing model accuracy. Similarly, when λ is too low, the model's alignment ability is insufficient. Therefore, in the experiments of this study, λ was set to 1 to ensure CLIP's optimal performance in the visual-text alignment task.

[0100] This embodiment explores the impact of applying alignment strategies at different levels of the Q-Former. Experiments compared four different alignment methods: (1) alignment of all layers; (2) alignment of only a single layer (layer 13); (3) alignment of lower layers (layers 1-7); and (4) alignment of higher layers (layers 8-13). Experimental results show that, during training, aligning only higher layers yields the best performance. Figure 4 As shown in the diagram. Specifically, aligning higher layers improves Top-1 accuracy by 15.3% compared to aligning all layers, while aligning lower layers performs slightly worse than aligning higher layers. Analysis shows that when all layers are aligned with the diffusion representation, the hidden states of the Q-Former tend to be similar across all layers, thus suppressing the model's ability to model the temporal dynamics of video information. Therefore, this embodiment focuses on aligning higher layers to more effectively capture temporal information in the video.

[0101] To evaluate the effectiveness of the proposed adaptive loss mechanism, ablation experiments were conducted on the LVU dataset in this embodiment, aiming to assess the necessity of balancing the loss weights across different layers. In contrast, a uniform weight distribution strategy was implemented, where the loss weights are equal across all layers.

[0102] Experimental results show that the adaptive loss mechanism outperforms the uniform weight strategy on the relationship recognition task, improving accuracy by 11%. This improvement is mainly attributed to the differences in hidden states at different levels, where simply aligning the diffuse representations is insufficient to improve performance. In contrast, the adaptive loss mechanism can dynamically adjust the loss weights of each layer, thereby achieving more effective feature alignment.

[0103] This embodiment explores whether representations generated by diffusion models can be effectively aligned with recent video multimodal large language models (MLLMs) to address the limitations of models that neglect fine-grained information in capturing detailed spatiotemporal dynamics. To this end, an adaptive multi-level alignment mechanism is proposed to achieve robust and accurate feature alignment. This mechanism enables seamless integration of diffusion supervision, thereby improving the quality of learned representations. Experimental results show that the pre-trained diffusion model is a powerful tool, and the proposed alignment mechanism significantly improves the performance of MLLMs in complex dynamic scenes.

[0104] Q-Former is a neural network architecture specifically designed for cross-modal tasks such as visual-language modeling. DiT is responsible for recovering the original video data from noisy data. CLIP (Contrastive Language-Image Pre-Training) is a multimodal model that achieves cross-modal semantic alignment between images and text through contrastive learning.

[0105] The device for long video understanding based on a diffusion model disclosed in this embodiment has the following beneficial effects:

[0106] 1. Improved Accuracy of Long Video Understanding: By introducing supervised diffusion modeling and a multi-level alignment mechanism, Diff-LMM significantly enhances the model's ability to capture fine-grained spatiotemporal dynamics in long videos. The fine-grained visual features extracted by the diffusion model are aligned with the output features of the Q-Former, generating more discriminative cross-modal features. This alignment significantly improves the model's understanding of video content, especially in complex and multi-object scenarios. Dynamic compression strategies reduce redundant features while preserving key time-series information, further improving the model's accuracy.

[0107] 2. Enhanced Temporal Dependency Modeling Capabilities: Diff-LMM effectively captures fine-grained temporal dependencies in videos through Q-Former and a memory mechanism. Q-Former captures the temporal dependencies of video frames using learnable query vectors and explicitly focuses on historical visual information through a cross-attention mechanism, achieving efficient long-term context modeling. The dynamic compression strategy dynamically manages the memory by evaluating the similarity of features in adjacent frames, ensuring the model can better understand the temporal evolution of video content.

[0108] 3. Reduced Memory Overhead and Computational Burden: Through dynamic compression strategies and adaptive weight adjustment mechanisms, Diff-LMM reduces computational overhead and improves computational efficiency while maintaining high accuracy. The dynamic compression strategy calculates the similarity of features in adjacent frames, dynamically manages the memory bank, reduces redundant features, retains key information, and significantly reduces memory overhead. The adaptive weight adjustment mechanism dynamically adjusts alignment weights to optimize feature alignment, further reducing the computational burden.

[0109] 4. Enhanced Discrimination Capability in Complex Scenes: Diff-LMM significantly enhances the device's discrimination capability in complex scenes by combining the supervision of the diffusion model with a multi-level alignment mechanism. The fine-grained visual features extracted by the diffusion model are aligned with the output features of the Q-Former, enabling the device to better utilize spatial context information and distinguish objects or scenes with similar appearances, especially performing exceptionally well in complex backgrounds or multi-object scenes.

[0110] 5. Performance Improvement Validated by Experiments: Experimental results on multiple long video understanding benchmark datasets demonstrate that Diff-LMM achieves significant performance improvements across various tasks (such as relationship recognition and scene understanding), particularly outperforming existing MA-LMM models in long video sequences. For example, on the LVU dataset, Diff-LMM achieves significantly higher accuracy in relationship recognition tasks than existing methods, especially excelling in complex tasks such as identifying spousal relationships.

[0111] 6. Broad Application Prospects: Diff-LMM has demonstrated broad application potential in multiple fields such as video surveillance, video retrieval, video caption generation, and cross-modal reasoning. Its superior performance and efficient computing power make it of great value in practical applications, especially in tasks that require handling complex dynamic scenes and long video sequences.

[0112] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0113] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0114] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0115] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0116] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0117] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0118] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.< / task>

Claims

1. A device for long video understanding based on a diffusion model, characterized in that, include: Diffusion model supervision module and multi-level alignment module; The diffusion model supervision module is used to extract fine-grained visual features from the pre-trained diffusion model as intermediate supervision signals. The multi-level alignment module adjusts the visual features of the alignment diffusion model and the visual encoder through adaptive weights. The multi-level alignment module is used to enhance the diffusion model's ability to capture spatiotemporal dynamics. The diffusion model supervision module utilizes the diffusion model to extract fine-grained representations of video frames and aligns the output features of the Q-Former using a knowledge distillation loss function. The diffusion model reconstructs the signal from noisy data through a denoising process, extracting visual features at different levels, including those from the noisy data. By using Gaussian distribution Add noise to clean data Generates in the following formula: ； in, This indicates the time step in the diffusion process. For noise dispatcher parameters, Noise representing actual pollution cleanliness data; The multi-level alignment module dynamically adjusts the alignment weights at different levels through an adaptive weight prediction network to optimize the feature alignment effect; the adaptive weight prediction network calculates the alignment weights based on the output features of Q-Former and the representation of the diffusion model. The multi-level alignment module aligns different levels of features in the Q-Former using a multi-level alignment loss function. The formula for calculating the alignment loss function is as follows: ； in, Indicates the first Layer alignment loss, Indicates the first Layer alignment weights; The diffusion model uses DiT to extract visual representations from each frame of video. The diffusion model supervision module extracts fine-grained visual features through a pre-trained diffusion model, including: using a positional encoding mechanism to combine temporal dynamics with spatial features, specifically: ； in, For the features of the nth frame, For location encoding, P is the number of image blocks, and t is the time step.

2. The apparatus for long video understanding based on a diffusion model according to claim 1, characterized in that, The diffusion model supervision module enhances visual representation through the diffusion encoder in the intermediate layer. The diffusion encoder improves the ability to capture fine-grained spatiotemporal relationships through feature alignment and knowledge distillation loss.

3. The apparatus for long video understanding based on a diffusion model according to claim 2, characterized in that, The multi-level alignment module projects the output of Q-Former to the same dimensional space as the output of the diffusion model through projection mapping, and aligns the projected output of Q-Former with the output of the diffusion model through the knowledge distillation loss function.

4. The apparatus for long video understanding based on a diffusion model according to claim 3, characterized in that, The knowledge distillation loss function is: ； in, For projection mapping, parameterization is performed using an MLP layer; For the output of Q-Former, This is the output of the diffusion model.

5. The apparatus for long video understanding based on a diffusion model according to claim 4, characterized in that, The multi-level alignment module uses a weight prediction mechanism to dynamically predict the loss weight of each layer, specifically: ; in, For the loss weight of each layer, It is the sigmoid activation function. For the first The output of the Q-Former layer, This is a weighted prediction mechanism.

Images