A video understanding enhancement method based on MoE multi-modal fusion

Abstract

This invention provides a video understanding enhancement method based on MoE multimodal fusion. The method includes: a processor standardizing video frames, extracting visual features using a CLIP visual encoder, and inputting these features into a single-layer Transformer encoder for long-term feature modeling; the processor converting audio data into text using a Whisper model, followed by BERT encoding to obtain audio semantic features; the processor adaptively selecting a fusion strategy to fuse visual and audio semantic features using a MoE multimodal fusion module composed of a routing network and multiple expert models; and the processor inputting the fused multimodal features into a LoRA-tuned LLama3 language encoder to achieve end-to-end video understanding output. The beneficial effects of this invention are: it enables dynamic semantic alignment of heterogeneous modalities, constructs a unified multi-task adaptation interface, and improves the understanding accuracy and robustness of long-term narrative videos.

Description

Technical Field

[0001] This invention relates to the field of cross-media generation technology, and in particular to a video understanding enhancement method based on MoE multimodal fusion. Background Technology

[0002] Mixture of Experts (MoE) is a common technique in Large Language Models (LLM) designed to improve model efficiency and accuracy. MoE replaces dense feedforward network (FFN) layers with sparse MoE layers and uses gating networks or routing to determine which token to send to which expert.

[0003] As a social narrative carrier spanning cross-cultural and historical dimensions, narrative videos present complex human behavioral patterns and causal logic through continuous visual-auditory signal flows. Their multi-layered semantic structures, including subject identity resolution, interaction relationship modeling, and temporal causal deduction, place high demands on video understanding technologies. While current mainstream video perception models such as SlowFast and TimeSformer have made progress in short-term action recognition and single-modal object detection, their design paradigms are limited to local feature extraction and atomic-level task optimization, making it difficult to meet the needs of long-term, multimodal, and structured semantic modeling in narrative videos.

[0004] To overcome the aforementioned limitations, existing video understanding technologies have attempted various approaches, including global attention mechanism sequence modeling, acoustic feature fusion, external knowledge injection, and visual-language mapping. However, three core problems still remain: Modal heterogeneity alignment is difficult. Videos contain heterogeneous modal features such as images, audio, and text, which have large spatial differences, making it difficult for existing methods to capture their deep semantic relationships. The models are not designed for single tasks such as subtitle generation and action recognition, and lack a unified architecture to handle diverse tasks such as character recognition, causal reasoning, and question answering. They also have weak cross-task generalization ability. Long-term dynamic context modeling is insufficient. Mainstream models such as 3D CNN are limited by local receptive fields or static templates, and cannot effectively model the global event chain and dynamic evolution process of long narrative videos.

[0005] The validation of the universality of large language models in natural language processing and multimodal scenarios provides a new direction for the development of video understanding technology. However, existing video understanding methods that combine large language models have not yet solved the problems of low modality fusion efficiency, poor modality adaptability of large language models, and insufficient modeling of long temporal features, and cannot achieve efficient end-to-end video understanding.

[0006] Therefore, there is an urgent need for a video understanding method that can achieve dynamic semantic alignment of heterogeneous modalities, construct a unified multi-task adaptation interface, and improve the understanding accuracy and robustness of long-term narrative videos. Summary of the Invention

[0007] To address the problems in existing technologies, this invention provides a video understanding enhancement method based on MoE multimodal fusion. It achieves long-term temporal modeling of video visual features through a CLIP visual encoder and a single-layer Transformer, extracts semantic features from audio data using Whisper and BERT models, adaptively fuses heterogeneous modal features through a MoE multimodal fusion module, and achieves end-to-end video understanding natural language output through a LoRA-tuned LLama3 language encoder. This method enables dynamic semantic alignment of heterogeneous modalities, constructs a unified multi-task adaptation interface, and improves the understanding accuracy and robustness of long-term narrative videos. It solves the problems of difficult modal heterogeneity alignment, insufficient task transfer capability, and inadequate long-term dynamic context modeling in existing technologies.

[0008] The present invention provides a video understanding enhancement method based on MoE multimodal fusion, comprising the following steps: Step 1: The processor performs spatial dimension standardization and pixel intensity normalization on the video frames, extracts the visual features of the video frames through the CLIP visual encoder, and inputs the visual features into a single-layer Transformer encoder to perform long-term inter-frame dependency modeling to obtain time-aware visual features. Step 2: The processor uses the Whisper model to process the audio data of the video into speech-to-text, obtaining a text sequence with timestamps. Then, the BERT model is used to encode the text sequence to obtain audio semantic features. Step 3: The processor inputs the time-aware visual features and audio semantic features into the MoE multimodal fusion module. The MoE multimodal fusion module adaptively selects an expert model based on the input multimodal data features through a routing network. The selected expert model is used to complete the multimodal feature fusion and obtain the fused multimodal features. Step 4: The processor inputs the fused multimodal features into the LLama3 language encoder, which is then adaptively fine-tuned by LoRA low-rank, to achieve natural language output for video understanding.

[0009] The present invention is further improved in that, in step 1, the visual features extracted by the CLIP visual encoder are represented as follows: , where N is the length of the feature sequence and D is the dimension parameter of the feature vector.

[0010] The present invention is further improved in that, in step 1, the single-layer Transformer encoder models the inter-frame spatiotemporal correlation through a self-attention mechanism. The calculation process of the self-attention mechanism includes using a learnable linear transformation matrix. , , Visual features Mapped to query matrices respectively Key matrix Sum matrix ,Right now ; Calculate the attention weight matrix using the dot product attention formula , ,in The scaling factor is used; the updated features are obtained by weighting the value matrix V with the weight matrix A. The updated features With original features After addition and layer normalization, the temporally perceptual visual features are obtained.

[0011] The present invention is further improved in that, in step 2, the Whisper model processes the audio data by extracting time-frequency features from the original audio waveform using a Mel spectrogram, as shown in the formula: ,in, For short-time Fourier transform, α is the scaling factor, t is the time step, and f is the Mel frequency channel. The time-frequency features are reduced in dimensionality by two convolutional layers and then input into the Transformer encoder. The decoder combines the language model and positional encoding to generate a text sequence with timestamps.

[0012] The present invention is further improved in that, in step 2, the BERT model uses a sine / cosine function for position encoding, and the position encoding formula is as follows: Where pos is the position index. For the embedding dimension; the BERT model encodes text sequences with positional encoding through a multi-layer Transformer encoder, outputting audio semantic features. ,in For the input embedding matrix, This is the semantic feature matrix.

[0013] In a further improvement to this invention, in step 3, the MoE multimodal fusion module includes a routing network and at least three expert models. The routing network outputs the selection weights of each expert model based on the semantic overlap, task type, and data complexity of the multimodal data, and selects the expert model with the highest weight to perform the feature fusion operation.

[0014] The present invention is further improved in that, in step 3, the expert model includes an element-wise multiplication fusion expert, a graph structure fusion expert, and a contrastive fusion expert. The element-wise multiplication fusion expert performs element-wise multiplication operations on visual features and audio semantic features to enhance the semantic overlap of multimodal features. The graph structure fusion expert transforms visual features and audio semantic features into a graph structure, using features as nodes and semantic associations within and between modalities as edges, and performs multi-hop information aggregation through the message passing mechanism of a graph neural network to obtain fused features. The contrastive fusion expert uses the InfoNCE contrastive loss function to map the multimodal features to a unified semantic space, using matching visual-audio feature pairs as positive samples and mismatched feature pairs as negative samples, thereby achieving feature fusion.

[0015] The present invention is further improved in that, in step 4, each layer of the LLama3 language encoder includes a self-attention mechanism and a feedforward network (FFN). l The output of the layer is ,in The LLaMA3 model encoder, fine-tuned by LoRA, uses the formula... Output natural language results, where For the first One generated word, These are the output layer weights.

[0016] The present invention is further improved in that, in step 1, the spatial dimension standardization process is to adjust all video frames to a uniform spatial resolution, and the pixel intensity normalization process is to map the pixel values ​​of the video frames to a preset numerical range.

[0017] In a further improvement, in step 3, the routing network of the MoE multimodal fusion module is a classification network based on a fully connected layer. The input of the routing network is a concatenated vector of visual features and audio semantic features, and the output is the probability distribution weights of each expert model.

[0018] The beneficial effects of this invention are as follows: This invention provides a video understanding enhancement method based on MoE multimodal fusion. It achieves long-term temporal modeling of video visual features through a CLIP visual encoder and a single-layer Transformer, extracts semantic features from audio data through Whisper and BERT models, adaptively completes heterogeneous modal feature fusion through the MoE multimodal fusion module, and achieves end-to-end video understanding natural language output through a LoRA-tuned LLama3 language encoder. This enables dynamic semantic alignment of heterogeneous modalities, constructs a unified multi-task adaptation interface, and improves the understanding accuracy and robustness of long-term narrative videos. By integrating multiple heterogeneous fusion strategies and combining them with an intelligent routing network, it can dynamically select the most suitable expert model for fusion calculation based on the input multimodal data features. This mechanism not only effectively improves the model's adaptability to modal heterogeneity but also avoids redundant computation caused by traditional fixed fusion methods, significantly improving inference efficiency. Simultaneously, the synergistic effect of multiple expert strategies enhances the model's robustness and generalization ability in different task scenarios, enabling it to more accurately capture the complex semantics between video and text. This approach effectively bridges the differences in multimodal feature spaces, enhancing the consistent joint representation of multi-source information in complex narrative scenarios and solving the inherent challenge of heterogeneous modal information fusion. It innovatively introduces a low-rank adaptive (LoRA)-based Large Language Model (LLM) fine-tuning mechanism, seamlessly integrating the LLM's prior knowledge base and complex reasoning capabilities into the video understanding framework. The LLM receives the structured multimodal semantic representation output from the front-end fusion layer and can be driven by natural language commands to dynamically adapt to downstream tasks in the open domain. This design overcomes the limitations of traditional single-task models, achieving zero-shot / few-shot transfer capabilities for multiple tasks within a single architecture, significantly improving system flexibility and deployment efficiency. A global temporal single-layer Transformer encoder processes visual feature sequences, with its global self-attention mechanism directly modeling long-range spatiotemporal dependencies between video frames, generating temporally aware structured visual sequences. This provides high-fidelity temporal context for back-end modules, significantly improving the accuracy and robustness of understanding complex dynamic processes in long narrative videos compared to traditional local operation models. This solves the problems of difficult modal heterogeneity alignment, insufficient task transfer capabilities, and inadequate long-temporal dynamic context modeling in existing technologies. Attached Figure Description

[0019] Figure 1 This is a flowchart of a video understanding enhancement method based on MoE multimodal fusion according to the present invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0021] Please see Figure 1The present invention provides a video understanding enhancement method based on MoE multimodal fusion, comprising the following steps: Step 1: The processor performs spatial dimension normalization and pixel intensity normalization on the video frames. It then extracts visual features from the video frames using the CLIP visual encoder. These visual features are input into a single-layer Transformer encoder for long-term inter-frame temporal dependency modeling, resulting in temporally-aware visual features. The visual features extracted by the CLIP visual encoder are represented as follows: Where N is the feature sequence length and D is the dimension parameter of the feature vector; the single-layer Transformer encoder models the inter-frame spatiotemporal correlation through a self-attention mechanism. The calculation process of the self-attention mechanism includes using a learnable linear transformation matrix. , , Visual features Mapped to query matrices respectively Key matrix Sum matrix ,Right now ; Calculate the attention weight matrix using the dot product attention formula , ,in The scaling factor is used; the updated features are obtained by weighting the value matrix V with the weight matrix A. The updated features With original features After addition and layer normalization, the temporally perceptible visual features are obtained; spatial dimension normalization is to adjust all video frames to a uniform spatial resolution, and pixel intensity normalization is to map the pixel values ​​of the video frames to a preset value range.

[0022] Step 2: The processor uses the Whisper model to perform speech-to-text conversion on the video's audio data, obtaining a timestamped text sequence. Then, the BERT model encodes the text sequence to obtain audio semantic features. The Whisper model's audio data processing includes extracting time-frequency features from the original audio waveform using Mel spectrograms, as shown in the formula... , For the Short Time Fourier Transform (SFT), α is the scaling factor, t is the time step, and f is the Mel frequency channel. The time-frequency features are dimensionality-reduced through two convolutional layers and then input into the Transformer encoder. The decoder combines the language model and positional encoding to generate a timestamped text sequence. The BERT model uses sine / cosine functions for positional encoding, and the positional encoding formula is... Where pos is the position index. For the embedding dimension; the BERT model encodes text sequences with positional encoding through a multi-layer Transformer encoder, outputting audio semantic features. ,in For the input embedding matrix, This is the semantic feature matrix.

[0023] Step 3: The processor inputs the time-aware visual features and audio semantic features into the MoE multimodal fusion module. The MoE multimodal fusion module adaptively selects an expert model based on the input multimodal data features through a routing network. The selected expert model completes the multimodal feature fusion to obtain the fused multimodal features. The MoE multimodal fusion module includes a routing network and at least three expert models. The routing network outputs the selection weights of each expert model based on the semantic overlap, task type, and data complexity of the multimodal data, and selects the expert model with the highest weight to perform the feature fusion operation. The expert models include element-wise multiplication fusion experts, graph structure fusion experts, and contrastive fusion experts. The element-wise multiplication fusion expert performs feature fusion on both visual features and audio semantic features. The feature model performs element-wise multiplication to enhance the semantic overlap of multimodal features. The graph structure fusion expert transforms visual and audio semantic features into a graph structure, using features as nodes and semantic associations within and between modalities as edges. Through the message passing mechanism of the graph neural network, multi-hop information aggregation is performed to obtain fused features. The contrastive fusion expert uses the InfoNCE contrastive loss function to map matching visual-audio feature pairs as positive samples and mismatched feature pairs as negative samples, thus mapping multimodal features to a unified semantic space and achieving feature fusion. The routing network of the MoE multimodal fusion module is a classification network based on fully connected layers. The input of the routing network is a concatenated vector of visual and audio semantic features, and the output is the probability distribution weights of each expert model.

[0024] Step 4: The processor inputs the fused multimodal features into the LLama3 language encoder, which is then adaptively fine-tuned using LoRA low-rank technology. The LLama3 language encoder then outputs natural language for video understanding. Each layer of the LLama3 language encoder includes a self-attention mechanism and a feedforward network (FFN). l The output of the layer is ,in The LLaMA3 model encoder, fine-tuned by LoRA, uses the formula... Output natural language results, where For the first One generated word, These are the output layer weights.

[0025] Please see Figure 1In step 1, the video frames are first standardized to a uniform spatial resolution and pixel intensity is normalized. Then, a pre-trained multimodal visual encoder is used to extract features from each frame. Taking a Transformer-based visual encoder as an example, it divides the input image into non-overlapping image patches of a preset size and maps each patch to a high-dimensional feature vector using a linear projection layer. Finally, a feature sequence containing temporal-spatial information is generated through a serialization operation; its length is determined by the total number of video frames and the spatial division ratio of the image patches. This feature representation can be formalized as: , Where N is the sequence length and D is the dimension parameter of the feature vector.

[0026] Extracted visual features The input is a single-layer Transformer encoder. This encoder captures the spatiotemporal correlations between frames through a self-attention mechanism. Specifically, the self-attention calculation process is as follows: Query, key, and value matrix generation: through learnable linear transformation matrices , , , eigenvalues Mapped to queries respectively ,key Sum , , Attention weight calculation: The weight matrix is ​​calculated using the dot product attention formula. , , in This is a scaling factor used to mitigate the gradient vanishing problem caused by excessively large dot product results.

[0027] Through the weight matrix value matrix Weighting yields the updated feature representation. , Updated features With original features The addition is performed, and the training process is stabilized through layer normalization (LayerNorm). , The output visual features after processing by a single-layer Transformer It already possesses spatiotemporal correlation and temporal alignment properties. At this point, the features can be further used for subsequent cross-modal fusion.

[0028] In step 2, the Whisper model directly models the temporal features of speech (such as Mel spectrograms) through the Transformer architecture. It can handle multilingual and multi-accent speech input and possesses strong noise robustness (such as background noise filtering). Its output text sequence not only preserves the original speech information but also achieves fine-grained speech-text alignment through timestamp annotation. Whisper first extracts time-frequency features from the original audio waveform using Mel spectrograms, with the following formula: , Among them, The function is a short-time Fourier transform, where α is the scaling factor, t is the time step, and f is the Mel frequency channel. Subsequently, the features are dimensionality-reduced through two convolutional layers and input into the Transformer encoder. The decoder combines the language model and positional encoding to progressively generate the text sequence.

[0029] The generated text is mapped to a vector space using BERT's embedding mechanism, where positional encoding employs sine / cosine functions: , pos is the position index. This is the embedding dimension. Subsequently, the text passes through a multi-layer Transformer encoder, capturing bidirectional contextual dependencies through a self-attention mechanism and a feedforward network (FFN), ultimately outputting a semantic vector representation: , in For the input embedding matrix, This is the semantic feature matrix.

[0030] In step 3, the hybrid expert model consists of a routing network and four expert models. The routing function adaptively selects the most suitable expert model based on the features of the input multimodal data, thereby achieving efficient modality fusion. Expert 1 enhances semantic overlap by performing element-wise multiplication of global features from video and text. This fusion method is simple and efficient, enhancing common semantics while suppressing irrelevant features. It is suitable for scenarios with high semantic overlap and clear task alignment.

[0031] Expert 2 transforms multimodal data features into graph structures, leveraging the advantages of graph structures in multimodal fusion. Graph structures can naturally fuse semantic information between different modalities, unifying visual features in videos and semantic information in text as nodes in the graph, and capturing potential relationships between them through edge construction. This approach can not only model the structured information within a modality, such as the temporal relationships between video frames and the linguistic structure between words in text, but also model the interactive relationships between modalities, such as the alignment between visual objects and corresponding descriptive words, thus achieving a more comprehensive semantic understanding. Graph neural networks, through message passing mechanisms, perform multi-hop information propagation and aggregation within the graph structure, enabling each node to absorb the semantic information of its neighboring nodes, thereby enhancing its semantic representation. This mechanism is particularly suitable for multimodal tasks requiring reasoning and contextual understanding. For example, in visual question answering tasks, when faced with questions like "who is doing what," the model needs to simultaneously understand objects, actions and their relationships in the image, as well as the semantic implications of the question. Graph structure fusion can organize this information into a unified semantic graph, enabling collaborative reasoning of multimodal information.

[0032] Expert 3 employs a contrastive fusion approach to process the input multimodal data. Its core lies in mapping videos and text into a unified semantic space through a contrastive learning mechanism, thereby enhancing their semantic consistency and alignment. This method utilizes paired positive samples (matched video-text pairs) and negative samples (mismatched video or text pairs), optimizing the model using contrastive loss functions such as InfoNCE. This makes matched samples closer in the semantic space, while distancing mismatched samples.

[0033] Contrast fusion eliminates the need for complex attention mechanisms, simplifying model design and improving computational efficiency. This method is particularly well-suited for zero-shot learning scenarios, demonstrating excellent generalization capabilities in tasks such as cross-modal retrieval and image-text matching. Furthermore, it effectively captures and strengthens subtle semantic connections between video and text, thereby enhancing the overall performance of multimodal tasks.

[0034] This MoE architecture integrates expert strategies with multiple non-attention mechanisms and dynamic routing mechanisms, enabling it to select the optimal fusion method based on different input content. It possesses excellent flexibility, efficiency, and task adaptability, making it suitable for various video-text fusion tasks.

[0035] In step 4, large-scale language models such as Meta's LLama3 represent the pinnacle of current text understanding and generation. Pre-trained on massive amounts of text corpora, it demonstrates exceptional capabilities in semantic parsing, logical reasoning, and fluent language generation. However, LLama3's fundamental design focuses on the text modality; its pre-training data consists almost entirely of text. This means that the model's internal parameters learn a purely joint distribution of linguistic symbols. When faced with multimodal data like video, which contains rich spatiotemporal dynamic visual information and is often accompanied by text or audio, the native LLama3 has a fundamental limitation: it lacks an intrinsic mechanism for establishing deep associations between video frame sequences and linguistic descriptions. The model has not been pre-trained on pairs of video segments and their corresponding text descriptions, making it difficult to spontaneously learn the complex and subtle joint probability distribution between visual features and linguistic concepts. This modal gap makes the unmodified LLama3 unsuitable for direct use in video content understanding or generating video-related natural language descriptions.

[0036] To overcome this challenge and fully leverage the powerful language capabilities of Llama3, Low-Rank Adaptation (LoRA) offers an extremely efficient and flexible solution. The core idea of ​​LoRA is not to completely retrain the massive original model parameters, as this full fine-tuning approach is computationally extremely expensive and can easily cause the model to "forget" the valuable general knowledge gained during pre-training for specific tasks. LoRA avoids the high cost of full parameter fine-tuning by introducing a low-rank matrix to incrementally update the original weights, while maintaining the model's generalization ability. Low-rank fine-tuning of LLama3 parameters significantly reduces computational and storage overhead while preserving the model's high-order semantic understanding capabilities. LoRA fine-tuning can dynamically adjust weights for specific tasks, improving task relevance. The fused multimodal features directly drive the LLama3 language generation process, eliminating the need for additional intermediate modules (such as classifiers), achieving end-to-end multimodal video understanding and generation.

[0037] The fused features F are input into the LLama3 language encoder. Each layer of LLama3 contains a self-attention mechanism and a feedforward network (FFN). (The text then repeats itself, so the translation stops.) l Taking a layer as an example, its output is: , in , In LLama3, LoRA introduces a low-rank matrix. For the original weights For incremental updates, the formula for updating the weights is: , Where α is the learning rate, Let be the loss function. The final weights are updated as follows: The LLaMA3 model, fine-tuned with LoRA, will fuse features. Mapped to natural language output, , in For the first One generated word, These are the output layer weights.

[0038] As can be seen from the above, the beneficial effects of this invention are as follows: This invention provides a video understanding enhancement method based on MoE multimodal fusion. It achieves long-term temporal modeling of video visual features through a CLIP visual encoder and a single-layer Transformer, extracts semantic features from audio data through Whisper and BERT models, adaptively completes heterogeneous modal feature fusion through the MoE multimodal fusion module, and achieves end-to-end video understanding natural language output through a LoRA-tuned LLama3 language encoder. This enables dynamic semantic alignment of heterogeneous modalities, constructs a unified multi-task adaptation interface, and improves the understanding accuracy and robustness of long-term narrative videos. By integrating multiple heterogeneous fusion strategies and combining them with an intelligent routing network, it can dynamically select the most suitable expert model for fusion calculation based on the input multimodal data features. This mechanism not only effectively improves the model's adaptability to modal heterogeneity but also avoids redundant computation caused by traditional fixed fusion methods, significantly improving inference efficiency. Simultaneously, the synergistic effect of multiple expert strategies enhances the model's robustness and generalization ability in different task scenarios, enabling it to more accurately capture the complexities between video and text. Semantic association effectively bridges the differences in multimodal feature spaces, enhancing the ability to consistently and jointly represent multi-source information in complex narrative scenarios and solving the inherent challenge of heterogeneous modal information fusion. It innovatively introduces a low-rank adaptive (LoRA)-based Large Language Model (LLM) fine-tuning mechanism, seamlessly integrating the LLM's prior knowledge base and complex reasoning capabilities into the video understanding framework. The LLM receives the structured multimodal semantic representation output from the front-end fusion layer and can be driven by natural language commands to dynamically adapt to downstream tasks in the open domain. This design breaks through the limitations of traditional single-task models, achieving [the desired result] within a single architecture. Multi-task zero-shot / few-shot transfer capability significantly improves system flexibility and deployment efficiency; a global temporal single-layer Transformer encoder is used to process visual feature sequences, and its global self-attention mechanism directly models the long-range spatiotemporal dependencies between video frames, generating temporally aware structured visual sequences, providing high-fidelity temporal context for backend modules. Compared with traditional local operation models, it significantly improves the accuracy and robustness of understanding complex dynamic processes in long narrative videos, and solves the problems of difficult modal heterogeneity alignment, insufficient task transfer capability, and insufficient long-temporal dynamic context modeling in existing technologies.

[0039] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A video understanding enhancement method based on MoE multimodal fusion, characterized in that, Includes the following steps: Step 1: The processor performs spatial dimension standardization and pixel intensity normalization on the video frames, extracts the visual features of the video frames through the CLIP visual encoder, and inputs the visual features into a single-layer Transformer encoder to perform long-term inter-frame dependency modeling to obtain time-aware visual features. Step 2: The processor uses the Whisper model to process the audio data of the video into speech-to-text, obtaining a text sequence with timestamps. Then, the BERT model is used to encode the text sequence to obtain audio semantic features. Step 3: The processor inputs the time-aware visual features and audio semantic features into the MoE multimodal fusion module. The MoE multimodal fusion module adaptively selects an expert model based on the input multimodal data features through a routing network. The selected expert model is used to complete the multimodal feature fusion and obtain the fused multimodal features. Step 4: The processor inputs the fused multimodal features into the LLama3 language encoder, which is then adaptively fine-tuned by LoRA low-rank, to achieve natural language output for video understanding.

2. The video understanding enhancement method based on MoE multimodal fusion as described in claim 1, characterized in that: In step 1, the visual features extracted by the CLIP visual encoder are represented as follows: , where N is the length of the feature sequence and D is the dimension parameter of the feature vector.

3. The video understanding enhancement method based on MoE multimodal fusion as described in claim 2, characterized in that: In step 1, the single-layer Transformer encoder models the inter-frame spatiotemporal correlation through a self-attention mechanism. The calculation process of the self-attention mechanism includes using a learnable linear transformation matrix. , , Visual features Mapped to query matrices respectively Key matrix Sum matrix ,Right now ; Calculate the attention weight matrix using the dot product attention formula , ,in The scaling factor is used; the updated features are obtained by weighting the value matrix V with the weight matrix A. The updated features With original features After addition and layer normalization, the temporally perceptual visual features are obtained.

4. The video understanding enhancement method based on MoE multimodal fusion as described in claim 3, characterized in that: In step 2, the Whisper model's process for processing audio data includes extracting time-frequency features from the original audio waveform using a Mel spectrogram, as shown in the formula: ,in, For short-time Fourier transform, α is the scaling factor, t is the time step, and f is the Mel frequency channel. The time-frequency features are reduced in dimensionality by two convolutional layers and then input into the Transformer encoder. The decoder combines the language model and positional encoding to generate a text sequence with timestamps.

5. The video understanding enhancement method based on MoE multimodal fusion as described in claim 4, characterized in that: In step 2, the BERT model uses a sine / cosine function for position encoding, and the position encoding formula is as follows: Where pos is the position index. For the embedding dimension; the BERT model encodes text sequences with positional encoding through a multi-layer Transformer encoder, outputting audio semantic features. ,in For the input embedding matrix, This is the semantic feature matrix.

6. The video understanding enhancement method based on MoE multimodal fusion as described in claim 5, characterized in that: In step 3, the MoE multimodal fusion module includes a routing network and at least three expert models. The routing network outputs the selection weights of each expert model based on the semantic overlap, task type, and data complexity of the multimodal data, and selects the expert model with the highest weight to perform the feature fusion operation.

7. The video understanding enhancement method based on MoE multimodal fusion as described in claim 6, characterized in that: In step 3, the expert model includes an element-wise multiplication fusion expert, a graph structure fusion expert, and a contrastive fusion expert. The element-wise multiplication fusion expert performs element-wise multiplication on visual features and audio semantic features to enhance the semantic overlap of multimodal features. The graph structure fusion expert transforms visual features and audio semantic features into a graph structure, using features as nodes and semantic associations within and between modalities as edges. It aggregates multi-hop information through the message passing mechanism of a graph neural network to obtain fused features. The contrastive fusion expert uses the InfoNCE contrastive loss function to map matching visual-audio feature pairs as positive samples and mismatched feature pairs as negative samples to map multimodal features to a unified semantic space, thereby achieving feature fusion.

8. The video understanding enhancement method based on MoE multimodal fusion as described in claim 7, characterized in that: In step 4, each layer of the LLama3 language encoder includes a self-attention mechanism and a feedforward network (FFN). l The output of the layer is ,in The LLaMA3 model encoder, fine-tuned by LoRA, uses the formula... Output natural language results, where For the first One generated word, These are the output layer weights.

9. The video understanding enhancement method based on MoE multimodal fusion as described in claim 8, characterized in that: In step 1, spatial dimension normalization is to adjust all video frames to a uniform spatial resolution, and pixel intensity normalization is to map the pixel values ​​of the video frames to a preset numerical range.

10. The video understanding enhancement method based on MoE multimodal fusion as described in claim 9, characterized in that: In step 3, the routing network of the MoE multimodal fusion module is a classification network based on a fully connected layer. The input of the routing network is a concatenated vector of visual features and audio semantic features, and the output is the probability distribution weights of each expert model.

Images