Gating-based streaming multimodal fusion audio-visual source separation method

Abstract

The application discloses a kind of based on the gating of streaming multimodal audiovisual source separation method.The method uses two-stage feature fusion strategy, gradually integrates audio, visual and motion modal information, dynamically adjusts the contribution of each mode in the fusion process.The method comprises: extracting the spectral feature of audio, extracting visual and motion features by target detection and video coding network;Introduce two-stage gating attention module at the position of the encoder bottleneck, generate the first stage and the second stage fusion weight, realize cross-modal dynamic feature integration;The fused multimodal feature is input into the audio-video Transform module, to generate audio mask, for restoring target sound source signal.The method of the application is suitable for audio-video source separation task in complex multi-source scene, with the advantages of high fusion precision, strong real-time, separation effect is excellent.

Description

Technical Field

[0001] This invention relates to the fields of audio and video signal processing and multimodal learning, and in particular to a multimodal feature fusion method based on a dynamic gating mechanism. Specifically, it is a streaming multimodal fusion method for audio-visual source separation, which is suitable for multi-source audio signal separation tasks in complex environments. Background Technology

[0002] With the continuous development of multimedia applications, how to accurately separate individual sound sources from complex and mixed audio has become a hot research topic. Traditional audio source separation methods mainly rely on single-modal techniques such as Independent Component Analysis (ICA) or Non-negative Matrix Factorization (NMF), which have limited effectiveness when dealing with real-world scenarios containing multiple sound sources and interference.

[0003] In recent years, thanks to advancements in deep learning, scholars have begun to incorporate visual information from videos to assist in audio modeling, forming an emerging research direction: "audiovisual source separation." Most existing works employ simple connections or element-wise multiplications between audio and visual features as fusion methods, but these fusion strategies struggle to model the dynamic dependencies between modalities. When introducing new modalities (such as motion), traditional single-stage fusion strategies exhibit significant limitations in cross-modal information collaborative modeling, impacting separation performance and generalization ability. Summary of the Invention

[0004] This invention provides a gated streaming multimodal fusion audio-visual source separation method, aiming to improve the flexibility and separation accuracy of multimodal feature fusion. The method introduces a streaming multimodal modeling mechanism that fuses visual, audio, and motion information. Through a gating mechanism, the fusion weights of each modality are dynamically adjusted at different stages to achieve more accurate feature integration. The fused multimodal representation is used to guide the generation of an audio mask, thereby separating the target sound source signal in the mixed audio.

[0005] This invention's method is built upon an end-to-end deep network framework, featuring a clear overall structure and strong modular composability, making it suitable for multi-source audio separation tasks in complex environments. By introducing a two-stage fusion strategy, a multi-scale visual attention module, and an audiovisual Transformer structure, this invention significantly improves intermodal collaborative modeling capabilities and separation performance, demonstrating broad application prospects in real-world multimodal perception systems. Attached Figure Description

[0006] Figure 1 System architecture diagram

[0007] The workflow of the audiovisual feature extraction network, the multimodal fusion network, and the audiovisual Transformer network is demonstrated.

[0008] Figure 2Streaming Multimodal Fusion Module (SMFM)

[0009] The detailed structure and feature fusion process of the streaming multimodal fusion module are shown.

[0010] Figure 3 Visual attention module

[0011] The detailed structure of the visual attention module is shown. Detailed Implementation

[0012] 1. Audiovisual Feature Extraction

[0013] The method of this invention first receives a set of inputs containing video frames and mixed audio. A pre-trained object detector (such as Faster R-CNN) is used to detect objects in each frame, extracting object-level visual features. Simultaneously, a video encoder (such as I3D) is used to extract motion features from the frame sequence. A short-time Fourier transform (STFT) is applied to the mixed audio signal to generate a spectrogram, and intermediate audio features, as well as the final audio features, are extracted using a U-Net structure.

[0014] 2. Streaming Multimodal Fusion Module

[0015] The first-stage fusion module performs channel-by-channel multiplication on the object features and intermediate audio features, then inputs the results into the multi-head cross-attention module to generate first-stage gating weights. These weights are used to adjust the fusion ratio of the two features, and the first-stage fusion result is output. The second-stage fusion module inputs the above fusion result along with motion features into the cross-attention mechanism to generate second-stage gating weights, achieving the final fusion of audio-visual-motion modalities.

[0016] 3. Visual Attention Module

[0017] A visual attention module was incorporated into the visual feature extraction process to enhance the expressive power of visual features through a multi-scale attention mechanism. Through this module, the model can better capture key components within visual features, further improving the separation performance.

[0018] 4. Audiovisual Transformer Decoder

[0019] The fused features are input into the Transformer module, which combines them with the audio query vector of the instrument category to generate an audio mask embedding for separation. The mask is generated by an MLP and multiplied point-by-point with the audio feature map to obtain a spectral-level mask map.

[0020] 5. Audio signal separation

[0021] By applying the inverse short-time Fourier transform to the separated audio spectrogram, the final separated audio signal is obtained. This signal contains features related to the target audio and removes background noise and other sound sources.

Claims

1. A gated streaming multimodal fusion audiovisual source separation method, comprising: Receive a set of video frames and the corresponding mixed audio signal; Use a pre-trained object detector to extract object features from video frames; Use a pre-trained video encoder to extract motion features from video frames; Use the U-Net structure to extract intermediate audio features from mixed audio; Multimodal feature fusion is performed through a two-stage dynamic gating fusion module. In the first stage, the fusion ratio of object features and audio features is dynamically adjusted according to the gating mechanism. In the second stage, the output of the first stage is further fused with motion features. The fused features are then input into the audio-visual Transformer module to model cross-modal interactions. Multiple audio mask embeddings are generated based on a query of a set of instrument categories. Mask prediction is performed on the mixed audio spectrogram using the generated mask embedding; Finally, the separated audio signal is restored by inverse short-time Fourier transform.

2. The end-to-end audio and video source separation method based on two-stage dynamic gating fusion according to claim 1, wherein the first-stage fusion generates initial fusion features by multiplying the object features and audio features through channel multiplication, and calculates the first-stage gating weights through a multi-head cross-attention mechanism to output the fusion result in a weighted manner.

3. The end-to-end audio and video source separation method based on two-stage dynamic gating fusion according to claim 1, wherein the second-stage fusion inputs the first-stage output and motion features into a multi-head cross-attention module to generate second-stage gating weights and outputs the final fused multimodal feature representation.

Images