Audio-visual segmentation method based on time sequence cross-modal design Mama

Abstract

The invention discloses an audio-visual segmentation method based on time sequence cross-modal design Mama, and the method comprises the steps: carrying out the feature extraction of an input video and audio sequence through a visual and audio encoder, and obtaining multi-scale visual and audio features; wherein the visual features of the second level to the fourth level are processed by a multi-scale time sequence encoder to obtain enhanced visual features; the enhanced visual features and the original audio features are sent to a modal aggregation decoder, and intra-frame and inter-frame visual-audio cross-modal fusion is performed through a visual-audio selective scanning module to obtain a fused audio query; sending the highest resolution feature map of the first level, the enhanced visual features and the original audio features into a context fusion pyramid, and performing cross-frame feature enhancement and up-sampling through a time sequence Mamba module and an audio-visual selective scanning module to generate optimized mask features; and the final segmentation mask is obtained by matrix multiplication of the mask features and the fused audio query.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal image and video processing, and relates to an audio-visual segmentation method based on a temporal cross-modal design Mamba, and in particular to an audio-visual segmentation method of a selective state space model combining temporal and cross-modal designs. Background Art

[0002] Human perception of the external world is inseparable from the assistance of visual and audio modalities. With the continuous development of deep learning technology, more and more audio-visual learning tasks such as audio-visual event localization, audio-visual semantic enhancement, and sound source localization have been continuously proposed to enhance the machine's understanding and perception of the two audio-visual modalities. However, the above methods are limited to the alignment and fusion of the two modalities at the image and region levels, and lack pixel-level understanding of sound signals. In this context, Zhou et al. proposed the audio-visual segmentation task in "Audio-visual segmentation", which aims to achieve pixel-by-pixel segmentation of sound objects in the video through audio guidance.

[0003] Previous audio-visual segmentation methods are mainly divided into methods based on convolutional neural networks (CNNs) and methods based on transformers. CNN-based methods are limited by their local receptive fields and cannot extract global information from two-dimensional image features and perform sufficient audio-visual modal interaction. To solve this problem, Gao et al. proposed AVSegFormer in "Avsegformer: Audiovisual segmentation with transformer", which uses audio features as queries to decode multi-scale visual features, thereby enhancing the perception of the sound area. Huang et al. proposed AQFormer in "Discovering sounding objects by audio queries for audio visual segmentation", which uses queries to correlate visual features across frames, thereby promoting the joint extraction of local and global information. Li et al. proposed CATR in "Catr: Combinatorial dependence audio-queried transformer for audio-visual video segmentation", which combines multiple modules to promote bidirectional feature fusion and greatly improves the audio-visual segmentation effect. However, the above methods are all limited by the quadratic complexity of Transformer and cannot effectively process multi-scale long video sequences. In addition, previous methods ignored the strong correlation between adjacent frames when dealing with cross-frame interactions, resulting in insufficient utilization of the spatiotemporal consistency in multimodal features.

[0004] Recently, Gu and Dao proposed the selective state space model (Mamba) in "Mamba: Linear-time sequence modeling with selective state spaces", which has linear complexity and its output is dynamically related to the input, which provides a new idea for solving the long context modeling problem in the visual base model. On this basis, Liu et al. proposed VMamba in "Vmamba: Visual state space model" and adopted a two-dimensional selective scanning strategy (SS2D) to alleviate its direction sensitivity. At present, a series of works have made specific improvements to Mamba and applied it to various downstream tasks, such as medical image segmentation, infrared small target detection, remote sensing image segmentation, etc. However, very few works have explored the capabilities of Mamba in multimodal learning, especially in the field of audio-visual segmentation.

[0005] Therefore, in response to the challenges in the audio-visual segmentation task, the present invention reasonably improves the Mamba structure and designs the temporal Mamba and cross-modal Mamba structures, which effectively improves the model's ability to extract and perceive sound-making objects. Summary of the invention

[0006] This paper proposes an audiovisual segmentation network that promotes the cross-modal perception ability of the model by introducing temporal and cross-modal designs into the Selective State Space Model (Mamba).

[0007] Technical solution of the present invention

[0008] A method for audio-visual segmentation based on temporal cross-modal design Mamba integrates cross-modal interaction and temporal design into the temporal Mamba model to achieve efficient and accurate audio-visual segmentation. First, the visual encoder and audio encoder are used to encode the input video sequence. and audio sequence Perform feature extraction to obtain multi-scale visual features and audio features ;in, Indicates the number of frames in the sequence, Represents different scales; among them, the visual features of the 2nd to 4th levels Enhanced visual features obtained by processing with a multi-scale temporal encoder ; Enhanced visual features and original audio features It is sent to the modality aggregation decoder, and the visual-audio selective scanning module performs intra-frame and inter-frame visual-audio cross-modal fusion to obtain the fused audio query. ; After that, the highest resolution feature map of the first level , enhanced visual features And the original audio features It is fed into the context fusion pyramid, where the temporal Mamba module and the audio-visual selective scanning module perform cross-frame feature enhancement and upsampling to generate optimized mask features. ; The final segmentation mask is composed of the mask features And the fused audio query Obtained through matrix multiplication.

[0009] Furthermore, the specific steps of integrating cross-modal interaction and timing design into the timing Mamba model to achieve efficient and accurate audio-visual segmentation are as follows:

[0010] Step 1: Extraction of visual and audio features

[0011] Given an input video sequence and audio sequence First, feature extraction is performed through the visual encoder and audio encoder to obtain multi-scale visual features. and audio features ;

[0012] Step 2: Multi-scale temporal encoder

[0013] The temporal Mamba module is designed, and a multi-scale temporal encoder is proposed based on it. The VSS module and the temporal Mamba module proposed in VMamba are used to encode the multi-scale visual features of the last three layers. Perform feature encoding and interaction within and between frames;

[0014] (1.1) VSS module

[0015] For multi-scale feature maps , ,in represents the feature channel dimension, and Respectively represent the scale The height and width of the feature map; first, expand and concatenate the feature maps of all scales in the order of height first, width second, and width first, height second, to obtain the concatenated feature sequence ,in, Represents the length of the concatenated feature sequence; the concatenated feature sequence is flipped in the length dimension and concatenated with the concatenated feature sequence to obtain , the specific formula is as follows:

[0016]

[0017] in, Represents a splicing operation, Indicates a flip operation;

[0018] Will It is sent into the state space model for causal sequence scanning. The specific formula is as follows:

[0019]

[0020]

[0021] in, Indicates The sequence is characterized by a state space model; Represents the modeled Reshape the features of the sequence; Represents the intermediate features Separate and reshape back to the original multi-scale visual features;

[0022] (1.2) Timing Mamba module

[0023] The multi-scale visual features processed by the VSS module The images are sent to the temporal Mamba module, which first extracts local features of the input multi-scale visual features through 3D deep convolution, and then uses 3D selective scanning to enhance the inter-frame interaction of the multi-scale visual features.

[0024] Step 3: Modality Aggregation Decoder

[0025] The modality aggregation decoder introduces a cross-modality design into Mamba, facilitating the aggregation of visual to audio modalities at the frame level and the temporal level respectively;

[0026] (2.1) Frame-level visual-audio fusion

[0027] First, deep two-dimensional convolution and causal one-dimensional convolution are used to extract visual features. and audio features , the specific formula is as follows:

[0028]

[0029]

[0030] in, represents a linear projection, Representation layer normalization, represents a deep 2D convolution, represents a causal one-dimensional convolution; and Represent the processed visual features and audio features respectively;

[0031] Afterwards, and Input the visual-audio selective scanning module and serialize;

[0032] Finally, the separated audio features After linear mapping and residual processing, the output of this stage is obtained ;

[0033] (2.2) Temporal visual-audio fusion

[0034] Multi-scale visual features After expanding in the order of time-width-height and time-height-width, they are merged in the sequence length dimension. ; Audio features is added to the end of the unfolded multi-scale visual feature sequence and flipped along the time dimension to obtain a multimodal sequence ; The multimodal sequence is selectively scanned through the state space model, so that the audio features can establish the perception of visual information in both time and space dimensions. The specific formula is as follows:

[0035]

[0036] at last, The enhanced audio features are obtained by mapping through the linear layer ;

[0037] Step 4: Context Fusion Pyramid Module

[0038] The contextual fusion pyramid module transforms the original audio features , enhanced multi-scale visual features And the highest resolution feature map of layer 1 As input, the features of each resolution are fused across frames with audio and visual information and upsampled layer by layer to obtain enhanced mask features; specifically, as follows:

[0039] Input visual features as well as First, the temporal Mamba module is used to enhance the cross-frame interaction of visual modalities, and a feature map with enhanced spatiotemporal consistency is obtained. ; Then the enhanced multi-scale feature map They are respectively sent to the audio-visual selective scanning module for feature aggregation; in the audio-visual selective scanning module, the input multimodal features and After a series of linear layer mapping and convolution processing, the enhanced visual features are obtained. and audio features ; Visual features It is expanded along the time-width-height and time-height-width directions to obtain , the audio features are flipped and concatenated to obtain ; Afterwards, the audio features are concatenated in front of the visual features to form a multimodal sequence , and sent to the state space model for sequence modeling. The formula is as follows:

[0040]

[0041]

[0042] The multi-scale features are fused layer by layer to obtain the mask features , and then the mask features are combined with the enhanced audio features Perform matrix multiplication and pass through a multi-layer perceptron to generate the final segmentation mask.

[0043] Furthermore, the specific process of step (1.2) is as follows: the multi-scale visual features of the input First, flatten them into one-dimensional sequences according to the order of time priority and space priority. ,in Indicates the length of the sequence; Flip in the length direction of the sequence and splice into 8 different sequences :

[0044]

[0045] Then, the eight different sequences are scanned in parallel through the state space model and then restored to the original state:

[0046]

[0047]

[0048] in, represents the sequence after reshaping and mixing, Represent the rearranged feature maps respectively; after the subsequent linear mapping layer and residual connection, the final output feature is obtained , the output feature will be subsequently fed into the modality aggregation decoder and the context fusion pyramid module.

[0049] Furthermore, in step (2.1) and The specific process of inputting the visual-audio selective scanning module and serializing it is as follows: First, Expand in the spatial dimension and then analyze the audio features Copy and Splice after copying Before, the visual features are fused into the audio features through the state space model:

[0050]

[0051] in, Indicates that the visual features are expanded in the order of width first, height second, and height first, width second. Indicates that the audio features are copied; is the spliced ​​multimodal sequence;

[0052] The spliced ​​multimodal sequences are passed through the state-space model in parallel to achieve cross-modal aggregation within the same frame;

[0053]

[0054] in, It represents the separation of audio features from the scanned multimodal sequence.

[0055] Beneficial effects of the present invention: After comparative experimental verification, the present invention has achieved the best results in the three sub-tasks of audio-visual segmentation, and has greatly exceeded the previous methods, proving the effectiveness of the proposed model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the overall model structure diagram of this application.

[0057] Figure 2 This is a schematic diagram of the structure of the timing Mamba module of this application.

[0058] Figure 3 This is a structural flow chart of the visual-audio fusion module of this application. DETAILED DESCRIPTION

[0059] 1. Datasets and evaluation metrics

[0060] The audio-visual segmentation task includes three subtasks: single sound source segmentation (S4), multiple sound source segmentation (MS3), and audio-visual semantic segmentation (AVSS). For each task, the corresponding dataset (AVSBench) is used to train and test the model.

[0061] The S4 subset contains 4932 videos of 23 categories, all of which contain only one sound-making object, and the MS3 subset contains 424 videos, all of which contain single or multiple sound-making targets. S4 and MS3 provide binary mask annotations for all videos. The audio-visual semantic segmentation dataset (AVSBench-semantic) contains 12356 videos of 70 target categories, each of which contains semantic mask annotations.

[0062] This scheme uses mIoU and Fscore as the evaluation criteria for segmentation results.

[0063] 2. Specific implementation steps

[0064] Step 1: Multimodal feature extraction

[0065] First, the input video sequence is processed by the visual and audio encoders and audio sequence ( Represents the number of frames in the sequence) to extract features and obtain multi-scale visual features and audio features (in represent different scales).

[0066] Step 2: Multi-scale temporal encoder

[0067] The visual features of levels 2-4 Enhanced visual features obtained by processing with a multi-scale temporal encoder The structure includes VSS module and timing Mamba module, the details are as follows:

[0068] (1) VSS module

[0069] For multi-scale feature maps ( )(in represents the feature channel dimension, and Respectively represent the scale The height and width of the feature map of each scale are first expanded and concatenated in the order of height first, width second, and width first, height second to obtain the concatenated feature sequence. ( represents the length of the concatenated feature sequence). Then the feature sequence is flipped in the length dimension and concatenated with the original sequence to obtain , the specific formula is as follows:

[0070]

[0071] in, Represents a splicing operation, Represents a flip operation.

[0072] After the above operations are completed, It is sent to the state space model (SSM) for causal order scanning. The specific formula is as follows:

[0073]

[0074]

[0075] in, Indicates The sequence is characterized by a state space model. Represents the modeled The features of the sequence are reshaped. Represents the intermediate features Separate and reshape back to the original multi-scale visual features.

[0076] (2) Timing Mamba module

[0077] After the VSS module performs intra-frame interaction on the multi-scale visual features, the processed visual features are sent to the temporal Mamba module, which first extracts the local features of the multi-scale feature map through three-dimensional deep convolution, and then uses three-dimensional selective scanning to enhance the inter-frame interaction of the multi-scale feature map.

[0078] Specifically, the input multi-scale feature map is first flattened into a one-dimensional sequence according to the order of time priority and space priority. (in represents the sequence length). Then Flip in the length direction of the sequence and splice into 8 different sequences :

[0079]

[0080] The eight sequences are then scanned in parallel using the state space model and then restored to their original state:

[0081]

[0082]

[0083] here represents the sequence after reshaping and mixing, Respectively represent the rearranged feature maps. After the subsequent linear mapping layer and residual connection, the final output feature is obtained. , this feature will be subsequently fed into the modality aggregation and context fusion pyramid module.

[0084] Step 3: Modal Aggregation Decoder

[0085] The enhanced multi-scale visual features and audio features Send it to the modality aggregation decoder for visual-audio cross-modal fusion to obtain the fused audio query .

[0086] (1) Frame-level visual-audio fusion

[0087] First, deep two-dimensional convolution and causal one-dimensional convolution are used to extract visual features. and audio features , the specific formula is as follows:

[0088]

[0089]

[0090] in, represents a linear projection, Representation layer normalization, represents a deep 2D convolution, represents a causal one-dimensional convolution. and Represent the processed visual and audio features respectively.

[0091] Afterwards, and Input the visual-audio selective scanning module and serialize it. Specifically, It is unfolded in the spatial dimension and spliced ​​into multiple copies. Before, in order to facilitate the subsequent fusion of visual features into audio features through SSM:

[0092]

[0093] in, Indicates that the visual features are expanded in the order of width first, height second, and height first, width second. Indicates that audio features are copied. is the concatenated multimodal sequence.

[0094] The multimodal sequence is passed through SSM in parallel, effectively achieving cross-modal aggregation within the same frame.

[0095]

[0096] Finally, the separated audio features After linear mapping and residual processing, the output of this stage is obtained

[0097] (2) Temporal visual-audio fusion

[0098] After intra-frame temporal aggregation, the multi-scale visual features After expanding in the order of time-width-height and time-height-width, we get , and then the audio features is added to the end of the sequence and flipped along the time dimension to obtain a multimodal sequence Then, the sequence is selectively scanned through SSM, so that the audio features can establish the perception of visual information in both time and space dimensions. The specific formula is as follows:

[0099]

[0100] at last, The enhanced audio features are obtained by mapping through the linear layer

[0101] Step 4: Context Fusion Pyramid

[0102] This module converts the original audio features , enhanced multi-scale visual features And the highest resolution feature map As input, audio-visual cross-frame fusion is performed on the features of each resolution and upsampled layer by layer to obtain enhanced mask features. The specific steps are as follows:

[0103] The input visual features are firstly passed through the temporal Mamba module to strengthen the cross-frame interaction of visual modalities, and a feature map with enhanced spatiotemporal consistency is obtained. The feature maps of each scale are then sent to the audio-visual selective scanning module for feature aggregation. In this module, the input multimodal features and After a series of linear layer mapping and convolution processing, the enhanced features are obtained. and Then the visual features are expanded along the time-width-height and time-height-width directions to obtain , the audio features are flipped and concatenated to obtain After that, the audio features are concatenated in front of the visual features to form a multimodal sequence , and sent to SSM for sequence modeling, the formula is as follows:

[0104]

[0105]

[0106] The multi-scale features are fused layer by layer to obtain the mask features , and then the mask features are combined with the enhanced audio features Matrix multiplication is performed and the final segmentation mask is generated through a multi-layer perceptron (MLP).

[0107] This example method is used for experimental verification:

[0108] Table 1: Comparison of segmentation performance of this method and previous audio-visual segmentation methods in MS3 and S4 datasets.

[0109]

[0110] Table 2: Comparison of segmentation performance between this method and previous audio-visual segmentation methods on audio-visual semantic segmentation datasets.

[0111]

[0112] Table 1 shows the performance comparison of our method with previous important research works on the S4 and MS3 subsets of the AVSBench dataset. In the S4 setting, our method shows significant performance improvement. For example, when using ResNet50 as the visual encoder, its performance exceeds the existing state-of-the-art methods, with an improvement of 0.6 mIoU and 1.9 Fscore, respectively; and when using PVT-v2, it improves by 0.5 mIoU and 1.0 Fscore, respectively. In the more challenging MS3 setting, our method shows significant performance improvement: when using ResNet50, it achieves an improvement of 8.2 mIoU and 8.0 Fscore, respectively; and when using PVT-v2, it achieves an improvement of 4.4 mIoU and 2.2 Fscore, respectively, which is significantly better than previous methods.

[0113] In addition, Table 2 provides a detailed comparison of the performance of our method with other methods on audio-visual semantic segmentation datasets. It is worth noting that the segmentation performance of our method is better than SelM (another recent implementation based on the Mamba architecture) whether using ResNet50 or PVT-v2 as the visual encoder.

[0114] The outstanding performance under different settings further demonstrates the robustness of the model structure proposed in this application and highlights its effectiveness in promoting multimodal fusion.

Claims

1. A method for audio-visual segmentation based on temporal cross-modal design Mamba, characterized in that: The cross-modal interaction and timing design are integrated into the timing Mamba model to achieve efficient and accurate audio-visual segmentation. First, the input video sequence is encoded by the visual encoder and the audio encoder. and audio sequence Perform feature extraction to obtain multi-scale visual features and audio features ;in, Indicates the number of frames in the sequence, Represents different scales; among them, the visual features of the 2nd to 4th levels Enhanced visual features obtained by processing with a multi-scale temporal encoder ; Enhanced visual features and original audio features It is sent to the modality aggregation decoder, and the visual-audio selective scanning module performs intra-frame and inter-frame visual-audio cross-modal fusion to obtain the fused audio query. ; After that, the highest resolution feature map of the first level , enhanced visual features And the original audio features It is fed into the context fusion pyramid, where the temporal Mamba module and the audio-visual selective scanning module perform cross-frame feature enhancement and upsampling to generate optimized mask features. ; The final segmentation mask is composed of the mask features And the fused audio query Obtained through matrix multiplication.

2. The audio-visual segmentation method according to claim 1, characterized in that: The specific steps of integrating cross-modal interaction and timing design into the timing Mamba model to achieve efficient and accurate audio-visual segmentation are as follows: Step 1: Extraction of visual and audio features Given an input video sequence and audio sequence First, feature extraction is performed through the visual encoder and audio encoder to obtain multi-scale visual features. and audio features ; Step 2: Multi-scale temporal encoder The temporal Mamba module is designed, and a multi-scale temporal encoder is proposed based on it. The VSS module and the temporal Mamba module proposed in VMamba are used to encode the multi-scale visual features of the last three layers. Perform feature encoding and interaction within and between frames; (1.1) VSS module For multi-scale feature maps , ,in represents the feature channel dimension, and Respectively represent the scale The height and width of the feature map; first, expand and concatenate the feature maps of all scales in the order of height first, width second, and width first, height second, to obtain the concatenated feature sequence ,in, Represents the length of the concatenated feature sequence; the concatenated feature sequence is flipped in the length dimension and concatenated with the concatenated feature sequence to obtain , the specific formula is as follows: in, Represents a splicing operation, Indicates a flip operation; Will It is sent into the state space model for causal sequence scanning. The specific formula is as follows: in, Expressing the The sequence is characterized by a state space model; Represents the modeled Reshape the features of the sequence; Represents the intermediate features Separate and reshape back to the original multi-scale visual features; (1.2) Timing Mamba module The multi-scale visual features processed by the VSS module The images are sent to the temporal Mamba module, which first extracts local features of the input multi-scale visual features through 3D deep convolution, and then uses 3D selective scanning to enhance the inter-frame interaction of the multi-scale visual features. Step 3: Modality Aggregation Decoder The modality aggregation decoder introduces a cross-modality design into Mamba, facilitating the aggregation of visual to audio modalities at the frame level and the temporal level respectively; (2.1) Frame-level visual-audio fusion First, deep two-dimensional convolution and causal one-dimensional convolution are used to extract visual features. and audio features , the specific formula is as follows: in, represents a linear projection, Representation layer normalization, represents a deep 2D convolution, represents a causal one-dimensional convolution; and Represent the processed visual features and audio features respectively; Afterwards, and Input the visual-audio selective scanning module and serialize; Finally, the separated audio features After linear mapping and residual processing, the output of this stage is obtained ; (2.2) Temporal visual-audio fusion Multi-scale visual features After expanding in the order of time-width-height and time-height-width, they are merged in the sequence length dimension. ; Audio features is added to the end of the unfolded multi-scale visual feature sequence and flipped along the time dimension to obtain a multimodal sequence ; The multimodal sequence is selectively scanned through the state space model, so that the audio features can simultaneously establish the perception of visual information in the time and space dimensions. The specific formula is as follows: at last, The enhanced audio features are obtained by mapping through the linear layer ; Step 4: Context Fusion Pyramid Module The contextual fusion pyramid module transforms the original audio features , enhanced multi-scale visual features And the highest resolution feature map of layer 1 As input, the features of each resolution are fused across frames with audio and visual information and upsampled layer by layer to obtain enhanced mask features; specifically, as follows: Input visual features as well as First, the temporal Mamba module is used to enhance the cross-frame interaction of visual modalities, and a feature map with enhanced spatiotemporal consistency is obtained. ; Then the enhanced multi-scale feature map They are respectively sent to the audio-visual selective scanning module for feature aggregation; in the audio-visual selective scanning module, the input multimodal features and After a series of linear layer mapping and convolution processing, the enhanced visual features are obtained. and audio features ; Visual features It is expanded along the time-width-height and time-height-width directions to obtain , the audio features are flipped and concatenated to obtain ; Afterwards, the audio features are concatenated in front of the visual features to form a multimodal sequence , and sent to the state space model for sequence modeling. The formula is as follows: The multi-scale features are fused layer by layer to obtain the mask features , and then the mask features are combined with the enhanced audio features Perform matrix multiplication and pass through a multi-layer perceptron to generate the final segmentation mask.

3. The audio-visual segmentation method according to claim 2, characterized in that: The specific process of step (1.2) is as follows: Input multi-scale visual features First, flatten them into one-dimensional sequences according to the order of time priority and space priority. ,in Indicates the length of the sequence; Flip in the length direction of the sequence and splice into 8 different sequences : Then, the eight different sequences are scanned in parallel through the state space model and then restored to the original state: in, represents the sequence after reshaping and mixing, Represent the rearranged feature maps respectively; after the subsequent linear mapping layer and residual connection, the final output feature is obtained , the output feature will be subsequently fed into the modality aggregation decoder and the context fusion pyramid module.

4. The audio-visual segmentation method according to claim 2, characterized in that: In step (2.1) and The specific process of inputting the visual-audio selective scanning module and serializing it is as follows: First, Expand in the spatial dimension and then analyze the audio features Copy and Splice after copying Before, the visual features are fused into the audio features through the state space model: in, Indicates that the visual features are expanded in the order of width first, height second, and height first, width second. Indicates that the audio features are copied; is the spliced ​​multimodal sequence; The spliced ​​multimodal sequences are passed through the state-space model in parallel to achieve cross-modal aggregation within the same frame; in, It represents the separation of audio features from the scanned multimodal sequence.

Images