An audio-visual segmentation method based on state space model and dynamic focus loss

Abstract

The application discloses an audio-visual segmentation method based on a state space model and a dynamic focus loss, and belongs to the technical field of computer vision and multi-modal information processing. The method comprises the following steps: acquiring a video sequence and a corresponding audio sequence; extracting multi-scale visual features and audio features; using a cross-time audio query generation module comprising an audio-video feature aggregation unit based on a state space model to generate enhanced audio query features; using the audio-video feature aggregation unit to perform cross-modal fusion to generate audio-video fusion features; performing post-processing fusion of the fusion features and the multi-scale visual features to generate a segmentation probability map; and training a model using a dynamic focus loss function. The application realizes deep dynamic alignment of audio-video features through a state space model, and adaptively processes difficult samples and unbalanced positive and negative samples through a dynamic focus loss function, so that the audio-visual target segmentation can be efficiently and accurately realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and multimodal information processing technology, specifically to an audiovisual segmentation method based on a state-space model and dynamic focus loss, used to perform pixel-level segmentation of sound-producing objects in a video based on audio signals. Background Technology

[0002] Audio-Visual Segmentation (AVS) aims to locate and segment sound-emitting objects at the pixel level based on the sounds they emit in a scene, and is an important branch of scene understanding. Existing AVS tasks can be divided into three categories: single-source segmentation (S4), multi-source segmentation (MS3), and audio-visual semantic segmentation (AVSS). Existing methods have explored aspects such as audio-visual relevance matrix modeling, cross-attention mechanisms, audio-guided class activation mapping, and long-distance dependency modeling of audio and video features, but still face two major challenges: insufficient interaction between audio and video features and imbalance in pixel-level sample distribution.

[0003] In existing technologies, the paper "AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation" published by Gong et al. (IEEE Transactions on Multimedia, 2025) proposes an audiovisual segmentation method based on a state-space model. This method utilizes the Mamba framework to process long sequences, but it only uses the state-space model for single-modal feature modeling. In multimodal fusion, it still employs simple multiplication and attention mechanisms for coarse-grained interaction, failing to fully leverage the potential of the state-space model in cross-modal dynamic alignment and not addressing the pixel-level sample distribution imbalance problem.

[0004] Another existing technique, the paper "AVSegFormer: Audio-Visual Segmentation with Transformer" published by Gao et al. (at the AAAI Conference on Artificial Intelligence, 2024), uses the Transformer architecture for audio-video feature interaction. This method guides visual features through audio queries, but its computational complexity is quadratic when processing long video sequences, making it difficult to efficiently model long-distance dependencies. Furthermore, its pixel-level sample distribution imbalance problem is not effectively solved, resulting in limited segmentation accuracy.

[0005] In summary, existing technologies suffer from shortcomings such as insufficient interaction of audio and video features, imbalance in pixel-level sample distribution, and high computational complexity in processing long sequences. Therefore, enhancing the deep interaction of audio and video features and designing effective optimization strategies to address the imbalance between positive and negative samples and between easy and difficult samples are crucial for improving the performance of audiovisual target segmentation. Summary of the Invention

[0006] The purpose of this invention is to overcome the above-mentioned defects and provide an audiovisual segmentation method based on a state-space model and dynamic focus loss that can achieve efficient cross-modal semantic alignment, adaptively handle imbalance between easy and difficult samples and positive and negative samples, and achieve efficient and accurate audiovisual target segmentation.

[0007] To achieve the above objectives, the following technical solution is adopted:

[0008] The present invention provides an audiovisual segmentation method based on a state-space model and dynamic focus loss, comprising the following steps:

[0009] (1) Obtain the video sequence and the corresponding audio sequence;

[0010] (2) Use a video encoder to extract features from the video sequence to generate multi-scale visual features; use an audio encoder to extract features from the audio sequence to generate audio features;

[0011] (3) Input the audio features into the cross-time audio query generation module to generate enhanced audio query features; the cross-time audio query generation module includes an audio and video feature aggregation unit based on a state space model;

[0012] (4) Input the enhanced audio query features and the visual features into the cross-modal audio and video interaction module, and perform cross-modal fusion through the state space model-based audio and video feature aggregation unit to generate audio and video fusion features;

[0013] (5) The audio-visual fusion features are post-processed and fused with the multi-scale visual features to generate the final segmentation probability map;

[0014] (6) The model is trained using a dynamic focus loss function, which includes an adaptive difficulty coefficient, an adaptive gradient correction factor, and an adaptive positive and negative pixel balance weight.

[0015] The above-mentioned audiovisual segmentation method based on state-space model and dynamic focus loss includes: the audio-visual feature aggregation unit based on state-space model is used to simultaneously model the dynamic dependency relationship between the main modality and the auxiliary modality; for the main modality input sequence and the auxiliary modality embedded features, independent state-space parameter matrices are used for evolution, and then the two outputs are fused.

[0016] The above-mentioned audiovisual segmentation method based on state-space model and dynamic focus loss, wherein: the cross-time audio query generation module is specifically used to: initialize learnable query parameters, use the audio-video feature aggregation unit to deeply interact the query parameters with the audio features, suppress background noise and enhance key semantic information, and generate audio query features.

[0017] The above-mentioned audiovisual segmentation method based on state space model and dynamic focus loss, wherein: the cross-modal audio-visual interaction module is specifically used to: adaptively refine the audio query features using the standard state space model, and then use the audio-visual feature aggregation unit to fuse the refined audio query features with the visual features, so that the audio query can dynamically locate the sound region in the visual feature sequence.

[0018] The above-mentioned audiovisual segmentation method based on a state-space model and dynamic focus loss, wherein the dynamic focus loss function is expressed as:

[0019]

[0020] in, Indicates the dynamic focusing parameters. This indicates the focus parameter, which is set based on prior experience. The dynamic learning coefficient is represented as follows:

[0021]

[0022] Represents the total number of pixels. Indicates the current video number Frame and current category The model outputs probabilities, and the true labels are represented as... ;

[0023] The global gradient correction factor is expressed as follows:

[0024]

[0025] in, For gradient balance coefficients, The balancing weighting coefficient is represented by the following formula:

[0026] .

[0027] Compared with the prior art, the present invention has the following beneficial effects, as can be seen from the above technical text:

[0028] 1. Full interaction of audio and video features: Through the audio and video feature aggregation unit based on the state space model, it is possible to dynamically capture the long-distance dependence and semantic correspondence between audio and video modalities, realize the deep alignment and interaction of features, and effectively improve the segmentation accuracy in complex dynamic scenes.

[0029] 2. Adaptive Sample Balancing: The dynamic focus loss function achieves differentiated weighted learning of easy and difficult samples and adaptive calibration of class distribution by using adaptive difficulty coefficients, global gradient correction factors, and positive and negative pixel balancing weights. This effectively alleviates the optimization bias caused by the disparity in the ratio of positive and negative samples and the uneven distribution of easy and difficult samples.

[0030] 3. High computational efficiency: The state-space model has linear computational complexity, which, compared to the quadratic complexity of the Transformer, can efficiently process long video sequences and reduce computational overhead.

[0031] 4. Excellent segmentation performance: Experiments on the AVSBench-Object and AVSBench-Semantic datasets show that the present invention achieves competitive performance in single-source, multi-source, and audiovisual semantic segmentation tasks. Attached Figure Description

[0032] Figure 1 A schematic diagram of the overall structure of the audiovisual segmentation network based on the state-space model and dynamic focus loss provided by this invention;

[0033] Figure 2 This is a schematic diagram of the cross-time audio query generation and cross-modal audio-video interaction module based on the state space model in this invention;

[0034] Figure 3 This is a heatmap comparison diagram between the present invention and the baseline model;

[0035] Figure 4 These are example images showing the segmentation results of this invention on different datasets; Detailed Implementation

[0036] The following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effectiveness of the audiovisual segmentation method based on a state-space model and dynamic focus loss proposed in this invention:

[0037] Example 1

[0038] An audiovisual segmentation method based on a state-space model and dynamic focus loss is illustrated in the following diagram: Figure 1 This includes the following steps:

[0039] Step 1: Data Acquisition and Preprocessing

[0040] Obtain the input video sequence and the corresponding audio sequence Where T represents the sequence length. The audio is resampled into a 16kHz mono signal, processed by a short-time Fourier transform and a 64th-order filter bank, and a log-Mel spectrogram of size T×96×64 is constructed, where the number of sampled segments is consistent with the video sequence length T;

[0041] Step 2: Feature Extraction

[0042] The video encoder uses the Pyramid Visual Transformer (PVTv2-B5) as its backbone network to process each frame of the image. Extracting multi-scale visual features: shallow features Preserving high-resolution detail information, the last three layers of features , , Deep fusion is performed using a multi-scale variable attention mechanism to generate enhanced video features. :

[0043] The audio encoder uses the VGGish model (loaded with AudioSet pre-trained weights), inputting a log-Mel spectrogram into the network to extract deep semantic information and obtain an audio feature sequence. ;

[0044] Step 3: Generate cross-time audio query

[0045] Specific details of the cross-time audio query generation module are as follows: Figure 2 Initialize a set of learnable query parameters Here, N and D are set to N=1029 and D=256 to ensure that the length and dimension of subsequent video sequences remain consistent. The query parameters are expanded along the batch dimension and then input into the audio-video feature aggregation unit (MAVFA) based on the state-space model, along with audio features. To enable deep interaction; the calculation process of MAVFA is as follows: Figure 2 The MAVFA module in the middle, the pseudocode of MAVFA is as follows:

[0046]

[0047] The working process of the audio and video feature aggregation unit based on the state-space model is as follows: For the main modality (query) and the auxiliary modality (audio), independent state-space parameter matrices are used for evolution, and the discretized form of the state-space model is as follows:

[0048]

[0049] in: , , The input-dependent discretization parameters are used; the two outputs are linearly projected, element-wise superimposed, and activated, then the final output is obtained through a shared output projection layer; this generates enhanced audio query features. ;

[0050] Step 4: Cross-modal audio and video interaction

[0051] First, using standard Mamba modules... Adaptive refinement is performed, and the feature distribution is stabilized through a normalization layer to obtain the enhanced query features. Subsequently, the MAVFA module was introduced as a cross-modal bridge to enhance audio queries. With video features By fusing audio data, audio queries can dynamically locate potential sound regions within video feature sequences, identify the spatial location and distance changes of sound sources, and generate a joint representation containing rich semantics. ;Specific details of the cross-modal audio and video interaction module are as follows Figure 2 As shown.

[0052] Step 5: Post-processing fusion

[0053] Utilizing shallow visual features As the basic spatial representation, the visual features of deep fusion in the encoder are upsampled using bilinear interpolation, and then superimposed to achieve the same resolution as the shallow features. Then, audio features are introduced separately. After interactively calculating similarity through matrix multiplication, it is processed by the sigmoid activation function, and then global fusion features are utilized. The result of semantic guidance is the fusion of global semantic information and local visual features. Finally, the residuals are stacked and a segmentation prediction head (convolutional layer + progressive upsampling + multilayer perceptron) is used to generate the final segmentation probability map. ;

[0054] Step 6: Dynamic Focus Loss Training

[0055] The model is optimized using a dynamic focus loss function:

[0056] The dynamic focus loss function is expressed as:

[0057]

[0058] in:

[0059] An adaptive difficulty coefficient;

[0060] This is the global gradient correction factor;

[0061] The adaptive positive and negative pixel balance weights are obtained after normalization.

[0062] Indicates the dynamic focusing parameters. This represents the basic focusing parameters, set based on prior experience. The dynamic learning coefficient is represented as follows:

[0063]

[0064] Represents the total number of pixels. Indicates the current sample Compared with the current category The model outputs probabilities, and the true labels are represented as... .

[0065] The global gradient correction factor is expressed as follows:

[0066]

[0067] in, For gradient balance coefficients, This represents the balancing weighting coefficient. The specific formula is as follows:

[0068]

[0069] During training, the AdamW optimizer was used with a batch size of 2 and an initial learning rate of 1. Both the encoder and decoder are configured with 6 layers, the embedding dimension is uniformly 256, and the dropout ratio is 0.1.

[0070] For single-source segmentation tasks, the training process lasts for 30 rounds, with the backbone network frozen in the first 5 rounds; for multi-source and semantic segmentation tasks, the training process lasts for 60 rounds.

[0071] Example 2

[0072] An audiovisual segmentation method based on a state-space model and dynamic focus loss includes:

[0073] The method is basically the same as in Example 1, except that the video encoder uses ResNet-50 as the backbone network, and the method for extracting multi-scale features is adjusted accordingly, using Cross Attention instead of the MambaAVS module. In the state-space model-based audio-video feature aggregation unit, the dimension of the state-space parameter matrix is ​​set to 256, and the initial learnable parameter N in the cross-time audio query generation is set to 300; this does not need to be consistent with subsequent audio. The basic focusing parameters in the dynamic focus loss function... The training rounds are set to 30 rounds. The remaining steps and parameters are the same as in Example 1. Figure 3 This is a heatmap comparison diagram of the present invention and the baseline model in the case of Example 2. The heatmap results show that the proposed MambaAVS is effective.

[0074] Example 3

[0075] An audiovisual segmentation method based on a state-space model and dynamic focus loss includes: [The method is] basically the same as in Example 1, except that: in this paper, Focal loss is used as the loss function for training. , The parameters are fixed, rather than dynamically adjustable based on the number of positive and negative samples as described in this paper. For detailed calculation methods, please refer to step six. The remaining steps and parameters are the same as in Example 1.

[0076] Example 4

[0077] An audiovisual segmentation method based on a state-space model and dynamic focus loss includes: The method is basically the same as in Example 1, except that: the video encoder is a Swing Transformer; the audio encoder is a CNN-based audio feature extraction network; and in the post-processing fusion step, a pyramid pooling module is used instead of bilinear interpolation upsampling. The remaining steps and parameters are the same as in Example 1. Figure 4 Examples of segmentation results of the method of the present invention on different datasets are provided using Example 4 as the basic framework. The results show that the proposed method can accurately segment emitting objects.

[0078] The above are merely preferred embodiments of the present invention and do not impose any limitations on the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from any technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. An audiovisual segmentation method based on a state-space model and dynamic focus loss, characterized in that, Includes the following steps: (1) Obtain the video sequence and the corresponding audio sequence; (2) Use a video encoder to extract features from the video sequence to generate multi-scale visual features; use an audio encoder to extract features from the audio sequence to generate audio features; (3) Input the audio features into the cross-time audio query generation module to generate enhanced audio query features; The cross-time audio query generation module includes an audio and video feature aggregation unit based on a state-space model; (4) Input the enhanced audio query features and the visual features into the cross-modal audio and video interaction module, and perform cross-modal fusion through the state space model-based audio and video feature aggregation unit to generate audio and video fusion features; (5) The audio-visual fusion features are post-processed and fused with the multi-scale visual features to generate the final segmentation probability map; (6) The model is trained using a dynamic focus loss function, which includes an adaptive difficulty coefficient, an adaptive gradient correction factor, and an adaptive positive and negative pixel balance weight.

2. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 1, characterized in that, The state-space model-based audio-visual feature aggregation unit is used to simultaneously model the dynamic dependencies between the primary modality and the auxiliary modality; for the primary modality input sequence and the auxiliary modality embedded features, The two outputs are then fused together using independent state-space parameter matrices.

3. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 1, characterized in that, The cross-time audio query generation module is specifically used to: initialize learnable query parameters, use the audio and video feature aggregation unit to deeply interact the query parameters with the audio features, suppress background noise and enhance key semantic information, and generate audio query features.

4. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 1, characterized in that, The cross-modal audio-visual interaction module is specifically used to: adaptively refine the audio query features using a standard state-space model, and then use the audio-visual feature aggregation unit to fuse the refined audio query features with the visual features, so that the audio query can dynamically locate the sound region in the visual feature sequence.

5. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 1, characterized in that, The dynamic focus loss function is expressed as: ； Where, ξ d ξ is the dynamic focusing parameter. g ξ is the global gradient correction factor. l To adaptively balance the positive and negative pixel weights, Output probabilities for the model.

6. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 5, characterized in that, The dynamic focusing parameters ,in This indicates the focus parameter, which is set based on prior experience. The dynamic learning coefficient is represented as follows: ； in, Represents the total number of pixels. Indicates the current video number Frame and current category The model outputs probabilities. This indicates the actual label.

7. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 5, characterized in that, The global gradient correction factor It is expressed as follows: ； in, This is the gradient balance coefficient.

8. The audiovisual segmentation method based on a state-space model and dynamic focus loss as described in claim 5, characterized in that, The aforementioned balance weight coefficient It is expressed as follows: 。

Images