Audio-visual segmentation method integrated with frequency domain design

Abstract

The invention discloses an audio-visual segmentation method integrated with frequency domain design. Firstly, a method for exploring comprehensive audio-visual alignment and fusion by combining a spatial domain and a frequency domain is introduced, audio-visual consistency is improved by integrating frequency domain information, and fine alignment of multi-modal features is enhanced. A frequency domain-oriented audio integration module and a frequency domain-based cross-modal fusion module are provided. Wherein the frequency domain oriented audio integration module encodes audio information into visual representation through early fusion based on frequency domain enhancement, so that more detailed visual representation of audio perception is generated, and subsequent generation of more robust multi-mode representation is facilitated; the cross-modal fusion module based on the frequency domain aims at exploring audio-visual association in combination with the spatial domain and the frequency domain and enhancing cross-modal feature alignment, so that the segmentation performance of the model is improved. The frequency-domain-oriented audio integration module is integrated to each stage of the encoder, frequency-domain information and audio clues are fully utilized, the difference between audio-visual modes is reduced, and more accurate audio-visual fusion and alignment are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal (audio-visual modality) information processing, and relates to an audio-visual segmentation method integrating frequency domain design, specifically an audio-visual segmentation method based on frequency domain drive, multimodal fusion and attention mechanism. Background Art

[0002] With the development of multimedia information processing technology, audio-visual segmentation (AVS) technology has emerged, aiming to accurately separate the sound-emitting visual objects from video and audio data. This technology has broad application prospects in many fields such as video editing, security monitoring, and human-computer interaction. Existing audio-visual segmentation methods mainly focus on using convolutional neural networks (CNNs) or attention mechanisms in the spatial domain for cross-modal feature interaction to achieve audio-visual segmentation. For example, Zhou et al. cleverly used a cross-modal attention module in Audio-visual segmentation to achieve early fusion and combined it with a fully convolutional network (FCN) decoder to achieve audio-visual segmentation. Although this method is simple and effective, it still has certain limitations, including the failure to fully utilize the full potential of audio-visual information, limited to pixel-level correlation exploration, and lack of in-depth correlation analysis at the instance level of the sound-emitting object. Gao et al. introduced a Transformer architecture based on the attention mechanism in Audio-Visual Segmentation with Transformer to achieve a more robust audio-visual cross-modal representation and improve the performance of audio-visual segmentation. In addition, some work has introduced contrastive learning to improve the effectiveness of cross-modal attention, thereby improving the accuracy of audio-visual segmentation. However, the above methods are limited to spatial domain audio-visual segmentation, lack the exploration of frequency domain information, and ignore the intrinsic connection of audio-visual features in the frequency domain, resulting in insufficient integration and alignment of multimodal features.

[0003] Therefore, in response to the deficiencies and challenges in the above studies, the present invention innovatively introduces a frequency-driven strategy and utilizes the synergy of spatial and frequency information to significantly improve the integration and alignment effects of multimodal features, thereby effectively improving the consistency of audiovisual features and the audiovisual segmentation performance. Summary of the invention

[0004] The present invention proposes a method for implementing efficient audio-visual segmentation by using frequency domain information to strengthen the integration and alignment strategy of multimodal data. This method can effectively improve the consistency of audio-visual correspondence, achieve more accurate segmentation boundaries of sounding objects and higher audio-visual segmentation accuracy.

[0005] The technical solution of the present invention:

[0006] An audio-visual segmentation method integrating frequency domain design, the steps are as follows:

[0007] Step 1: Feature encoding via frequency-domain oriented audio integration module optimization

[0008] In the audio encoding stage, the VGGish model is used as an audio encoder to extract audio features. In the visual encoding stage, a frequency domain-oriented audio integration module is designed, and a design combining frequency domain enhancement and early fusion is introduced in the image encoding process to generate more effective visual representations.

[0009] ResNet-50 or PVT-v2 is used as the visual encoder. The visual encoder contains four stages for gradually extracting features, which are represented as stage1, stage2, stage3 and tag4. ResNet-50 is based on a convolutional neural network. It gradually reduces the resolution of the feature map through residual connections and convolution operations, while increasing the number of channels. PVT-v2 uses the Transformer architecture to extract features through a self-attention mechanism and patch embedding. In each stage, global information is effectively captured through spatial downsampling and optimized calculation methods. The frequency-domain-oriented audio integration module is inserted after the output of each stage to process the feature map generated by the corresponding stage. After being processed by the frequency-domain-oriented audio integration module, the visual enhancement features obtained are used as the output of the current stage and are passed to the next stage as input. The details of each frequency-domain-oriented audio integration module are as follows: For the audio features output by the audio encoder and the spatial domain visual feature map from the current stage of the visual encoder First, the spatial domain visual features are transformed using fast Fourier transform Convert to frequency domain:

[0010]

[0011] in, represents the frequency domain feature map, represents the fast Fourier transform, Represents the visual feature map The height and width index of Respectively represent the frequency domain feature maps after conversion The height and width index of the

[0012] Introducing a threshold The frequency domain feature map Divided into high-frequency components and low-frequency components, as shown in the following formula:

[0013]

[0014]

[0015] Furthermore, by introducing learnable parameters and Adaptively adjust the decoupled frequency components to generate frequency domain enhanced visual representation. The process is expressed as follows:

[0016]

[0017] in, represents the visual representation after frequency domain enhancement, represents the inverse fast Fourier transform, which is used to map frequency domain features back to the spatial domain;

[0018] Afterwards, a cross-modal attention mechanism is introduced to convert the audio features generated by the audio encoder to visual representation, and use residual connections to obtain audio-aware visual representation , as the final output of the current stage and the input of the next stage; the process is expressed in the following formula:

[0019]

[0020] in, Represents the cross-modal attention mechanism;

[0021] Finally, the visual encoder outputs four multi-scale encoding features { },in , T represents the time of the input video frame, H and W represent the height and width of each frame of the input video respectively, Indicates the number of channels for each encoded feature;

[0022] Afterwards, a pixel decoder based on a multi-scale deformable Transformer is used to perform multi-scale visual feature fusion; the details are as follows: Multi-scale encoded features extracted by the visual encoder is further fed into the pixel decoder, the multi-scale encoding feature { The visual features of the last three scales in are expanded, connected along the channel dimension, and then fused through the multi-scale deformable Transformer layer to obtain enhanced multi-scale visual features { ;

[0023] Step 2: Cross-modal fusion based on frequency domain

[0024] Design a frequency-domain-based cross-modal fusion module to fully explore the audio-visual correspondence relationship; the frequency-domain-based cross-modal fusion module consists of two branches: a spatial guidance branch and a frequency-domain perception branch;

[0025] (1) Spatial-guided branch: including spatial domain cross-modal attention operation: using cross-modal attention mechanism to perform audio-visual fusion in spatial domain, for the audio features output by the audio encoder and the multi-scale visual features output by the pixel decoder { Maximum resolution visual features in , respectively, the encoded audio features As query, spatial domain visual features Input as key and value into the cross-modal attention mechanism to obtain fused multimodal features , the process is described as follows:

[0026]

[0027] in, , , It maps the features to a dimension of The learnable projection matrix of the intermediate layer features; then, the multimodal features For subsequent optimization and adjustment of visual features, Through a Weighted residual connections, and encoded audio features Combined to achieve enhanced audio characteristics :

[0028]

[0029] (2) Frequency-domain-aware branch: The frequency-domain-aware branch works in parallel with the spatial-guided branch. First, the maximum resolution visual features in the spatial domain are transformed using the fast Fourier transform. Convert to the frequency domain to obtain the frequency domain visual feature map, and then use global average pooling to reduce the dimension of the frequency domain visual feature while retaining the global information; considering that the main structure and texture information of the frequency domain visual feature are reflected in the amplitude spectrum, and the audio feature is obtained by extracting the audio spectrum map and encoding it using the audio encoder, the frequency domain visual feature map is further extracted with spectrum features to facilitate subsequent audio-visual interaction and alignment; the process is expressed as follows:

[0030]

[0031]

[0032]

[0033] in, , , , , They represent the frequency domain visual feature map after Fourier transform, the frequency domain features after global average pooling, the global average pooling operation, The real part and The imaginary part of

[0034] Finally, the frequency-domain-aware branch works in tandem with the spatially guided branch to optimize and adjust the visual representation; spectral features And fused multimodal features First, each is processed independently by a convolution layer, and then the two are deeply fused through element-by-element addition and convolution layer, so that the spectral features Multimodal features with fusion Mutual complementation and enhancement promote the effective interaction and fusion of multimodal information in the frequency domain; the process is expressed as:

[0035]

[0036] in, represents the convolution operation, Represents the output result, which is used to weight and adjust the original visual features; Adjustment by multiplication , also through residual connections with learnable parameter weights, to obtain enhanced visual features ;

[0037]

[0038] in, represents the weighted learnable parameters;

[0039] Through the frequency domain-based cross-modal fusion module, the optimized audio features are finally obtained and visual features ;

[0040] Step 3: Implement mask generation through decoder

[0041] The Mask2Former architecture is used as the decoder to generate the sound object query embedding. The sound object query embedding is then combined with the one obtained in step 2. Combine to generate a prediction mask;

[0042] Specifically, the optimized audio features obtained by the frequency domain cross-modal fusion module and the learnable query obtained by initializing the learnable parameters are combined through point-by-point addition as the query of the decoder; at the same time, the pixel decoder performs multi-scale feature fusion to output the enhanced multi-scale visual features { As keys and values, they interact with audio-based queries through a layer-by-layer cross-modal attention mechanism to generate query embeddings for sounding objects. ;at last, The classification prediction is obtained through the linear layer, The optimized maximum resolution visual features obtained by the frequency domain-based cross-modal fusion module Multiply them together to get the prediction mask;

[0043] Step 4: Model training

[0044] During the training process, both the category loss and the predicted mask loss are considered. In order to utilize the potential temporal coupling of audio-visual segmentation to improve the performance of the model, an adaptive inter-frame consistency loss is used. The three losses are combined to obtain the total loss and back-propagated to optimize the model:

[0045]

[0046]

[0047]

[0048]

[0049] in, represents the true category, Represents the model prediction category, category loss function Calculated by the cross entropy function; and Represents the label of a point on the real mask and the predicted mask, respectively, and the mask loss function It consists of binary cross entropy and Dice loss; represents the prediction mask of the t-th frame, That is, adaptive inter-frame consistency loss; , , They represent weighting parameters, That is the total loss function.

[0050] Beneficial effects of the present invention:

[0051] (1) The present invention first introduces a method that combines spatial domain and frequency domain to explore comprehensive audio-visual alignment and fusion. The integration of frequency domain information improves audio-visual consistency and enhances the fine alignment of multimodal features.

[0052] (2) Two innovative frequency domain design modules are proposed, namely, a frequency domain-oriented audio integration module and a frequency domain-based cross-modal fusion module. The frequency domain-oriented audio integration module encodes audio information into visual representations through early fusion based on frequency domain enhancement, thereby generating a more detailed audio-perceived visual representation, which helps to generate more robust multimodal representations later; the frequency domain-based cross-modal fusion module aims to explore audio-visual associations by combining spatial and frequency domains, enhancing cross-modal feature alignment, and thus improving the segmentation performance of the model.

[0053] (3) The frequency-domain-oriented audio integration module is integrated into each stage of the encoder, making full use of frequency-domain information and audio cues, reducing the differences between audiovisual modalities, and facilitating more accurate audiovisual fusion and alignment. Different from previous work that only explored audiovisual interactions in the spatial domain, the frequency-domain-based cross-modal fusion module coordinates the exploration of audiovisual interactions in the spatial and frequency domains, promoting more refined and accurate segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the overall architecture diagram of the audio-visual segmentation method with integrated frequency domain design involved in this application.

[0055] Figure 2 This is a detailed diagram of the frequency domain-oriented audio integration module within the method involved in this application.

[0056] Figure 3 This is a detailed diagram of the frequency domain-based cross-modal fusion module in the method involved in this application. DETAILED DESCRIPTION

[0057] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0058] Reference Figure 1 , an audio-visual segmentation method integrating frequency domain design, comprising the following steps:

[0059] Step 1: Data preprocessing, including the following steps: Dataset selection and cropping: The AVSBench dataset for audio-visual segmentation tasks is used, which includes two datasets: AVSBench-object and AVSBench-sematic. AVSBench-object also includes two sub-datasets: single-source audio-visual segmentation dataset (S4) and multi-source audio-visual segmentation dataset (MS3). S4 contains 4932 videos and uses a semi-supervised training method. The training set, validation set, and test set are divided into 70 / 15 / 15. MS3 includes 424 videos and uses a fully supervised training method. The training set, validation set, and test set are divided into the same ratio as S4. AVSBench-semantic adds semantic labels for audio-visual semantic segmentation (AVSS) tasks. It contains 11356 videos, of which 8498 are used for training, 1304 are used for validation, and 1554 are used for testing. During the training and testing experiments, this design uniformly crops the video frames to a size of 224×224. In addition, two data augmentation methods including horizontal flipping and color enhancement are used on all input video frames.

[0060] Step 2: Audio-visual encoding, including the following steps: First, the audio Mel spectrogram obtained from the audio is sent to the pre-trained audio encoder VGGish to extract audio features . To ensure that the representation ability of the audio encoder remains stable, its weights are frozen during the training process. After that, the video frames obtained in the data processing step are encoded using the pre-trained visual encoder ResNet-50 or PVT-v2 model in conjunction with the frequency-domain-oriented audio integration module. The specific details of this step are as follows: Keeping the overall structure of the visual encoder unchanged, inserting the frequency-domain-oriented audio integration module into the four main stages of the visual encoder, using frequency-domain enhancement and gradually integrating audio cues in the process of extracting image features to generate audio-perceived visual representations. Among them, the specific details of the frequency-domain-oriented audio integration module are as follows: Use Fourier transform to convert the spatial domain visual feature map into a frequency domain feature map, and then use a threshold of 0.1 to divide the frequency domain feature map into high and low frequency components. Set the learnable parameters 1.0 and 1.5 for the high and low frequency components respectively, and perform multiplication operations. Add the adjusted high and low frequency components to obtain an enhanced frequency domain feature map. Then use inverse Fourier transform to convert the frequency domain feature map back to the spatial domain. Then, the frequency-domain enhanced visual representation is fused with audio cues through a multi-head cross-modal attention mechanism, and the original visual information is retained through residual connections to obtain an audio-perceived visual representation. Finally, the visual encoder of this design outputs each frequency-domain-oriented audio integration module to obtain a multi-scale visual feature map. Subsequently, a pixel decoder based on a multi-scale deformable Transformer is used to perform multi-scale feature fusion on the multi-scale visual feature map. The details of the pixel decoder are as follows: six layers of Transformer layers are used to perform feature fusion on the three multi-scale visual feature maps of i=2~4 output by the visual encoder to enhance the multi-scale visual feature map, and all multi-scale features are also mapped to the same channel dimension through convolutional layers.

[0061] Step 3: Audio-visual bidirectional interaction optimization. The specific steps are as follows: The highest resolution visual features output by the pixel decoder The encoded audio features are fed into the frequency domain-based cross-modal fusion module, which uses spatial and frequency domain information to interactively and bidirectionally optimize visual and audio features to obtain improved audio and visual features. The specific details are as follows: In the spatially guided branch, this design uses a multi-head cross-modal attention mechanism for spatial domain audio-visual fusion, with 8 attention heads, where audio is used as the query for attention operation, and visual features are used as keys and values ​​to obtain fused multi-modal features. The fused multimodal features are further fused with the original audio features through a residual connection with a weight of 1.0 to obtain enhanced audio features. In the frequency domain perception branch, the spatial domain visual features are first converted to the frequency domain using fast Fourier transform, the visual feature dimension is reduced using global average pooling, and then the spectral features of the frequency domain visual feature map are extracted. Furthermore, the spatial guidance branch outputs are fused through convolution and point-by-point addition. and the frequency domain sensing branch and use the result As the weights adjust the visual features, the enhanced visual features are finally obtained through a residual connection with weights The learnable parameters in this process are Set to 1.0, the formula is as follows:

[0062]

[0063] Step 4: Mask generation, including the following steps: Using a Transformer-based multi-layer cascade decoder architecture, a set of learnable query embeddings are combined with the audio features optimized in step 3 by point-by-point addition as the initial query, and each query corresponds to a potential sound target object. The decoder is performed layer by layer on the multi-scale visual features. Each decoding layer includes a self-attention layer for the query, a cross-attention layer that interacts with the query and the visual feature map as keys and values, and a feed-forward network consisting of normalization and linear layers. The final decoder outputs a sound object query embedding .

[0064] Final query embedding Input two branches respectively: classification head and masking head. The classification head predicts the category of the sounding object. The details are as follows: the query embedding is mapped to the same embedding space as the number of categories through a linear layer to obtain the category prediction. The formula is as follows:

[0065]

[0066] The mask header generates a binary segmentation mask by interacting with the high-resolution feature map at the pixel level. The specific details are as follows: First, the sound object query embedding is processed by a multi-layer perceptron, and then it is multiplied with the enhanced visual features output by step 3 to obtain the predicted mask. The formula is as follows:

[0067]

[0068] in () indicates multi-layer perceptron processing, Represents matrix multiplication.

[0069] Step 5: Loss function and optimization. The specific steps are as follows: Consider the category prediction loss, mask prediction loss, and adaptive inter-frame consistency loss during training. The total loss is obtained by combining the three losses and back-propagating the optimization model. Category prediction loss function formula:

[0070]

[0071] in, represents the true category, Represents the model prediction category. Mask prediction loss function formula:

[0072]

[0073] in, and Represents the label of a point on the real mask and the predicted mask respectively. The formula of adaptive inter-frame consistency loss function is: in, represents the predicted mask of the tth frame. The total loss function formula can be expressed as:

[0074]

[0075] in, , , They represent weighted parameters, which are set to 2.0, 5.0, and 10.0 respectively. Finally, the total loss is calculated through the total loss function and the model is back-propagated to optimize.

[0076] Experimental verification:

[0077] 1. Verification details

[0078] The method is based on the Mask2Former model architecture: Two visual encoder settings are used in the experiment: ResNet-50 and PVT-v2 pre-trained on ImageNet, and the audio encoder uses a pre-trained VGGish model. The pixel decoder uses a multi-scale deformable attention deformer architecture. In the decoder, the number of learnable queries is set to 100. This method also introduces various data augmentation techniques, including flipping and random cropping. The AdamW optimizer combined with momentum and adaptive learning rate is selected during training to improve training stability.

[0079] During the training process, the weight coefficients of the loss function were set to λcls = 2, λmas = 5, λada = 10, the learning rate was set to 0.0001, the weight decay was set to 0.05, the crop size was set to 224×224, and the batch size was set to 8. The single sound source subset in AVSBench-Object and AVSBench-Semantic were trained for 90,000 iterations, while the single sound source subset in AVSBench-Object was trained for 20,000 iterations.

[0080] To ensure fairness, following previous studies, we compare models using the same visual encoder and use the same evaluation metric: Jaccard index. and F-score .

[0081] 2. Quantitative Results

[0082] Table 1 shows the comparison results between the advanced audio-visual segmentation method and this design. The performance of the model designed in this application has achieved excellent performance, especially on the multi-source subset and audio-visual semantic segmentation datasets, surpassing the previous state-of-the-art (SOTA) method. Regardless of whether the image encoder used is ResNet-50 or PVT-v2 network, the audio-visual segmentation method with integrated frequency domain design in this design has shown excellent performance, surpassing the latest SOTA method in both evaluation indicators.

[0083] Table 1 shows the comparison results of ResNet-50 and PVT-v2 visual encoders with current advanced audio-visual segmentation methods on the AVSBench dataset.

[0084] .

Claims

1. An audio-visual segmentation method integrating frequency domain design, characterized in that: Here are the steps: Step 1: Feature encoding via frequency-domain oriented audio integration module optimization In the audio encoding stage, the VGGish model is used as an audio encoder to extract audio features; in the visual encoding stage, a frequency domain-oriented audio integration module is designed, and a design combining frequency domain enhancement and early fusion is introduced in the image encoding process to generate more effective visual representations; the details are as follows: ResNet-50 or PVT-v2 is used as the visual encoder. The visual encoder contains four stages for gradually extracting features, which are represented as stage1, stage2, stage3 and tag4. ResNet-50 is based on a convolutional neural network. It gradually reduces the resolution of the feature map through residual connections and convolution operations, while increasing the number of channels. PVT-v2 uses the Transformer architecture to extract features through a self-attention mechanism and patch embedding. In each stage, global information is effectively captured through spatial downsampling and optimized calculation methods. The frequency-domain-oriented audio integration module is inserted after the output of each stage to process the feature map generated by the corresponding stage. After being processed by the frequency-domain-oriented audio integration module, the visual enhancement features obtained are used as the output of the current stage and are passed to the next stage as input. The details of each frequency-domain-oriented audio integration module are as follows: For the audio features output by the audio encoder and the spatial domain visual feature map from the current stage of the visual encoder First, the spatial domain visual features are transformed using fast Fourier transform Convert to the frequency domain: in, represents the frequency domain feature map, represents the fast Fourier transform, Represents the visual feature map The height and width index of Respectively represent the frequency domain feature maps after conversion The height and width index of the Introducing a threshold The frequency domain feature map Divided into high-frequency components and low-frequency components, as shown in the following formula: Furthermore, by introducing learnable parameters and Adaptively adjust the decoupled frequency components to generate a visual representation enhanced in the frequency domain. The process is expressed as follows: in, represents the visual representation after frequency domain enhancement, represents the inverse fast Fourier transform, which is used to map frequency domain features back to the spatial domain; Afterwards, a cross-modal attention mechanism is introduced to convert the audio features generated by the audio encoder to visual representation, and use residual connections to obtain audio-aware visual representation , as the final output of the current stage and the input of the next stage; the process is expressed in the following formula: in, Represents the cross-modal attention mechanism; Finally, the visual encoder outputs four multi-scale encoding features { },in , T represents the time of the input video frame, H and W represent the height and width of each frame of the input video respectively, Indicates the number of channels for each encoded feature; Afterwards, a pixel decoder based on a multi-scale deformable Transformer is used to perform multi-scale visual feature fusion; the details are as follows: Multi-scale encoded features extracted by the visual encoder is further fed into the pixel decoder, the multi-scale encoding feature { The visual features of the last three scales in are expanded, connected along the channel dimension, and then fused through the multi-scale deformable Transformer layer to obtain enhanced multi-scale visual features { ; Step 2: Cross-modal fusion based on frequency domain A frequency-domain-based cross-modal fusion module is designed to fully explore the audio-visual correspondence relationship. The frequency-domain-based cross-modal fusion module consists of two branches: a spatial guidance branch and a frequency-domain perception branch. The details are as follows: (1) Spatial guided branch: including spatial domain cross-modal attention operation: using cross-modal attention mechanism to perform audio-visual fusion in spatial domain, for the audio features output by the audio encoder and the multi-scale visual features output by the pixel decoder { Maximum resolution visual features in , respectively, the encoded audio features As query, spatial domain visual features Input as key and value into the cross-modal attention mechanism to obtain fused multimodal features , the process is described as follows: in, , , It maps the features to a dimension of The learnable projection matrix of the intermediate layer features; then, the multimodal features For subsequent optimization and adjustment of visual features, Through a Weighted residual connections, and encoded audio features Combined to achieve enhanced audio characteristics : (2) Frequency-domain-aware branch: The frequency-domain-aware branch works in parallel with the spatial-guided branch. First, the maximum resolution visual features in the spatial domain are transformed using the fast Fourier transform. Convert to the frequency domain to obtain the frequency domain visual feature map, and then use global average pooling to reduce the dimension of the frequency domain visual feature while retaining the global information; considering that the main structure and texture information of the frequency domain visual feature are reflected in the amplitude spectrum, and the audio feature is obtained by extracting the audio spectrum map and encoding it using the audio encoder, the frequency domain visual feature map is further extracted with spectrum features to facilitate subsequent audio-visual interaction and alignment; the process is expressed as follows: in, , , , , They represent the frequency domain visual feature map after Fourier transform, the frequency domain features after global average pooling, the global average pooling operation, The real part and The imaginary part of Finally, the frequency-domain-aware branch works in tandem with the spatially guided branch to optimize and adjust the visual representation; spectral features And fused multimodal features First, each is processed independently by a convolution layer, and then the two are deeply fused through element-by-element addition and convolution layer, so that the spectral features Multimodal features with fusion Mutual complementation and enhancement promote the effective interaction and fusion of multimodal information in the frequency domain; the process is expressed as: in, represents the convolution operation, Represents the output result, which is used to weight and adjust the original visual features; Adjustment by multiplication , also through residual connections with learnable parameter weights, to obtain enhanced visual features ; in, represents the weighted learnable parameters; Through the frequency domain-based cross-modal fusion module, the optimized audio features are finally obtained and visual features ; Step 3: Implement mask generation through decoder The Mask2Former architecture is used as the decoder to generate the sound object query embedding. The sound object query embedding is then combined with the one obtained in step 2. Combine to generate a prediction mask; Specifically, the optimized audio features obtained by the frequency domain cross-modal fusion module and the learnable query obtained by initializing the learnable parameters are combined through point-by-point addition as the query of the decoder; at the same time, the pixel decoder performs multi-scale feature fusion to output the enhanced multi-scale visual features { As keys and values, they interact with audio-based queries through a layer-by-layer cross-modal attention mechanism to generate query embeddings for sounding objects. ;at last, The classification prediction is obtained through the linear layer, The optimized maximum resolution visual features obtained by the frequency domain-based cross-modal fusion module Multiply them together to get the prediction mask; Step 4: Model training In the training process, both the category loss and the predicted mask loss are considered. In order to utilize the potential temporal coupling of audio-visual segmentation to improve the performance of the model, an adaptive inter-frame consistency loss is used. The three losses are combined to get the total loss and backpropagate to optimize the model: in, represents the true category, Represents the model prediction category, category loss function Calculated by the cross entropy function; and Represents the label of a point on the real mask and the predicted mask, respectively, and the mask loss function It consists of binary cross entropy and Dice loss; represents the prediction mask of the t-th frame, That is, adaptive inter-frame consistency loss; , , They represent weighting parameters, That is the total loss function.

Images