Audio-visual segmentation method based on implicit anti-fact learning

Abstract

The invention discloses an audiovisual segmentation method based on implicit anti-fact learning, and provides an implicit anti-fact framework to realize unbiased cross-modal understanding. Due to the lack of semantic information, heterogeneous representations may lead to mismatches, especially in complex scenes where visual content is blurred or interfered by multiple audio sources. According to the method, multi-granularity implicit texts including a video level, a fragment level and a frame level are introduced to serve as bridges for establishing a modal shared space, modal gaps are reduced, and prior guidance is provided. Visual content typically carries more information and takes the dominant position, thereby marginalizing audio features in decisions. In order to relieve knowledge preference, semantic anti-facts are provided, orthogonal representation is learned in potential space, diversified anti-fact samples are generated, and therefore deviation caused by complex function design and explicit modification of text structures or attributes is avoided. The invention further provides cooperative distribution perception contrast learning, and in combination with fact-anti-fact and cross-modal contrast alignment representation, cohesion is promoted and decoupling is realized. Wide experiments on three public data sets verify that the proposed method achieves the most advanced performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio-visual segmentation, and more specifically, relates to an audio-visual segmentation (AVS) method based on implicit counterfactual learning. Background Art

[0002] Reference video segmentation uses text or audio cues to identify matching objects in visual content and has been applied to various tasks such as embodied intelligence and autonomous driving. Unlike highly structured text, the low density and ambiguity of audio are closer to natural properties, which poses a significant challenge to audio-visual segmentation (AVS).

[0003] Audiovisual segmentation aims to segment objects in videos based on audio cues. Based on the complexity of audio and visual inputs, the AVS problem can be divided into four parts: Sound objects transition from static to highly dynamic scenes, ranging from cases that can be handled by visual segmentation models alone to those requiring strong temporal and cross-modal joint representations. As audio sources evolve from single-source to multi-source and multi-category, the need to disentangle modal representations and achieve accurate alignment becomes increasingly critical. These challenges can be summarized as follows: a) Multi-source audio and complex dynamics. The coexistence of multiple audio sources and temporal variations complicate effective association, leading to mismatches, especially in complex scenes within a frame and rapid changes between frames, resulting in low intra-modal cohesion; b) Learning bias: Models favor information-dense visual features and underestimate sparse audio, resulting in weak inter-modal coupling. Current work focuses on designing problem-specific functional components and combining them. While effective, learning optimal parameters in a fixed data space can capture spurious correlations. For example, since guitar is typically played by a person, the model may incorrectly identify a person as the speaker based on statistical regularities.

[0004] Audiovisual source localization (AVL) determines the approximate location of the sound-making object in a video by leveraging region-level audio cues, primarily through unsupervised learning to establish correlations in cross-modal representations. Recently, Zhou et al. advanced the AVL task by exploring pixel-level audiovisual understanding and introducing the audiovisual segmentation (AVS) task. Existing fully supervised AVS methods can be broadly categorized into feature decoupling and fusion methods, query generation methods, and prompt injection methods. In addition, some studies have further explored efficient label learning, 3D spatial perception, and medical scenario applications. However, these works generally overlook learning biases caused by differences in modal representation and distribution. Existing AVS methods are primarily designed to improve interaction efficiency, but pay limited attention to differences and imbalances in modal representations. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an audio-visual segmentation method based on implicit counterfactual learning to achieve unbiased cross-modal understanding, semantic guidance and balance, and improve the audio-visual segmentation performance.

[0006] To achieve the above-mentioned object of the invention, the present invention provides an audio-visual segmentation method based on implicit counterfactual learning, which is characterized by comprising the following steps:

[0007] (1) Generate multi-granularity implicit factual text

[0008] Use the visual encoder to generate multi-scale features F, whose i-th scale feature T represents the number of frames, C i Indicates the number of channels, H i Indicates height, W i Indicates width. For audio, a Mel-spectrogram is obtained through short-time Fourier transform and processed using an audio encoder to generate audio features A, whose dimensions are T x D, where D represents the feature dimension. For visual features, the basic model is used to match the text features corresponding to the video-level, segment-level, and frame-level features, i.e., implicit text, to obtain fused text features. Similar operations are performed on the audio features to generate Further forming a composite multi-granularity implicit fact text z. Specifically, it includes the following steps:

[0009] 1.1) Obtain video, clip, and frame-level features

[0010] The visual encoder is used to obtain the multi-scale features F of the input video, and then the long-distance pixel-level dependency relationship between its frames is constructed to obtain the correlation matrix According to the correlation matrix Update the multi-scale feature F to obtain the video level feature F v , divide its multi-scale feature F into multiple segments, build long-distance pixel-level dependencies between segments, and obtain the correlation matrix According to the correlation matrix Update the multi-scale feature F to obtain the segment level feature F s , based on the contextual correlation within the frame of the multi-scale feature F, a correlation matrix is ​​constructed According to the correlation matrix Update the multi-scale feature F to obtain the frame level feature F f ;

[0011] 1.2) Generate and fuse implicit text at the video, segment, and frame levels

[0012] Initialize a visually conditioned implicit text pyramid, search for the element that best matches the semantics, and generate video-level features F vThe corresponding implicit text l v , fragment-level features F s The corresponding implicit text l s , use CLIP to obtain T factual texts, and then, with the frame level feature F f Calculate the similarity and get the frame level feature F f The corresponding implicit text l f , and finally adaptively fuse implicit text l v 、l s 、l f Get fused text features

[0013] 1.3) For audio, obtain the Mel spectrum through short-time Fourier transform and use the audio encoder to encode it to generate audio features A. Use the CLAP text encoder to encode the implicit text based on audio to generate audio text features F. at ,Search for the element that best matches the semantics and generate audio features F a The corresponding implicit text is the audio fact text

[0014] 1.4) Generate multi-granularity implicit fact text z:

[0015]

[0016] in, Represents convolution, Concat represents concatenation by dimension, and Gate represents the gate mechanism;

[0017] (2) Generate counterfactual text z *

[0018] First, the factual text As initial factual text At the video level, the feature F v As a condition, gradually add Gaussian noise ∈ to the fact text in the forward process to obtain the fact text Where, T′ is the total number of steps;

[0019] Then, for the fact text z T′ After normalization and orthogonalization based on the Gram-Schmidt strategy, we get the counterfactual text z′ T′ ;

[0020] Then, for the counterfactual text z′ T′ , in the video level feature F v As a condition, the counterfactual text is gradually denoised in the backward process to obtain the counterfactual text z′0, denoted as z * , using K* Different orthogonalized control parameters, get K * counterfactual text z * , recorded as Select the K counterfactual texts with the greatest similarity to the factual text z to form the counterfactual text pool

[0021] Finally, calculate the counterfactual text generation loss

[0022]

[0023] Among them, ∈ θ (z′ t′ ,t′) represents the prediction noise of step t′, represents the expectation of all T′ steps, λ ortho represents the orthogonal loss Weight, orthogonal loss for:

[0024]

[0025] Among them, z T′ 、z′T ′ are the factual text and counterfactual text of step T′ respectively;

[0026] (3) Collaborative distributed perception contrastive learning

[0027] First, Gaussian distribution is used to model T frames to potentially aggregate sequence information and generate video-level features F v The mean μ v and covariance Σ v , and the mean μ of audio feature A a and covariance Σ a , and then calculate the video level feature F v Continuous information entropy Continuous information entropy of audio feature A

[0028]

[0029]

[0030] Among them, d v d a Represent the video level features F v , the dimension of audio feature A, det represents the determinant of the matrix;

[0031] Then, Wasserstein distance and modal entropy are used to measure the distance between distributions.

[0032] Where Tr represents the trace of the matrix and γ is the coefficient;

[0033] The same (F v ,A) is considered as a positive sample set The rest are considered as negative sample sets Using Audio Facts Text As an anchor point, the sample space of the audio feature A is divided to generate a positive audio feature set A + And the negative audio feature set A - , according to the distance The calculation formula for calculating the video level feature F v The loss between the audio feature A

[0034]

[0035] in, Represents the positive sample set Zhongzheng audio feature set A + The g-th audio feature, Represents the negative sample set Medium negative audio feature set A - The h-th audio feature of

[0036] Will Defined as the negative sample set in, Counterfactual text pool The kth counterfactual text z in * , according to the distance The calculation formula for calculating the video level feature F v The loss between the fact text z

[0037]

[0038] in, Represents the negative sample set The kth counterfactual text in ω k is the weight coefficient of k counterfactual texts, which is determined according to the difficulty of the counterfactual texts;

[0039] Similarly, Defined as the negative sample set Calculate the loss between audio feature A and fact text z

[0040]

[0041] in, Represents the negative sample set The kth counterfactual text in .

[0042] Finally, the co-distribution-aware contrastive learning loss is calculated

[0043]

[0044] (4) Calculate the mask segmentation loss

[0045] The multi-scale feature F and the audio feature A are input into the pixel decoder to obtain the segmentation image, and then the mask segmentation loss is calculated with the real segmentation image.

[0046] (5) Process the image X in the training dataset according to steps (1) to (4) and calculate the total loss

[0047]

[0048] Among them, λ cf ,λ CDCL Counterfactual text generation loss Co-distribution-aware contrastive learning loss The weight of

[0049] Based on total losses Training the audio-visual segmentation system constructed according to steps (1) to (4);

[0050] (6) Audiovisual segmentation

[0051] The video to be segmented is sent to the audio-visual segmentation system and processed according to step (1) to obtain the multi-scale feature F and the audio feature A, which are input into the pixel decoder to obtain the segmented image and complete the audio-visual segmentation.

[0052] The object of the present invention is achieved in this way.

[0053] The present invention proposes an implicit counterfactual framework (ICF) based on the audio-visual segmentation method of implicit counterfactual learning to achieve unbiased cross-modal understanding. Due to the lack of semantic information, heterogeneous representations may lead to mismatching, especially in complex scenes where the visual content is blurred or interfered with by multiple audio sources. The present invention introduces multi-granularity implicit text (MIT), including video level, segment level and frame level, as a bridge to establish a modal shared space, reduce modal gaps and provide prior guidance. Visual content usually carries more information and dominates, thereby marginalizing audio features in decision-making. In order to alleviate knowledge bias, the present invention proposes semantic counterfactual (SC), which learns orthogonal representations in the latent space and generates diverse counterfactual samples, thereby avoiding biases introduced by complex function design and explicit modification of text structure or attributes. This paper further proposes Collaborative Distribution-aware Contrastive Learning (CDCL), which combines fact-counterfactual and cross-modal contrastive alignment representations to promote cohesion and achieve disentanglement. Extensive experiments on three public datasets validate that the proposed method achieves state-of-the-art performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a general structural diagram of a specific embodiment of the audio-visual segmentation method based on implicit counterfactual learning of the present invention;

[0055] Figure 2 It is a flowchart of a specific embodiment of the audio-visual segmentation method based on implicit counterfactual learning of the present invention;

[0056] Figure 3 is a qualitative comparison diagram on audio-visual segmentation (AVS). DETAILED DESCRIPTION

[0057] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0058] Figure 1 This is an overall structural diagram of a specific implementation of the audio-visual segmentation method based on implicit counterfactual learning of the present invention.

[0059] like Figure 1As shown in the figure, for any given audio-video pair, a visual encoder and an audio encoder are used to obtain high-dimensional features respectively. For visual features, global temporal correlation, segment semantic partitioning, and intra-frame context alignment are established at the video-segment-frame level. Subsequently, a visually conditioned implicit text pyramid is initialized to search for the elements that best match the semantics and perform adaptive fusion. For audio features, similar operations are performed using CLAP. Through gated filtering, we orthogonalize the factual text in the diffusion latent space to generate a pool of counterfactual texts. Modal features are converted into a distribution form and contrastive learning is constructed, with query learning serving as the basis for subsequent decoding.

[0060] Inspired by the human brain's ability to self-project and simulate non-real scenarios, e.g. Figure 1 As shown in the figure, this paper proposes the Implicit Counterfactual Framework (ICF), which uses text to bridge visual and audio modalities, constructs counterfactual text samples, and establishes a contrastive strategy based on representation distribution. This leads to three key questions: 1) Why use language as a bridge instead of direct interaction? 2) Why construct implicit counterfactual samples? 3) Why build contrastive learning based on representation distribution?

[0061] We answer the first question. In complex scenes, multiple visual objects may sound synchronously or asynchronously (many-to-many), or one sound may correspond to multiple visual objects (one-to-many), which makes direct matching prone to ambiguity. In addition, the time delay between visual content and audio may exacerbate the problem. Our goal is to leverage the semantic information of text to capture objects, actions, and their relationships in the scene, promote cross-modal alignment, and model inter-frame context continuity (challenge a). Unlike methods that generate explicit text or categories, we match the best implicit text for vision and audio at the feature level. This approach is based on the following two points: a) implicit text comes from a high-order space, is not easily affected by low-level noise, and is more suitable for constructing counterfactual samples; b) unlike visual semantics that usually contain silent objects, audio semantics are inherently localized and focused.

[0062] We address the second question. To alleviate knowledge gaps in data structures, some work modifies explicit text attributes such as nouns, colors, sizes, or spatial orientations, or uses generative models to create counterfactual images or adjust audio spectra (challenge b). However, explicit strategies typically require selecting the elements to modify from a pool of predefined hypotheses, whose upper bound on likelihood may be smaller than implicit strategies. Our approach also ensures that the representations of factual and counterfactual text gradually decouple, facilitating curriculum-based model training. Furthermore, explicit strategies incur significant storage and training costs and are difficult to implement in an end-to-end manner.

[0063] We address the third question. Previous audiovisual work constructs contrastive learning at the feature level, which is sensitive to variations in its internal structure. In contrast, our strategy focuses on: a) leveraging statistical distributions to model contextual features, enhancing robustness to scene changes, audio noise / mixture, and implicit text inaccuracies, thereby mitigating mismatches caused by hard contrastive learning; and b) maintaining internal compactness while maintaining stable alignment of visual and audio features with the factual text, preventing performance degradation caused by counterfactual text. Technically, we propose Multi-Grained Implicit Text (MIT), which feeds video-level, segment-level, and frame-level visual features into a base model to retrieve the best-matching implicit text representation. Similar principles apply to audio features. We further introduce Semantic Counterfactuals (SC), which leverages a latent diffusion model to establish continuous and controllable intra- and inter-sample orthogonality in the noisy space during the forward pass and generates counterfactual samples through a denoising process. Finally, we propose Collaborative Distribution-Aware Contrastive Learning (CDCL), which transforms modality features into Gaussian distributions and constructs contrasts in a joint embedding space based on an auxiliary entropy (uncertainty) measure.

[0064] Figure 2 It is a flowchart of a specific implementation of the audio-visual segmentation method based on implicit counterfactual learning of the present invention.

[0065] The present invention aims to achieve semantic guidance and representation balance through implicit text and counterfactual samples. Specifically, Figure 2 As shown, the audio-visual segmentation method based on implicit counterfactual learning of the present invention includes the following steps:

[0066] Step S1: Generate multi-granular implicit fact text

[0067] In this embodiment, a visual encoder is used to generate multi-scale features F, where the i-th scale feature T represents the number of frames, C i Indicates the number of channels, H i Indicates height, W i Indicates width. For audio, a Mel-spectrogram is obtained through short-time Fourier transform and processed using an audio encoder to generate audio features A, whose dimensions are T x D, where D represents the feature dimension. For visual features, the basic model is used to match the text features corresponding to the video-level, segment-level, and frame-level features, i.e., implicit text, to obtain fused text features. Similar operations are performed on the audio features to generate Further forming a composite multi-granularity implicit fact text z. Specifically, it includes the following steps:

[0068] Step S1.1: Obtain video, segment, and frame-level features

[0069] Inspired by the way humans associate visual content with auditory cues, which usually relies on the following three points: 1) global scene perception (video level); 2) object motion and semantic changes between frames (segment level); 3) current moment context (frame level). In this paper, the multi-scale features F of the input video are obtained using a visual encoder, and then long-range pixel-level dependencies are constructed between its frames to obtain the correlation matrix

[0070] Specifically, in this embodiment, three matrices are used to map the multi-scale feature F to the query Q v , key K v Sum V v , thus obtaining the correlation matrix

[0071]

[0072] Among them, τ and Represent the temperature coefficient and transpose respectively. In order to reduce the computational cost, we establish the spatiotemporal correlation in the channel dimension instead of the spatial dimension (the complexity is reduced from Reduce to We have further updated it to process data more efficiently and reduce computational burden, which significantly improves computational efficiency and resource utilization while maintaining model performance.

[0073] According to the correlation matrix Update the multi-scale feature F to obtain the video level feature F v :

[0074]

[0075] For segment-level features, we adopt a similar sampling strategy. The difference is that the time window is set to or And slide backward. When the number of frames in the window is insufficient, we use the previous frames to supplement. Divide its multi-scale feature F into multiple segments, build long-distance pixel-level dependencies between the segments, and obtain the correlation matrix According to the correlation matrix Update the multi-scale feature F to obtain the segment level feature F s .

[0076] Based on the contextual correlation within the frame of the multi-scale feature F, a correlation matrix is ​​constructed According to the correlation matrix Update the multi-scale feature F to obtain the frame level feature F f .

[0077] Step S1.2: Generate and fuse video, segment, and frame-level implicit text

[0078] Initialize a visually conditioned implicit text pyramid, search for the element that best matches the semantics, and generate video-level features F v The corresponding implicit text l v , fragment-level features F s The corresponding implicit text l s , use CLIP to obtain T factual texts, and then, with the frame level feature F f Calculate the similarity and get the frame level feature F f The corresponding implicit text l f , and finally adaptively fuse implicit text l v 、l s 、l f Get fused text features

[0079] In this invention, based on the text inversion method, VideoCLIP is used to generate the video level feature F v The corresponding implicit text l v :

[0080]

[0081] Among them, we introduced k t learnable parameters l k′ , to capture multiple visual concepts, thus avoiding focusing only on the most salient semantics. For the segment-level feature F s , and the same is true. For each frame level feature F f , we use CLIP to obtain T factual texts. Therefore, we can obtain semantic representations from coarse-grained to fine-grained. In order to measure the contribution of each element, we introduce a weight query w to dynamically adjust the weight to form a fused text feature

[0082]

[0083] Among them, N p Indicates the actual amount of text corresponding to each scale.

[0084] Step S1.3: For audio, obtain the Mel spectrum through short-time Fourier transform and encode it using the audio encoder to generate audio features A. Use the CLAP text encoder to encode the audio-based implicit text to generate audio text features F. at ,Search for the element that best matches the semantics and generate audio features F a The corresponding implicit text is the audio fact text

[0085] We introduce a gating mechanism (three MLP layers) to alleviate semantic conflicts and further generate composite fact text z.

[0086] Step S1.4: Generate multi-granular implicit fact text z:

[0087]

[0088] in, Represents convolution, Concat represents concatenation by dimension, and Gate represents the gate mechanism.

[0089] At the feature level, counterfactual samples can be generated by directly adding noise or applying transformations. This strategy may disturb the original semantic distribution and is difficult to control (as shown in the quantitative ablation study of the proposed components in the table). We exploit the stable sample construction ability of the diffusion model and impose orthogonality in the latent space. To avoid confusion in learning direction, we condition on the video-level feature F v In the case of , Gaussian noise is gradually added to z in the forward process. This ensures that the generated counterfactual samples maintain a certain degree of controllability while effectively exploring different possibilities of the data, thereby improving the robustness and generalization ability of the model.

[0090] Step S2: Generate counterfactual text z *

[0091] Based on the ladder theory of causal inference, previous AVS work has focused on constructing biased modal associations (i.e., "if..., then..."). Due to the interference of confounding factors, a strong statistical correlation between two variables does not necessarily imply a causal relationship. Intervention and counterfactuals are effective methods to achieve impartiality and have been applied in various scenarios such as visual question answering, image segmentation, and visual language navigation. Exploring unexperienced situations through interventions is a significant challenge. We instead use counterfactual reasoning (i.e., "if not..., then...") to construct a continuous hypothesis space, thereby expanding the potential boundaries of samples and establishing unbiased and accurate correlations.

[0092] First, the fact text z is taken as the initial fact text z0, and the video level feature F v As a condition, gradually add Gaussian noise ∈ to the fact text in the forward process to obtain the fact text z T′ , where T′ is the total number of steps. The process of adding noise is as follows:

[0093]

[0094] Where β and I represent the variance scheduling parameter and the identity matrix, respectively. Represents a Gaussian distribution.

[0095] Then, for the fact text z T′ After normalization and orthogonalization based on the Gram-Schmidt strategy, we get the counterfactual text z′ T′ .

[0096] Then, for the counterfactual text z′ T′ , in the video level feature F v As a condition, the counterfactual text is gradually denoised in the backward process to obtain the counterfactual text z′0, denoted as z * , using K * Different orthogonalized control parameters, get K * counterfactual text z * , recorded as Select the K counterfactual texts with the greatest similarity to the factual text z to form the counterfactual text pool

[0097] Finally, calculate the counterfactual text generation loss

[0098]

[0099] Among them, ∈ θ (z′ t′ ,t′) represents the prediction noise of step t′, represents the expectation of all T′ steps, λ orth represents the orthogonal loss Weight, orthogonal loss for:

[0100]

[0101] Among them, z T′ 、z′ T′ are the factual text and counterfactual text of step T′ respectively.

[0102] Step S3: Collaborative Distribution-Aware Contrastive Learning

[0103] Contrastive learning aims to minimize the distance between paired samples or categories while maximizing the distance between unmatched samples or different categories. Some studies use semantic categories (cues) as anchors to construct positive and negative sample pairs to promote representation disentanglement. In contrast, other studies focus on audio features. Our strategy differs in two aspects: 1) measuring modal differences from representation distributions; 2) establishing heterogeneous trimodal contrastive learning.

[0104] Feature-level contrastive learning is very sensitive to anomalies, such as blank video frames, mixed scenes, and drastic changes in the audio spectrum, which can lead to errors in sample attraction and repulsion. This problem is further exacerbated when the implicit textual semantic description of the video or audio content is inaccurate. To address this issue, we convert features into statistical probabilities to enhance tolerance to anomalies. Specifically, we use a Gaussian distribution to model T frames to potentially aggregate sequence information and generate a mean μ v and covariance Σ v .

[0105]

[0106] First, Gaussian distribution is used to model T frames to potentially aggregate sequence information and generate video-level features F v The mean μ v and covariance Σ v , and the mean μ of audio feature A a and covariance Σ a .

[0107] The introduction of low-quality data increases aleatoric uncertainty. Previous work has paid little attention to this issue and has treated features from different modalities equally. While we mitigate this issue through distribution modeling, accurate quantification remains crucial. We utilize continuous information entropy H to reflect average uncertainty.

[0108] Then calculate the video level feature F v Continuous information entropy Continuous information entropy of audio feature A

[0109]

[0110] Among them, d v d a Represent the video level features F v , the dimension of the audio feature A, det represents the determinant of the matrix.

[0111] Then, Wasserstein distance and modal entropy are used to measure the distance between distributions.

[0112] where Tr is the trace of the matrix and γ is the coefficient. The latter term measures the degree of looseness (distance) within the distribution, and low entropy (i.e., low uncertainty) of the joint modal distribution indicates high cohesion.

[0113] The same (F v ,A) is considered as a positive sample set The rest are considered as negative sample sets Using Audio Facts Text As an anchor point, the sample space of the audio feature A is divided to generate a positive audio feature set A + And the negative audio feature set A - , according to the distance The calculation formula for calculating the video level feature F v The loss between the audio feature A

[0114]

[0115] in, Represents the positive sample set Zhongzheng audio feature set A + The g-th audio feature, Represents the negative sample set Medium negative audio feature set A - The h-th audio feature of .

[0116] Will Defined as the negative sample set in, Counterfactual text pool The kth counterfactual text z in * , according to the distance The calculation formula for calculating the video level feature F v The loss between the fact text z

[0117]

[0118] in, Represents the negative sample set The kth counterfactual text in ω k is the weight coefficient of k counterfactual texts, which is determined according to the difficulty of the counterfactual texts.

[0119] Similarly, Defined as the negative sample set Calculate the loss between audio feature A and fact text z

[0120]

[0121] in, Represents the negative sample set The kth counterfactual text in .

[0122] Finally, the co-distribution-aware contrastive learning loss is calculated

[0123]

[0124] Step S4: Calculate the mask segmentation loss

[0125] The multi-scale feature F and the audio feature A are input into the pixel decoder to obtain the segmentation image, and then the mask segmentation loss is calculated with the real segmentation image.

[0126] Step S5: Process the image X in the training dataset according to steps S1 to S4 and calculate the total loss

[0127]

[0128] Among them, λ cf ,λ CDCL Counterfactual text generation loss Co-distribution-aware contrastive learning loss The weight of .

[0129] Based on total losses Training the audio-visual segmentation system constructed according to steps S1 to S4;

[0130] Step S6: Audiovisual Segmentation

[0131] The video to be segmented is sent to the audio-visual segmentation system and processed according to step (1) to obtain the multi-scale feature F and the audio feature A, which are input into the pixel decoder to obtain the segmented image and complete the audio-visual segmentation.

[0132] The main contributions of this paper are as follows: We model the learning biases in AVS tasks based on causal inference theory and introduce a method for generating counterfactual examples from implicit text, achieving semantic guidance and balance. We propose MIT and SC to establish semantic correlation and differentiation in continuous space, and further propose CDCL to disentangle and condense modal feature distributions. Extensive experiments on three public datasets demonstrate that our method achieves state-of-the-art results and can be seamlessly integrated into other AVS methods, achieving a 3%-4% performance improvement.

[0133] experiment

[0134] Datasets: We conducted experiments on two dataset benchmarks, AVS-Object and AVS-Semantic (AVSS). AVS-Object includes two sub-datasets: single source segmentation (S4) and multi-source segmentation (M3). Each data has 5 sample frames. The training / validation / test sample sizes for single source segmentation (S4) and multi-source segmentation (M3) are 3450 / 740 / 740 and 296 / 64 / 64 respectively. Unlike AVS-Object which only provides binary masks, AVSS further provides classification labels and extends to ten sample frames, including 12356 (8498 / 1304 / 1554) videos, covering 70 categories.

[0135] Implementation details:

[0136] We use the PyTorch toolbox to implement the proposed algorithm on NVIDIA A100 GPU. All video frames are resized to 224 × 224. For the training phase, we use AdamW

[0137] We use the AdamW optimizer to update parameters, with a batch size of 8, an initial learning rate of le-4, and a total of 60 training epochs. During training, we do not use any post-processing corrections, except for test-time augmentation. For fair comparison, we use the ResNet-50 and PVT-v2 models pre-trained on ImageNet as the visual backbone network, and the VGGish model pre-trained on YouTube-8M as the audio backbone network. To generate implicit orthogonal text, we use Latent Diffusion to reduce computational complexity. We use the Multi-Scale Deformable Attention Transformer as the pixel decoder.

[0138] Evaluation metric: We use the Jaccard coefficient F-score and mean As an evaluation indicator. The higher the evaluation indicator, the better the model performance.

[0139] 1. Quantitative comparison

[0140]

[0141] Table 1

[0142] Table 1 shows the Jaccard coefficients on datasets S4, M3, and AVSS. F-score and mean Quantitative comparison of external models such as SAM. ↑ means the bigger the better, the best performance is bolded, and the second best performance is underlined. Indicates that no data is available.

[0143] In Table 1, we classify existing AVS into CNN-based and Transformer-based (partially overlapping) audio-visual segmentation methods based on visual backbone networks such as ResNet-50 or PVT-v2. Our method has an average performance of They surpass the second-best model, COMBO, by 1.77%, 4.69%, and 4.06%, respectively. However, their superiority is less pronounced on dataset S4. Our analysis reveals that visual scene representations from single-source audio are relatively simple and pure, requiring less discriminative power and representational spatial scope. In contrast, in complex visual scenes with multi-source audio, a lack of semantics can lead to catastrophic misalignment of audio and video content. We focus on comparing and analyzing two approaches that employ text. BAVS incorporates object classification information, whereas our implicit text provides a broader semantic understanding of the video (including bokeh and background) without prior knowledge. Compared to TeSO, which directly uses scene descriptions and relies on offline advanced LLMs (such as LLaMA2) to detect object perception, each step yielding incomplete and suboptimal results. We perform direct matching and fusion in the feature space, mitigating mutual interference and noise introduction through end-to-end learning. Furthermore, DiffusionAVS constructs a latent diffusion model conditioned on audio and introduces pairwise comparison learning. Our method outperforms DiffusionAVS by an average of 4.82%. We can attribute this performance gap to: 1) the inherent uncertainty (instability) of audio, such as semantic deviations caused by multiple factors (such as noise interference); and 2) insufficient disentanglement due to the lack of high-precision reference and sample separation.

[0144] 2. Qualitative comparison

[0145] exist Figure 3 In this paper, we provide visual comparison results under various settings. From left to right, the samples are from datasets S4, M3, and AVSS. On the left, the snow on the coyote creates camouflage, making complete segmentation difficult. In the middle, the image shows rapid movement and continuous noise interference (birdsong). On the right, the subtitle background bars affect multiple audio and visual categories.

[0146] As the complexity of sound and vision increases, other methods produce false negatives or false positives. Our method remains effective in establishing robust cross-modal correlations and capturing transient changes, enabling accurate segmentation of sound-generating objects and determining their categories.

[0147] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. An audio-visual segmentation method based on implicit counterfactual learning, characterized in that The following steps are involved: (1) Generate multi-granularity implicit factual text Use the visual encoder to generate multi-scale features F, whose i-th scale feature T represents the number of frames, C i Indicates the number of channels, H i Indicates height, W i Indicates width. For audio, a Mel-spectrogram is obtained through short-time Fourier transform and processed using an audio encoder to generate audio features A, whose dimensions are Tx D, where D represents the feature dimension. For visual features, the basic model is used to match the text features corresponding to the video-level, segment-level, and frame-level features, namely the implicit text, to obtain the fused text feature l v . Similar operations are performed on the audio features to generate l a , further forming a composite multi-granularity implicit fact text z; (2) Generate counterfactual text z * First, the fact text z is taken as the initial fact text z0, and the video level feature F v As a condition, gradually add Gaussian noise ∈ to the fact text in the forward process to obtain the fact text z T′ , where T ′ is the total number of steps; Then, for the fact text z T′ After normalization and orthogonalization based on the Gram-Schmidt strategy, we get the counterfactual text z ′ T ′ ; Then, for the counterfactual text z′ T′ , under the condition of video level feature Fv, the counterfactual text is denoised step by step in the backward process to obtain the counterfactual text z′0, denoted as z * , using K * Different orthogonalized control parameters, get K * counterfactual text z * , recorded as k=1,2,…,K * , select the K counterfactual texts with the greatest similarity to the factual text z to form the counterfactual text pool l cf ; Finally, calculate the counterfactual text generation loss Among them, ∈ θ (z′ t′ ,t′) represents the prediction noise of step t′, represents the expectation of all T′ steps, λ ortho represents the orthogonal loss Weight, orthogonal loss for: Among them, z T′ 、z′T ′ are the factual text and counterfactual text of step T′ respectively; (3) Collaborative distributed perception contrastive learning First, Gaussian distribution is used to model T frames to potentially aggregate sequence information and generate video-level features F v The mean μ v and covariance Σ v , and the mean μ of audio feature A a and covariance Σ a , and then calculate the video level feature F v Continuous information entropy Continuous information entropy of audio feature A Among them, d v d a Represent the video level features F v , the dimension of audio feature A, det represents the determinant of the matrix; Then, Wasserstein distance and modal entropy are used to measure the distance between distributions. Where Tr represents the trace of the matrix and γ is the coefficient; The same (F v ,A) is considered as a positive sample set The rest are considered as negative sample sets Use audio fact text a As an anchor point, the sample space of the audio feature A is divided to generate a positive audio feature set A + And the negative audio feature set A - , according to the distance The calculation formula for calculating the video level feature F v The loss between the audio feature A in, Represents the positive sample set Zhongzheng audio feature set A + The g-th audio feature, Represents the negative sample set Medium negative audio feature set A - The h-th audio feature of Will Defined as the negative sample set in, For the counterfactual text pool e cf The kth counterfactual text z in * , according to the distance The calculation formula for calculating the video level feature F v The loss between the fact text z in, Represents the negative sample set The kth counterfactual text in ω k is the weight coefficient of k counterfactual texts, which is determined according to the difficulty of the counterfactual texts; Similarly, Defined as the negative sample set Calculate the loss between audio feature A and fact text z in, Represents the negative sample set The kth counterfactual text in . Finally, the co-distribution-aware contrastive learning loss is calculated (4) Calculate the mask segmentation loss The multi-scale feature F and the audio feature A are input into the pixel decoder to obtain the segmentation image, and then the mask segmentation loss is calculated with the real segmentation image. (5) Process the image X in the training dataset according to steps (1) to (4) and calculate the total loss Among them, λ cf ,λ CDCL Counterfactual text generation loss Co-distribution-aware contrastive learning loss The weight of Based on total losses Training the audio-visual segmentation system constructed according to steps (1) to (4); (6) Audiovisual segmentation The video to be segmented is sent to the audio-visual segmentation system and processed according to step (1) to obtain the multi-scale feature F and the audio feature A, which are input into the pixel decoder to obtain the segmented image and complete the audio-visual segmentation.

2. The audio-visual segmentation method based on implicit counterfactual learning according to claim 1, characterized in that Generating multi-granularity implicit fact text includes the following steps: 1.1) Obtain video, clip, and frame-level features The visual encoder is used to obtain the multi-scale features F of the input video, and then the long-distance pixel-level dependency relationship is constructed between its frames to obtain the correlation matrix According to the correlation matrix Update the multi-scale feature F to obtain the video level feature F v , divide its multi-scale feature F into multiple segments, build long-distance pixel-level dependencies between segments, and obtain the correlation matrix According to the correlation matrix Update the multi-scale feature F to obtain the segment level feature F s , based on the contextual correlation within the frame of the multi-scale feature F, a correlation matrix is ​​constructed According to the correlation matrix Update the multi-scale feature F to obtain the frame level feature F f ; 1.2) Generate and fuse implicit text at the video, segment, and frame levels Initialize a visually conditioned implicit text pyramid, search for the element that best matches the semantics, and generate video-level features F v The corresponding implicit text l v , fragment-level features F s The corresponding implicit text l s , use CLIP to obtain T factual texts, and then, with the frame level feature F f Calculate the similarity and get the frame level feature F f The corresponding implicit text l f , and finally adaptively fuse implicit text l v 、l s 、l f Get the fused text feature l v ; 1.3) For audio, obtain the Mel spectrum through short-time Fourier transform and use the audio encoder to encode it to generate audio features A. Use the CLAP text encoder to encode the implicit text based on audio to generate audio text features F. at ,Search for the element that best matches the semantics and generate audio features F a The corresponding implicit text is the audio fact text l a ; 1.4) Generate multi-granularity implicit fact text z: in, Represents convolution, Concat represents concatenation by dimension, and Gate represents the gate mechanism.

3. The audio-visual segmentation method based on implicit counterfactual learning according to claim 2, characterized in that Use three matrices to map multi-scale features F to query Q v , key K v Sum V v , thus obtaining the correlation matrix where τ and T represent the temperature coefficient and transpose, respectively. According to the correlation matrix Update the multi-scale feature F to obtain the video level feature F v :

4. The audio-visual segmentation method based on implicit counterfactual learning according to claim 2, characterized in that Based on the text reversal method, VideoCLIP is used to generate the video level feature F v The corresponding implicit text l v : Among them, we introduced k t learnable parameters l k′ .

Images