An open-vocabulary audiovisual segmentation method based on semantic consistency

Abstract

The application belongs to the technical field of artificial intelligence and multi-modal information processing, and discloses an open vocabulary audio-visual segmentation method based on semantic consistency. An audio semantic enhancement module is designed to explicitly enhance the audio semantic discrimination ability, improve the cross-modal alignment and semantic recognition accuracy of the model, and enhance the robustness and precision of audio-visual semantic segmentation. A symmetrical cross-modal attention guide module and a hierarchical modal fusion decoder are proposed. Through fine cross-modal interaction and multi-modal decoding, the temporal and spatial semantics in audio-visual information are fully mined, the audio-visual features are aggregated in the spatial and temporal dimensions, and the accurate positioning and classification of the sound object are ensured. By jointly using CLIP and CLAP and aligning audio-visual features based on shared real labels, the application not only enhances the segmentation performance of known category sound objects, but also significantly improves the segmentation and classification ability of unknown categories and the generalization ability of the model in the open vocabulary scene through the knowledge of the pre-trained basic model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal (audiovisual) information processing technology, and relates to an open vocabulary audiovisual segmentation method based on semantic consistency. Background Technology

[0002] With the development of multimedia information processing technology, audio-visual segmentation (AVS) technology has emerged, aiming to accurately separate visually emitting objects from video and audio data. This technology has broad application prospects in virtual reality, security monitoring, human-computer interaction, and many other fields. Existing AVS segmentation methods mainly focus on recognizing a limited number of predefined categories in the training data, lacking generalization ability for novel, unseen categories. This limits its application in open-world scenarios and the identification of unknown sound sources, affecting the robustness and flexibility of AVS segmentation systems. To address these challenges, scholars recently proposed a new task: Open-Vocabulary Audio-Visual Semantic Segmentation (OVAVSS), formally defining the problem and constructing relevant evaluation benchmarks. OVAVSS aims to segment and identify corresponding sound-emitting objects based on given video and audio information, including entirely new categories of sound-emitting objects not encountered during training. However, OV-AVSS employs video-level decoding and category-independent mask generation strategies, which present challenges in temporal alignment and fail to explicitly align audiovisual semantics, making it difficult to effectively handle issues such as overlapping multi-source sounds in complex scenes. Furthermore, its over-reliance on pre-trained visual-language models makes it prone to semantic drift and misclassification when audio signals are blurry or semantically ambiguous. In addition, the two-stage audiovisual semantic segmentation process—segmentation followed by classification—weakens cross-modal consistency, affecting the inherent correlation and alignment of overall audiovisual semantics.

[0003] Therefore, this invention addresses the shortcomings and challenges of existing research by innovatively introducing a strategy that bridges the visual-language model (CLIP) and the audio-language model (CLAP). It fully leverages the pre-trained knowledge and synergistic effects of both models to effectively solve semantic drift and misclassification problems, thereby improving audiovisual consistency. By deeply mining audio cues, it achieves accurate identification and classification of sound-producing objects, significantly enhancing the deep fusion and semantic alignment of audiovisual modalities, thus realizing open-vocabulary audiovisual segmentation with strong generalization capabilities. Summary of the Invention

[0004] This invention proposes an open-vocabulary audiovisual segmentation method based on semantic consistency, thereby achieving robust open-vocabulary audiovisual semantic segmentation. This method effectively improves the semantic consistency between audiovisual modalities, enhances the ability to recognize and segment sound-producing objects, and significantly improves the model's generalization performance.

[0005] The technical solution of the present invention:

[0006] An open-lexical audiovisual segmentation method based on semantic consistency is as follows:

[0007] (1) The specific implementation steps of the symmetrical cross-modal attention guidance module are as follows:

[0008] Audio features encoded by the CLAP audio encoder Multiscale visual features encoded by CLIP image encoder First, multi-scale visual features are projected onto a unified feature dimension through convolution. Then, it is expanded along the spatial dimension and merged into multi-scale features, resulting in Simultaneously, audio features are mapped to the same feature dimension through a multilayer perceptron, denoted as . Then use a cross-modal attention mechanism for fusion. and The fused multimodal features are obtained, denoted as . :

[0009]

[0010] in, Represents row-by-row normalization Activation function Represents the learnable projection matrix;

[0011] To fully capture complex audiovisual relationships, a two-layer cross-modal interaction layer is designed, based on visual features. and audio features For queries, all data are derived from the fused multimodal features. As a key or value, through a cross-modal attention mechanism Selective injection of fused information into visual features and audio features The enhanced visual and audio features are obtained respectively, guiding them to more accurately focus on the sound-producing object; the corresponding calculation formulas are as follows:

[0012]

[0013]

[0014] in, and These represent the enhanced visual and audio features, respectively. , All are learnable projection matrices;

[0015] Subsequently, The input is fed into a pixel decoder based on multi-scale deformable attention for multi-scale feature interaction, outputting more accurate visual features, denoted as... ;

[0016] (2) The hierarchical modal fusion decoder, the specific implementation steps of which are as follows:

[0017] First, the enhanced audio features are temporally unfolded, and then optimized using a self-attention mechanism and temporal context association to obtain temporally enhanced audio features, denoted as . Then, a set of target queries initialized with learnable parameters are processed through cross-modal attention and temporally enhanced audio features. Interact to obtain the initial target query with embedded audio clues. ;

[0018] The hierarchical modality fusion decoder consists of three decoding layers. In each decoding layer, the target query is combined with the visual features output by the pixel decoder. Perform cross-modal attention interaction to extract visual information of objects corresponding to vocal cues, and obtain an updated target query. :

[0019]

[0020] in, ; This indicates the target query from the output of the previous decoding layer. When the decoding layer is the first layer, This represents a cross-modal attention mechanism;

[0021] Subsequently, the updated target query undergoes further fine-grained association modeling through a self-attention layer to enhance its discriminative ability. To preserve audio cues and improve the recognition capability in complex multi-source scenes, the updated target query further interacts with audio features to obtain a target query integrating audio information. :

[0022]

[0023] Next, the target query is explicitly modeled by performing self-attention computation on the target query that integrates audio information in the spatiotemporal dimensions. This cross-frame association further enhances the temporal consistency of the target query:

[0024]

[0025] in, This represents the self-attention mechanism; Indicates that the shape is Tensor unfolds into shape The target query, The number of target queries, Indicates the channel dimension; This represents the reverse operation of restoring the result of the above unfolding operation to its original shape; finally, it involves performing a target query to enhance temporal consistency. After processing by the feedforward network, the target query that integrates audiovisual information is output, denoted as... ;

[0026] (3) Audio semantic enhancement module, the specific implementation steps of which are as follows:

[0027] First, target query Audio features with temporal enhancement Interacting through cross-modal attention allows each target query to selectively aggregate relevant audio information and explicitly fuse audio semantics; subsequently, residual connections are applied to the attention output, and the fused features are further integrated through a multilayer perceptron. The CLAP text hint template "This sound contains the {}" is used for the category, and the embedded text is obtained through the CLAP text encoder. Finally, fusion features and text embedding The final audio semantic classification result is obtained by calculating similarity through dot product. The above process can be represented as:

[0028]

[0029]

[0030] To fully utilize the pre-trained knowledge of CLAP, enhance the audio semantic discrimination ability, and improve the ability of the open-vocabulary audiovisual segmentation model to accurately identify the sounding object using audio, the audio semantic enhancement module introduces an auxiliary audio-supervised semantic classification loss. The loss formula is expressed as:

[0031]

[0032] in, This indicates the calculation of cross-entropy loss. This indicates the actual category label corresponding to the audio.

[0033] (4) Model mask generation and overall loss calculation, the specific steps are as follows:

[0034] To achieve open-lexical audiovisual semantic segmentation, a CLIP-based classification head and a masking mechanism are used for audiovisual semantic segmentation; in the masking mechanism, the target query... First, after processing by a multilayer perceptron, it is compared with the highest resolution visual features output by a pixel-level decoder. The mask features are used to generate the corresponding segmentation mask prediction through matrix multiplication. In the CLIP-based classification header, segmentation mask prediction and mask features are obtained through mask pooling to obtain mask pooled features, which are then combined with the target query. Residual connections are performed, and category features are obtained after processing through a multilayer perceptron; simultaneously, text embedding is performed. The category was generated using the CLIP text hint template "A photo of {}" via a CLIP text encoder; finally, the category features were embedded with the text. Similarity is calculated using dot products to generate corresponding category predictions. ;

[0035] During training, this design considers both semantic classification loss and prediction mask loss; the semantic classification loss consists of two parts: one is the CLIP-based main classification loss from the classification head. Secondly, the CLAP-based auxiliary audio supervised classification loss introduced by the audio semantic enhancement module. The total losses are shown below:

[0036]

[0037]

[0038] Among them, the mask loss function From the prediction mask and the real mask The focus and Diceloss are calculated. , , These represent the weighting parameters, That is, the total loss function;

[0039] During the inference phase, the segmentation prediction mask generated by the masking tool is used. Another set of semantic classification predictions is obtained by calculating similarity between masked pooling and text embeddings containing unknown categories encoded using the frozen CLIP text encoder. Among them, the output of the classification header is Responsible for classifying and recognizing known categories. The focus is on identifying unknown categories; subsequently, and By fusing through geometric integration, a final semantic category prediction is generated that combines the ability to distinguish known categories with the ability to generalize from open vocabularies. Finally, this semantic category prediction is combined with the prediction mask to generate an audiovisual semantic segmentation mask.

[0040] The beneficial effects of this invention are:

[0041] (1) This invention enhances the robustness and accuracy of audio-visual semantic segmentation by designing a novel audio semantic enhancement module to explicitly enhance the audio semantic discrimination capability and improve the cross-modal alignment and semantic recognition accuracy of the model.

[0042] (2) Two innovative cross-modal fusion interaction modules were proposed: a symmetrical cross-modal attention guidance module and a hierarchical modal fusion decoder. Through refined cross-modal interaction and multimodal decoding, the spatiotemporal semantics in audiovisual information were fully explored, promoting the aggregation of audiovisual features in the spatial and temporal dimensions, and effectively ensuring the accurate positioning and classification of sound-producing objects.

[0043] (3) By using CLIP and CLAP together and aligning audiovisual features based on shared real labels, this invention not only enhances the segmentation performance of known categories of sounding objects, but also significantly improves the segmentation and classification capabilities of unknown categories and the generalization ability of the model in open vocabulary scenarios through the knowledge of the pre-trained basic model. Attached Figure Description

[0044] Figure 1 This is a diagram of the overall architecture of the open vocabulary audiovisual segmentation method based on semantic consistency involved in this application (the diagram includes a structural diagram of the audio semantic enhancement module).

[0045] Figure 2 This is a detailed diagram of the symmetric cross-modal attention guidance module within the method involved in this application.

[0046] Figure 3 This is a detailed diagram of the hierarchical modal fusion decoder involved in the method of this application. Detailed Implementation

[0047] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0048] Reference Figure 1 An open-lexical audiovisual segmentation method based on semantic consistency includes the following steps:

[0049] Step 1: Audiovisual encoding, including the following steps:

[0050] For image encoding, this design uses the CLIP image encoder to extract visual features from each video frame, obtaining multi-scale visual features; simultaneously, the CLAP audio encoder processes the audio, extracting corresponding audio features. Furthermore, during retraining, all training categories (known categories) are combined with CLIP and CLAP text cue templates, respectively, and their corresponding CLIP-encoded and CLAP-encoded text embeddings are output through their respective text encoders (CLIP and CLAP text encoders). During inference, all categories (including known and unknown categories) are used for text encoding.

[0051] Step 2: Optimize audiovisual bidirectional interaction through a symmetrical cross-modal attention guidance module. The specific steps are as follows:

[0052] CLIP-encoded multi-scale visual features are mapped to a unified feature dimension through convolutional layers, and then merged along the sequence length dimension through spatial unfolding. Simultaneously, CLIP-encoded audio features are mapped to the same feature dimension as the multi-scale visual features through a multilayer perceptron. The preprocessed audiovisual features are input into a symmetrical cross-modal attention guidance module, which utilizes complementary audiovisual information to perform bidirectional optimization of the visual and audio features, resulting in improved audio and visual features.

[0053] Step 3: Decoding stage based on hierarchical modal fusion decoder, including the following steps:

[0054] First, the improved audio features output from step 2 are temporally unfolded, and then optimized using a self-attention mechanism and temporal context association to obtain temporally enhanced audio features. Second, a set of target queries initialized with learnable parameters interacts with the temporally enhanced audio features through cross-modal attention to obtain initial target queries embedded with audio cues. The target queries interact with the audiovisual features layer by layer through the hierarchical modal fusion decoder decoding layer, aggregating the audiovisual information of the sound-producing object, and outputting the final target query. .

[0055] Step 4: Training loss function assisted by prediction generation and audio semantic enhancement modules, the specific steps are as follows:

[0056] This design proposes a one-stage open-vocabulary audiovisual semantic segmentation framework, which uniformly generates prediction masks and corresponding semantic categories. Specifically, in the mask generation process, the target query... First, after processing by a multilayer perceptron, it is compared with the highest resolution visual features output by a pixel-level decoder. The mask features are used to generate the corresponding segmentation mask prediction through matrix multiplication. In the CLIP-based classification header, segmentation mask prediction and mask features are obtained through mask pooling to obtain mask pooled features, which are then combined with the target query. Residual connections are performed, and category features are obtained after processing through a multilayer perceptron; simultaneously, text embedding is performed. The category was generated using the CLIP text hint template "A photo of {}" via a CLIP text encoder; finally, the category features were embedded with the text. Similarity is calculated using dot products to generate corresponding category predictions. The audio semantic enhancement module explicitly models audio semantics and outputs an auxiliary audio semantic classification based on the frozen CLAP text encoder. .

[0057] During training, this design considers both semantic classification loss and prediction mask loss; the semantic classification loss consists of two parts: one is the CLIP-based main classification loss from the classification head. Secondly, the CLAP-based auxiliary audio supervised classification loss introduced by the audio semantic enhancement module. The total losses are as follows:

[0058]

[0059] Among them, the mask loss function From the prediction mask and the real mask The focus and Diceloss are calculated. , , These represent the weighting parameters, That is, the total loss function.

[0060] During the inference phase, this invention adopts the FC-CLIP strategy to achieve accurate and robust open-vocabulary audiovisual semantic segmentation. Specifically, it utilizes the segmentation prediction mask generated by the masking tool. Another set of semantic classification predictions is obtained by calculating similarity between masked pooling and text embeddings containing unknown categories encoded using the frozen CLIP text encoder. Among them, the output of the classification header is Responsible for classifying and recognizing known categories. The focus is on identifying unknown categories; subsequently, and By fusing through geometric integration, a final semantic category prediction is generated that combines the ability to distinguish known categories with the ability to generalize from open vocabularies. Finally, this semantic category prediction is combined with the prediction mask to generate an audiovisual semantic segmentation mask.

[0061] Experimental verification:

[0062] I. Verification Details

[0063] The proposed method employs the ConvNeXt-Large CLIP and HTS-AT CLAP models. The pixel decoder utilizes a multi-scale deformable attention deformer architecture. In the decoder, the number of learnable queries is set to 100. Following previous work on audiovisual segmentation, various data augmentation techniques are introduced, including flipping and random cropping. During training, the AdamW optimizer is used with a batch size of 1 and a learning rate of 0.0001 to improve training stability.

[0064] During training, the weight coefficients of the loss function were set to... =2.0, , =5.0, trained on the AVSBench_OV dataset for 96228 iterations.

[0065] To ensure fairness, following previous research, the same evaluation metrics were used: the overall mean Intersection-over-Union (mIoU), the base mIoU (for known classes), the novel mIoU (for unknown classes), and the harmonic mIoU (for both). The formula for calculating the harmonic mIoU is as follows:

[0066]

[0067] II. Quantitative Results

[0068] Table 1 shows the comparison results between the open-vocabulary audiovisual semantic segmentation model proposed in this invention and existing state-of-the-art methods. As can be seen from the data in the table, the model designed in this application achieves excellent performance across all evaluation metrics, significantly outperforming the OV-AVSS method. These results fully verify the balanced and robust generalization ability of this model.

[0069] Table 1. Comparison results with state-of-the-art methods on the AVSBench-OV dataset.

[0070] .

Claims

1. An open-lexical audiovisual segmentation method based on semantic consistency, characterized in that, Specifically as follows: (1) The specific implementation steps of the symmetrical cross-modal attention guidance module are as follows: Audio features encoded by the CLAP audio encoder Multiscale visual features encoded by CLIP image encoder First, multi-scale visual features are projected onto a unified feature dimension through convolution. Then, it is expanded along the spatial dimension and merged into multi-scale features, resulting in Simultaneously, audio features are mapped to the same feature dimension through a multilayer perceptron, denoted as . Then use a cross-modal attention mechanism for fusion. and The fused multimodal features are obtained, denoted as . : ； in, Represents row-by-row normalization Activation function Represents the learnable projection matrix; To fully capture complex audiovisual relationships, a two-layer cross-modal interaction layer is designed, based on visual features. and audio features For queries, all data are derived from the fused multimodal features. As a key or value, through a cross-modal attention mechanism Selective injection of fused information into visual features and audio features The enhanced visual and audio features are obtained separately, guiding them to more accurately focus on the sound-producing object; the corresponding calculation formulas are as follows: ； ； in, and These represent the enhanced visual and audio features, respectively. , All are learnable projection matrices; Subsequently, The input is fed into a pixel decoder based on multi-scale deformable attention for multi-scale feature interaction, resulting in more accurate visual features, denoted as... ; (2) The hierarchical modal fusion decoder, the specific implementation steps of which are as follows: First, the enhanced audio features are temporally unfolded, and then optimized using a self-attention mechanism and temporal context association to obtain temporally enhanced audio features, denoted as . Then, a set of target queries initialized with learnable parameters are processed through cross-modal attention and temporally enhanced audio features. Interact to obtain the initial target query with embedded audio clues. ; The hierarchical modality fusion decoder consists of three decoding layers. In each decoding layer, the target query is combined with the visual features output by the pixel decoder. Perform cross-modal attention interaction to extract visual information of objects corresponding to vocal cues, and obtain an updated target query. : ； in, ; This indicates the target query from the output of the previous decoding layer. When the decoding layer is the first layer, This represents a cross-modal attention mechanism; Subsequently, the updated target query is further interacted with audio features to obtain a target query that integrates audio information. : ； Next, the target query is explicitly modeled by performing self-attention computation on the target query that integrates audio information in the spatiotemporal dimension. This cross-frame association further enhances the temporal consistency of the target query: ； in, This represents the self-attention mechanism; Indicates that the shape is Tensor unfolds into shape The target query, The number of target queries, Indicates the channel dimension; This represents the reverse operation of restoring the result of the above unfolding operation to its original shape; finally, it involves performing a target query to enhance temporal consistency. After processing by the feedforward network, the target query that integrates audiovisual information is output, denoted as... ; (3) Audio semantic enhancement module, the specific implementation steps of which are as follows: First, target query Audio features with temporal enhancement Interacting through cross-modal attention allows each target query to selectively aggregate relevant audio information and explicitly fuse audio semantics; subsequently, residual connections are applied to the attention output, and the fused features are further integrated through a multilayer perceptron. The CLAP text hint template "This sound contains the {}" is used for the category, and the embedded text is obtained through the CLAP text encoder. Finally, fusion features and text embedding The final audio semantic classification result is obtained by calculating similarity through dot product. The above process can be represented as: ； ； The audio semantic enhancement module introduces an auxiliary audio-supervised semantic classification loss. The loss formula is expressed as: ； in, This indicates the calculation of cross-entropy loss. This represents the actual category label corresponding to the audio. (4) Model mask generation and overall loss calculation, the specific steps are as follows: To achieve open-lexical audiovisual semantic segmentation, a CLIP-based classification head and a masking mechanism are used for audiovisual semantic segmentation; in the masking mechanism, the target query... First, after processing by a multilayer perceptron, it is compared with the highest resolution visual features output by a pixel-level decoder. The mask features are used to generate the corresponding segmentation mask prediction through matrix multiplication. In the CLIP-based classification header, segmentation mask prediction and mask features are obtained through mask pooling to obtain mask pooled features, which are then combined with the target query. Residual connections are performed, and category features are obtained after processing through a multilayer perceptron; simultaneously, text embedding is performed. The category was generated using the CLIP text hint template "A photo of {}" via a CLIP text encoder; finally, the category features were embedded with the text. Similarity is calculated using dot products to generate corresponding category predictions. ; During training, both semantic classification loss and prediction mask loss are considered; the semantic classification loss consists of two parts: one is the CLIP-based main classification loss from the classification head. Secondly, the CLAP-based auxiliary audio supervised classification loss introduced by the audio semantic enhancement module. The total losses are shown below: ； ； Among them, the mask loss function From the prediction mask and the real mask The focus and Dice loss are calculated. , , These represent the weighting parameters, That is, the total loss function; During the inference phase, the segmentation prediction mask generated by the masking tool is used. Another set of semantic classification predictions is obtained by calculating similarity between masked pooling and text embeddings containing unknown categories encoded using the frozen CLIP text encoder. Among them, the output of the classification header is Responsible for classifying and recognizing known categories. The focus is on identifying unknown categories; subsequently, and By fusing through geometric integration, a final semantic category prediction is generated that combines the ability to distinguish known categories with the ability to generalize from open vocabularies. Finally, this semantic category prediction is combined with the prediction mask to generate an audiovisual semantic segmentation mask.

Images