Audio-guided visual segmentation method and device based on multi-granularity cross-modal coupling

Abstract

This invention relates to the fields of artificial intelligence and multimedia technology, and particularly to an audio-guided visual segmentation method and apparatus based on multi-granularity cross-modal coupling. The method includes: extracting multi-level visual features and audio Mel-spectral features from a target video frame; performing intra-modal enhancement on the multi-level visual features and audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features; fusing the enhanced multi-level visual features and enhanced audio Mel-spectral features across modalities to generate a semantically enhanced query vector; training a pre-constructed Transformer attention decoder using the semantically enhanced query vector to generate a pixel-level segmentation mask; and fusing the pixel-level segmentation mask with the multi-level visual features to obtain a mask prediction result. This solves the problems of intra-modal noise interference, insufficient audio guidance, and multi-source sound entanglement in existing audiovisual segmentation methods.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimedia technology, and in particular to an audio-guided visual segmentation method and apparatus based on multi-granularity cross-modal coupling. Background Technology

[0002] Visual and auditory signals are naturally complementary and play a crucial role in human perception and understanding of the real world. With the rapid development of artificial intelligence and multimedia technologies, various audiovisual understanding tasks have emerged, such as Audio-Visual Correspondence (AVC), Audio-Visual Instance Discrimination (AVID), and Audio-Visual Event Localization (AVEL). Among these, Audio-Visual Segmentation (AVS) is a more fine-grained and challenging task, requiring precise alignment between audio semantics and visual pixel regions. The AVS task aims to accurately locate the frame-level pixel region corresponding to the sound source in a given video, based on the audio source category. As a cross-modal task, AVS requires fine-grained alignment from audio semantics to visual pixel-level targets. AVS includes three sub-tasks: Single-Source Segmentation (S4), Multi-Source Segmentation (MS3), and Audio-Visual Semantic Segmentation (AVSS). In real-world scenarios, AVS faces a series of complex challenges. From the perspective of visual targets, audio sources are often intertwined, influenced by background noise and off-screen audio. From the perspective of sound sources, the coexistence of multiple visual objects increases the ambiguity of semantic alignment. Furthermore, due to temporal dynamics, the intertwined nature of visual and audio streams makes it difficult for models to capture temporally consistent cross-modal semantic boundaries.

[0003] Traditional work primarily focuses on feature-level interactions between audio and visual signals. These methods typically extract high-dimensional representations from visual frames and audio spectrograms, then use fine-grained cross-modal interaction modules to capture audiovisual semantic associations. Under audio-guided queries, the model decodes pixel-level masks of sound-producing objects in the visual scene. The AVSBench model first proposed the AVS cross-modal fusion paradigm, and subsequent research has largely followed this architecture. However, recent studies have shown that this paradigm may lead to an overemphasis on visual semantics, degrading the accuracy of audio source-object segmentation. Furthermore, the COMBO model significantly improves segmentation performance by introducing mask patterns at the input stage using a general segmentation big model, thereby imposing pixel-level constraints on visual objects, but its attention to audio queries remains insufficient. In summary, the existing methods have the following drawbacks: (1) Intramodal noise interference: Existing methods ignore intramodal semantic optimization and retain too much background noise, resulting in a decrease in segmentation accuracy; (2) Insufficient audio guidance: Visual modality is used as the main decoding source, underestimating the key role of audio semantics in cross-modal alignment, relying on single visual feature decoding, and the audio semantic guidance capability is insufficient; (3) Multi-source sound entanglement: When multiple sources of sound coexist in complex scenes, existing methods are difficult to effectively decouple semantics, resulting in blurred segmentation, and the lack of pixel-level contrast learning strategies makes it difficult to optimize edge details. Summary of the Invention

[0004] This invention provides an audio-guided visual segmentation method and apparatus based on multi-granularity cross-modal coupling to solve problems such as intramodal noise interference, insufficient audio guidance, and multi-source sound entanglement in existing audiovisual segmentation methods.

[0005] The first aspect of this invention provides an audio-guided visual segmentation method based on multi-granularity cross-modal coupling, comprising the following steps:

[0006] Multi-level visual features and audio Mel-spectral features of the target video frame are extracted; intra-modal enhancement is performed on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features; cross-modal fusion is performed on the enhanced multi-level visual features and the enhanced audio Mel-spectral features to generate a semantically enhanced query vector; the semantically enhanced query vector is used to train a pre-built Transformer attention decoder until a preset hybrid loss function converges to generate a pixel-level segmentation mask; and the pixel-level segmentation mask is fused with the multi-level visual features to obtain the mask prediction result.

[0007] Optionally, the extraction of multi-level visual features and audio Mel-spectral features of the target video frame includes:

[0008] The target video frame is input into a preset image segmentation model to generate prior mask features; four-layer multi-level features of the target video frame are extracted, and the four-layer multi-level features and the prior mask features are fused to generate the multi-level visual features; the target video frame is input into a pre-constructed deep learning-based audio feature extraction model to extract the audio Mel-spectral features.

[0009] Optionally, the intramodal enhancement of the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features includes:

[0010] The multi-level visual features are input into a pre-constructed intramodal noise suppression model to calculate channel attention and spatial attention, thereby generating visual channel weights and visual spatial weights.

[0011] The multi-level visual features are weighted using the visual channel weights and visual spatial weights to generate the enhanced multi-level visual features; the audio Mel-spectral features are input into the intramodal noise suppression model for channel attention calculation to generate audio channel weights; the audio Mel-spectral features are weighted using the audio channel weights to generate the enhanced audio Mel-spectral features.

[0012] Optionally, the step of cross-modal fusion of the enhanced multi-level visual features and the enhanced audio Mel-spectral features to generate a semantically enhanced query vector includes:

[0013] A pre-built semantic fusion model is used to map the enhanced multi-level visual features and the enhanced audio Mel-spectral features to calculate a cross-modal attention map. The enhanced multi-level visual features are then weighted using the cross-modal attention map to obtain multi-level visual features that specifically guide audio semantics. A pre-built semantic-aware audio query model is used to extract visual temporal features from the enhanced multi-level visual features and audio temporal features from the enhanced audio Mel-spectral features. A cross-modal similarity matrix is ​​calculated using the visual temporal features and the audio temporal features. The semantically enhanced query vector is calculated based on the cross-modal similarity matrix and the multi-level visual features that specifically guide audio semantics.

[0014] A second aspect of this invention provides an audio-guided visual segmentation device based on multi-granularity cross-modal coupling, comprising:

[0015] An extraction module is used to extract multi-level visual features and audio Mel-spectral features of the target video frame; an enhancement module is used to perform intra-modal enhancement on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features; a generation module is used to perform cross-modal fusion on the enhanced multi-level visual features and the enhanced audio Mel-spectral features to generate a semantically enhanced query vector; a training module is used to train a pre-built Transformer attention decoder using the semantically enhanced query vector until a preset hybrid loss function converges to generate a pixel-level segmentation mask, and then fuse the pixel-level segmentation mask with the multi-level visual features to obtain a mask prediction result.

[0016] Optionally, the extraction module includes:

[0017] The generation unit is used to input the target video frame into a preset image segmentation model to generate prior mask features; the first extraction unit is used to extract four-layer multi-level features of the target video frame and fuse the four-layer multi-level features and the prior mask features to generate the multi-level visual features; the second extraction unit is used to input the target video frame into a pre-constructed deep learning-based audio feature extraction model to extract the audio Mel-spectral features.

[0018] Optionally, the enhancement module includes:

[0019] The first calculation unit is used to input the multi-level visual features into a pre-constructed intra-modal noise suppression model to perform channel attention and spatial attention calculations, so as to generate visual channel weights and visual spatial weights; the first weighting unit is used to weight the multi-level visual features using the visual channel weights and visual spatial weights, so as to generate the enhanced multi-level visual features; the second calculation unit is used to input the audio Mel-frequency spectral features into the intra-modal noise suppression model to perform channel attention calculations, so as to generate audio channel weights; the second weighting unit is used to weight the audio Mel-frequency spectral features using the audio channel weights, so as to generate the enhanced audio Mel-frequency spectral features.

[0020] Optionally, the generation module includes:

[0021] The third calculation unit is used to map the enhanced multi-level visual features and the enhanced audio Mel-spectral features using a pre-built semantic fusion model to calculate a cross-modal attention map; the third weighting unit is used to weight the enhanced multi-level visual features using the cross-modal attention map to obtain multi-level visual features that specifically guide audio semantics; the second extraction unit is used to extract the visual temporal features from the enhanced multi-level visual features and the audio temporal features from the enhanced audio Mel-spectral features using a pre-built semantic-aware audio query model; the fourth calculation unit is used to calculate a cross-modal similarity matrix using the visual temporal features and the audio temporal features; the fifth calculation unit is used to calculate the semantically enhanced query vector based on the cross-modal similarity matrix and the multi-level visual features that specifically guide audio semantics.

[0022] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the audio-guided visual segmentation method based on multi-granularity cross-modal coupling as described in the above embodiments.

[0023] The fourth aspect of this invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described audio-guided visual segmentation method based on multi-granularity cross-modal coupling.

[0024] The audio-guided visual segmentation method and apparatus based on multi-granularity cross-modal coupling proposed in this invention effectively solves the problems of intramodal noise interference, insufficient audio guidance, and multi-source sound entanglement by establishing a multi-modal coupling mechanism and a pixel-level cross-modal contrastive learning strategy. It optimizes the edge details of sound source targets and strengthens the guiding role of audio semantics in visual segmentation, thereby solving the semantic decoupling problem in multi-source sound scenarios. It achieves accurate matching of audio semantics and visual regions in complex audiovisual scenarios and improves the cross-modal understanding capabilities of applications such as video content analysis and intelligent monitoring.

[0025] Additional aspects and advantages of this invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 The flowchart shows an audio-guided visual segmentation method based on multi-granularity cross-modal coupling provided in an embodiment of the present invention.

[0028] Figure 2 This is an overall architecture diagram of an audio-guided visual segmentation method based on multi-granularity cross-modal coupling provided in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of a MASS model provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram illustrating the interaction process between an AGSF model and an SAQ model provided in an embodiment of the present invention.

[0031] Figure 5 This is a block diagram of an audio-guided visual segmentation device based on multi-granularity cross-modal coupling provided in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0034] The following description, with reference to the accompanying drawings, describes an embodiment of the present invention: an audio-guided visual segmentation and apparatus based on multi-granularity cross-modal coupling.

[0035] Figure 1 This is a flowchart illustrating an audio-guided visual segmentation method based on multi-granularity cross-modal coupling provided in an embodiment of the present invention.

[0036] like Figure 1 As shown, the audio-guided visual segmentation method based on multi-granularity cross-modal coupling includes the following steps:

[0037] In step S101, multi-level visual features and audio Mel-spectral features of the target video frame are extracted.

[0038] In some embodiments, extracting multi-level visual features and audio Mel-spectral features of the target video frame includes:

[0039] The target video frame is input into a preset image segmentation model to generate prior mask features;

[0040] Extract four-level multi-layer features from the target video frame and fuse the four-level multi-layer features with the prior mask features to generate multi-level visual features;

[0041] The target video frame is input into a pre-built deep learning-based audio feature extraction model to extract audio Mel-spectral features.

[0042] In actual execution, the target video frame (e.g., 224×224 resolution) is input into a preset image segmentation model (Segment Anything Model, SAM) to generate prior mask features F. mask The four-layer multi-level feature F is extracted using ResNet-50 or PVT-v2. visual (1 / 4 to 1 / 32 scale).

[0043] Furthermore, the four-layer multi-level feature F visual and prior mask features F mask The data is fused to generate multi-level visual features F, specifically expressed as:

[0044]

[0045] In the formula, F i For the fused image features, Visual features extracted by the image encoder The generated mask features are defined by W, where W is the linear layer weight and i represents the layer of the current feature.

[0046] Furthermore, the target video frame is input into a pre-built deep learning-based audio feature extraction model (VGG-like audio classification model, VGGish) to extract the audio signal (16Hz) from the target video frame, and the audio signal (16Hz) (16kHz) is converted into a Mel spectrum (96×64) to obtain the Mel spectrum feature F. a Timing alignment is performed to the video frame number T.

[0047] In step S102, intramodal enhancement is performed on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features.

[0048] In some embodiments, intramodal enhancement is performed on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features, including:

[0049] Multi-level visual features are input into a pre-built intramodal noise suppression model to calculate channel attention and spatial attention, thereby generating visual channel weights and visual spatial weights.

[0050] Multi-level visual features are weighted by visual channel weights and visual spatial weights to generate enhanced multi-level visual features.

[0051] The audio Mel-spectral features are input into the intramodal noise suppression model to calculate channel attention and generate audio channel weights.

[0052] The audio Mel-spectral features are weighted by audio channel weights to generate enhanced audio Mel-spectral features.

[0053] In actual implementation, such as Figure 2-3 As shown, multi-level visual features are input into a pre-built Intra-Modal Noise Suppression Model with Advanced Suppression Strategy (IMASS). The IMASS model employs a channel-space dual attention mechanism to perform noise suppression and semantic enhancement on visual and audio features respectively, as detailed below:

[0054] Multi-level visual or audio features F are input into the IMASS model for channel attention. Features are aggregated using both GAP and GMP paths and then processed through a convolutional layer to generate visual channel weights F. avg and visual spatial weight F max And through visual channel weight F avg and visual spatial weight F max The multi-level visual or audio features F are weighted to generate enhanced multi-level visual or audio features F. c The specific formula is as follows:

[0055] F avg =ReLU((GAP(F)W K1 W K2 (2)

[0056] F max =ReLU((GMP(F)W Q1 W Q2 (3)

[0057] F c =σ(F avg +F max )·F (4)

[0058] In the formula, W K1 W K2 W Q1 W Q2 All are linear layer weight parameters, and σ represents the sigmoid activation function.

[0059] Furthermore, F c The input is fed into the spatial attention layer, where it undergoes GAP and GMP processing on the feature dimension, and then concatenates to form an output with a feature dimension of 2. This output is then processed by a 7×7 convolution to generate the weights F. s Through weight F s For F c Weighting is performed to generate the enhanced sound feature F. I The specific formula is as follows:

[0060] F s =σ(Conv 7×7 ([GAP(F a GMP(F) a )]))·(F a (5)

[0061] F I =F s +F a (6)

[0062] In step S103, the enhanced multi-level visual features and the enhanced audio Mel spectrum features are fused across modalities to generate a semantically enhanced query vector.

[0063] In some embodiments, enhanced multi-level visual features and enhanced audio Mel-spectral features are fused across modally to generate a semantically enhanced query vector, including:

[0064] A pre-built semantic fusion model is used to map enhanced multi-level visual features and enhanced audio Mel-spectral features to compute cross-modal attention maps;

[0065] We use cross-modal attention maps to weight the enhanced multi-level visual features to obtain multi-level visual features that specifically guide audio semantics;

[0066] Using a pre-built semantic-aware audio query model, visual temporal features from enhanced multi-level visual features and audio temporal features from enhanced audio Mel-spectral features are extracted respectively.

[0067] Calculate the cross-modal similarity matrix using visual and audio temporal features;

[0068] The semantically enhanced query vector is calculated based on the cross-modal similarity matrix and the multi-level visual features that specifically guide the audio semantics.

[0069] In actual implementation, such as Figure 4 As shown, the enhanced multi-level visual features and enhanced audio Mel-spectral features The input is fed into an Adaptive Graph-based Semantic Fusion (AGSF) model, where the enhanced visual features are processed by an MLP. Compared with enhanced audio Mel-spectral features Mapped to keys respectively With query A cross-modal attention map is generated by tanh activation. The enhanced multi-level visual features are then weighted using the cross-modal attention map to obtain multi-level visual features that specifically guide audio semantics, thereby focusing on strongly correlated regions.

[0070] Enhanced multi-level visual features and enhanced audio Mel-spectral features The input is fed into a pre-built semantic-aware audio query model (SAQ), and the enhanced multi-level visual features F are processed using a bidirectional LSTM (Long Short-Term Memory) network. c and enhanced audio Mel-spectral features F I To output visual temporal features With audio timing features And utilize visual temporal features With audio timing features Calculate the similarity matrix M s Furthermore, based on the cross-modal similarity matrix M s The semantically enhanced query vector Q is generated by calculating multi-level visual features that specifically guide audio semantics. a As the input to the subsequent TTransformer attention decoder, the specific formula is as follows:

[0071]

[0072] In the formula, W v1 W a W v2 W1 and W2 are both linear layer weight parameters, and D is the number of channels for the temporal feature. These are the visual features after similarity matrix processing.

[0073] In step S104, the pre-built Transformer attention decoder is trained using the semantically enhanced query vector until the preset hybrid loss function converges to generate a pixel-level segmentation mask. The pixel-level segmentation mask is then fused with multi-level visual features to obtain the mask prediction result.

[0074] In actual implementation, the Transformer attention decoder uses Q...a For querying, iterative training is performed using multi-level visual features F until the preset hybrid loss function (Adaptive Gradient Penalty for Continuity, AGPC) converges to generate a pixel-level segmentation mask. The pixel-level segmentation mask is then fused with the multi-level visual features to obtain the mask prediction result.

[0075] The AGPC hybrid loss function is calculated using weighted IoU and iteratively optimizes the pixel-level classification boundary. The specific formula is as follows:

[0076]

[0077]

[0078] Among them, A I To predict the pixel intersection of the mask and the real mask, x i y is the predicted value of the pixel. i A represents the actual value of the pixel. U To predict the pixel union of the mask and the real mask, The loss is AGPC mixture loss, where N is the total number of classes. Let be the pixel intersection of the predicted mask and the true mask for the nth class. Let be the pixel union of the predicted mask and the true mask for the nth category.

[0079] The AdamW optimizer is employed with a learning rate of 1e-4 and weight decay of 0.05. Training is performed 90k times for a single-source task (S4) and 20k times for a multi-source task (MS3). The batch size is dynamically adjusted based on the backbone network (16 for ResNet-50 and 4 for PVT-v2). Furthermore, during the inference stage, audio queries are fused with visual features to output a pixel-level segmentation mask, supporting real-time processing of complex scene video streams.

[0080] Furthermore, the AGPC mixing loss (λ) in this embodiment of the invention agpc =5) and the combined cross-entropy (λ) of the total loss function ce =2), Dice loss (λ) mask =5) and Focal loss (λ) f =5), the calculation process for mixed loss is as follows:

[0081]

[0082] in, For mask loss, For binary cross-entropy loss, For Dice's loss, For Focal loss, For the total loss, This represents the cross-entropy loss.

[0083] It should be noted that the embodiments of the present invention also perform performance verification. In the AVSBench-Object task, AG-AVCN (Audio-Guided Multi-Granularity Audio-Visual Coupling Network) (PVT-v2 backbone network) achieved 84.8% mIoU on the S4 task and 74.3% F-score on the MS3 task, which is 3.2% higher than the COMBO (Cooperation of Multi-order Bilateral relations) model. Ablation experiments show that removing the SAQ model leads to a 13.0% decrease in mIoU on the MS3 task, verifying its key role.

[0084] In summary, the audio-guided visual segmentation method based on multi-granularity cross-modal coupling proposed according to the embodiments of the present invention has the following beneficial effects:

[0085] (1) The IMASS module used reduces background noise interference through a channel-space dual attention mechanism, which improves the mIoU of the S4 task by 6.5% compared with traditional methods (such as AVSBench);

[0086] (2) The SAQ module used enhances the semantic expression of audio queries, and improves the F-score of multi-source scenarios (MS3) by 12.9%, effectively solving the problem of multi-source entanglement;

[0087] (3) The AGPC loss method combined with weighted IoU and multi-task loss improves edge segmentation accuracy by 3.1%, which is better than the traditional Dice loss.

[0088] (4) It features high performance and lightweight design, making it easy to deploy in real-world application scenarios. The number of model parameters is reduced by 20% compared to QDFormer, and the inference speed is increased by 30% in AVSS tasks, making it suitable for real-time scenarios.

[0089] Next, referring to the accompanying drawings, an audio-guided visual segmentation device based on multi-granularity cross-modal coupling according to an embodiment of the present invention is described.

[0090] Figure 5 This is a block diagram of an audio-guided visual segmentation device based on multi-granularity cross-modal coupling according to an embodiment of the present invention.

[0091] like Figure 5As shown, the audio-guided visual segmentation device 50 based on multi-granularity cross-modal coupling includes: an extraction module 501, an enhancement module 502, a generation module 503, and a training module 504.

[0092] The extraction module 501 extracts multi-level visual features and audio Mel-spectral features from the target video frame. The enhancement module 502 performs intra-modal enhancement on the multi-level visual features and audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features. The generation module 503 performs cross-modal fusion of the enhanced multi-level visual features and enhanced audio Mel-spectral features to generate a semantically enhanced query vector. The training module 504 trains a pre-built Transformer attention decoder using the semantically enhanced query vector until a preset hybrid loss function converges to generate a pixel-level segmentation mask. The pixel-level segmentation mask is then fused with the multi-level visual features to obtain the mask prediction result.

[0093] In some embodiments, the extraction module 501 includes:

[0094] The generation unit is used to input the target video frame into a preset image segmentation model to generate prior mask features;

[0095] The first extraction unit is used to extract four-level multi-level features of the target video frame and fuse the four-level multi-level features with the prior mask features to generate multi-level visual features.

[0096] The second extraction unit is used to input the target video frame into a pre-built deep learning-based audio feature extraction model to extract audio Mel-spectral features.

[0097] In some embodiments, the enhancement module 502 includes:

[0098] The first computing unit is used to input multi-level visual features into a pre-built intramodal noise suppression model to perform channel attention and spatial attention calculations in order to generate visual channel weights and visual spatial weights.

[0099] The first weighting unit is used to weight multi-level visual features by visual channel weights and visual spatial weights to generate enhanced multi-level visual features.

[0100] The second computational unit is used to input the audio Mel-spectral features into the intramodal noise suppression model to perform channel attention calculation in order to generate audio channel weights;

[0101] The second weighting unit is used to weight the audio Mel-spectral features by the audio channel weights to generate enhanced audio Mel-spectral features.

[0102] In some embodiments, the generation module 503 includes:

[0103] The third computational unit is used to map enhanced multi-level visual features and enhanced audio Mel-spectral features using a pre-built semantic fusion model to compute cross-modal attention maps.

[0104] The third weighting unit is used to weight the enhanced multi-level visual features using cross-modal attention maps to obtain multi-level visual features that specifically guide audio semantics.

[0105] The second extraction unit is used to extract visual temporal features from the enhanced multi-level visual features and audio temporal features from the enhanced audio Mel spectrum features using a pre-built semantic-aware audio query model.

[0106] The fourth computational unit is used to calculate the cross-modal similarity matrix using visual and audio temporal features;

[0107] The fifth computational unit is used to calculate the semantically enhanced query vector based on the cross-modal similarity matrix and the multi-level visual features of the specific guided audio semantics.

[0108] It should be noted that the foregoing explanation of the embodiment of the audio-guided visual segmentation method based on multi-granularity cross-modal coupling also applies to the audio-guided visual segmentation device based on multi-granularity cross-modal coupling in this embodiment, and will not be repeated here.

[0109] The audio-guided visual segmentation device based on multi-granularity cross-modal coupling proposed in this invention has the following beneficial effects:

[0110] (1) The IMASS module used reduces background noise interference through a channel-space dual attention mechanism, which improves the mIoU of the S4 task by 6.5% compared with traditional methods (such as AVSBench);

[0111] (2) The SAQ module used enhances the semantic expression of audio queries, and improves the F-score of multi-source scenarios (MS3) by 12.9%, effectively solving the problem of multi-source entanglement;

[0112] (3) The AGPC loss method combined with weighted IoU and multi-task loss improves edge segmentation accuracy by 3.1%, which is better than the traditional Dice loss.

[0113] (4) It features high performance and lightweight design, making it easy to deploy in real-world application scenarios. The number of model parameters is reduced by 20% compared to QDFormer, and the inference speed is increased by 30% in AVSS tasks, making it suitable for real-time scenarios.

[0114] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0115] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0116] When the processor 602 executes the program, it implements the audio-guided visual segmentation method based on multi-granularity cross-modal coupling provided in the above embodiments.

[0117] Furthermore, electronic devices also include:

[0118] Communication interface 603 is used for communication between memory 601 and processor 602.

[0119] The memory 601 is used to store computer programs that can run on the processor 602.

[0120] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0121] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0123] Processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0124] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described audio-guided visual segmentation method based on multi-granularity cross-modal coupling.

[0125] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0126] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0127] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the present invention pertain.

[0128] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0129] It should be understood that various parts of this invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0130] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0131] Furthermore, the functional units in the various embodiments of this invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0132] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. An audio-guided visual segmentation method based on multi-granularity cross-modal coupling, characterized in that, Includes the following steps: Extract multi-level visual features and audio Mel-spectral features from the target video frame; Intramodal enhancement is performed on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features; The enhanced multi-level visual features and the enhanced audio Mel-spectral features are fused across modally to generate a semantically enhanced query vector, specifically including: The enhanced multi-level visual features and the enhanced audio Mel-spectral features are mapped using a pre-built semantic fusion model to compute a cross-modal attention map; The enhanced multi-level visual features are weighted using the cross-modal attention map to obtain multi-level visual features that specifically guide audio semantics; The visual temporal features in the enhanced multi-level visual features and the audio temporal features in the enhanced audio Mel-spectral features are extracted using a pre-built semantic-aware audio query model. Calculate the cross-modal similarity matrix using the visual temporal features and the audio temporal features; The semantic enhancement query vector is calculated based on the cross-modal similarity matrix and the multi-level visual features of the specific guided audio semantics; The semantically enhanced query vector is used to train the pre-built Transformer attention decoder until the preset hybrid loss function converges to generate a pixel-level segmentation mask. The pixel-level segmentation mask is then fused with the multi-level visual features to obtain the mask prediction result.

2. The audio-guided visual segmentation method based on multi-granularity cross-modal coupling according to claim 1, characterized in that, The extraction of multi-level visual features and audio Mel-spectral features of the target video frame includes: The target video frame is input into a preset image segmentation model to generate prior mask features; Extract the four-layer multi-level features of the target video frame, and fuse the four-layer multi-level features with the prior mask features to generate the multi-level visual features; The target video frame is input into a pre-built deep learning-based audio feature extraction model to extract the audio Mel-spectral features.

3. The audio-guided visual segmentation method based on multi-granularity cross-modal coupling according to claim 1, characterized in that, The intramodal enhancement of the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features includes: The multi-level visual features are input into a pre-constructed intramodal noise suppression model to calculate channel attention and spatial attention, thereby generating visual channel weights and visual spatial weights. The multi-level visual features are weighted by the visual channel weights and visual spatial weights to generate the enhanced multi-level visual features. The audio Mel-spectral features are input into the intramodal noise suppression model to perform channel attention calculation, thereby generating audio channel weights; The audio Mel-spectral features are weighted using the audio channel weights to generate the enhanced audio Mel-spectral features.

4. An audio-guided visual segmentation device based on multi-granularity cross-modal coupling, characterized in that, include: The extraction module is used to extract multi-level visual features and audio Mel-spectral features of the target video frame; An enhancement module is used to perform intramodal enhancement on the multi-level visual features and the audio Mel-spectral features to obtain enhanced multi-level visual features and enhanced audio Mel-spectral features. The generation module is used to perform cross-modal fusion of the enhanced multi-level visual features and the enhanced audio Mel-spectral features to generate a semantically enhanced query vector, specifically including: The third computing unit is used to map the enhanced multi-level visual features and the enhanced audio Mel-spectral features using a pre-built semantic fusion model to compute a cross-modal attention map; The third weighting unit is used to weight the enhanced multi-level visual features using the cross-modal attention map to obtain multi-level visual features that specifically guide audio semantics. The second extraction unit is used to extract the visual temporal features from the enhanced multi-level visual features and the audio temporal features from the enhanced audio Mel-spectral features using a pre-built semantic-aware audio query model. The fourth calculation unit is used to calculate the cross-modal similarity matrix using the visual temporal features and the audio temporal features; The fifth calculation unit is used to calculate the semantic enhancement query vector based on the cross-modal similarity matrix and the multi-level visual features of the specific guided audio semantics; The training module is used to train the pre-built Transformer attention decoder using the semantically enhanced query vector until the preset hybrid loss function converges to generate a pixel-level segmentation mask, and then fuse the pixel-level segmentation mask with the multi-level visual features to obtain the mask prediction result.

5. The audio-guided visual segmentation device based on multi-granularity cross-modal coupling according to claim 4, characterized in that, The extraction module includes: The generation unit is used to input the target video frame into a preset image segmentation model to generate prior mask features; The first extraction unit is used to extract the four-layer multi-level features of the target video frame and fuse the four-layer multi-level features with the prior mask features to generate the multi-level visual features. The second extraction unit is used to input the target video frame into a pre-built deep learning-based audio feature extraction model to extract the audio Mel-spectral features.

6. The audio-guided visual segmentation device based on multi-granularity cross-modal coupling according to claim 4, characterized in that, The enhancement module includes: The first computing unit is used to input the multi-level visual features into a pre-constructed intramodal noise suppression model to perform channel attention and spatial attention calculations, so as to generate visual channel weights and visual spatial weights. The first weighting unit is used to weight the multi-level visual features using the visual channel weights and visual spatial weights to generate the enhanced multi-level visual features. The second calculation unit is used to input the audio Mel-spectral features into the intramodal noise suppression model to perform channel attention calculation in order to generate audio channel weights; The second weighting unit is used to weight the audio Mel-spectral features using the audio channel weights to generate the enhanced audio Mel-spectral features.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the audio-guided visual segmentation method based on multi-granularity cross-modal coupling as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the audio-guided visual segmentation method based on multi-granularity cross-modal coupling as described in any one of claims 1-3.

Images