Training method, audiovisual segmentation method, electronic device, and storage medium

Abstract

The application relates to a training method, an audio-visual segmentation method, an electronic device and a storage medium. Training samples are obtained, and the samples comprise an audio signal, an image, a target semantic label and a segmentation label. Audio features and image features are extracted. The audio features are subjected to self-attention enhancement based on the target semantic label to impose a target semantic consistency constraint, and enhanced audio features are obtained. Cross-attention reweighting is performed on the unenhanced audio features based on the guidance of the enhanced audio features, and target-oriented audio features are obtained. Bidirectional interaction fusion is performed based on the image features and the target-oriented audio features. A sparse self-attention network is used in the image feature fusion process to suppress the responses of unmatched image regions. Segmentation prediction results are generated based on the fused image features and the audio features, and an audio-visual segmentation model is trained. A video to be processed is input into the trained audio-visual segmentation model, and a segmentation result is output. The application can improve the image segmentation accuracy of a sound-producing target.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to an audiovisual segmentation method, system, electronic device, and storage medium. Background Technology

[0002] Image segmentation is a crucial technique in computer vision, classifying pixels in an image to extract the contours, categories, and spatial locations of targets. As application scenarios have expanded from single static images to complex dynamic scenes, image segmentation methods relying solely on single visual information are insufficient for practical needs, leading to the development of multimodal image segmentation techniques. Among these, audiovisual segmentation aims to combine audio and visual modalities to segment sounding targets within image or video frames and identify their corresponding categories.

[0003] However, existing audiovisual segmentation methods often exhibit strong sensitivity to salient targets in the image during sound source target identification. This means that even when the input audio changes, the model's prediction of the sound source target may remain unchanged. This is because existing methods place too much emphasis on the image modality during the fusion of audio and visual modalities, resulting in insufficient influence of audio features on the segmentation results. Furthermore, relying solely on a single frequency band for analysis can easily overlook important frequency cues that aid in accurate target localization, thus affecting the accuracy of both target localization and segmentation. Therefore, enhancing the preference of audio features for target semantic content and comprehensively utilizing audio information from different frequency bands to improve the segmentation accuracy of sound-producing targets has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a training method, an audiovisual segmentation method, an electronic device, and a storage medium to solve the problem of insufficient image segmentation accuracy in the prior art.

[0005] The present invention provides a training method for training an audiovisual segmentation model, comprising: acquiring training samples, each sample including an audio signal, an image, a target semantic label, and a segmentation label; inputting the audio signal and the image into the audiovisual segmentation model to be trained; extracting audio features and image features; performing self-attention enhancement on the audio features based on the target semantic label to impose a target semantic consistency constraint on the audio features, thereby obtaining enhanced audio features; performing cross-attention reweighting on the unenhanced audio features based on the guidance of the enhanced audio features, thereby obtaining target-oriented audio features; performing bidirectional interactive fusion based on the image features and the target-oriented audio features, thereby obtaining fused image features and audio features, wherein, during the image feature fusion process, a sparse self-attention network is used to suppress the response of image regions that do not match the target-oriented audio features; generating segmentation prediction results based on the fused image features and the fused audio features, and calculating target semantic loss and segmentation loss based on the target semantic label and the segmentation label, respectively, to train the audiovisual segmentation model, thereby obtaining the trained audiovisual segmentation model.

[0006] Furthermore, the extraction of the audio features includes: performing a short-time Fourier transform on the audio signal; The effective range of the frequency domain after the short-time Fourier transform is truncated and divided into multiple adjacent frequency bands. When extracting the audio features, the extraction is performed based on each of the multiple adjacent frequency bands. A Mel domain transform is performed based on the truncated and segmented frequency components to generate a Mel spectrogram, and the audio features are extracted based on the Mel spectrogram.

[0007] Furthermore, the guidance based on the enhanced audio features to perform cross-attention reweighting on the unenhanced audio features includes: generating a first query matrix based on the enhanced audio features, generating a first key matrix and a first value matrix based on the unenhanced audio features, obtaining a first attention weight based on the matching relationship between the first query matrix and the first key matrix, and weighting and converging the first value matrix based on the first attention weight to obtain the target-oriented audio feature.

[0008] Furthermore, the bidirectional interactive fusion includes audio feature fusion that guides the target-oriented audio features based on the image features; and image feature fusion that guides the image features based on the target-oriented audio features.

[0009] Furthermore, the image feature fusion includes: generating a second query matrix based on the image features; generating a second key matrix and a second value matrix based on the target-oriented audio features; obtaining a second attention weight based on the matching relationship between the second query matrix and the second key matrix; weighting and converging the second value matrix based on the second attention weight to obtain an intermediate interaction result, which serves as the input to the sparse self-attention network; converging the output of the sparse self-attention network with the intermediate interaction result; and outputting the fused image features based on the convergence result.

[0010] Furthermore, the audio feature fusion includes: generating a third query matrix based on the target-oriented audio features; generating a third key matrix and a third value matrix based on the intermediate interaction results; obtaining a third attention weight based on the matching relationship between the third query matrix and the third key matrix; and weighting and converging the third value matrix based on the third attention weight to obtain the fused audio features.

[0011] Further, the calculation of the target semantic loss includes: inputting the self-attention-enhanced audio features into a fully connected layer to obtain the target semantic prediction result, and calculating the target semantic loss based on the cross-entropy between the target semantic prediction result and the target semantic label, so that the audio features form a preference for target-related semantic content; the calculation of the segmentation loss includes: based on the fused image features and audio features, changing the number of channels of the prediction mask map through a fully connected layer to obtain the segmentation prediction result, and calculating the segmentation loss based on the cross-entropy between the segmentation prediction result and the segmentation label; weighting and summing the target semantic loss and the segmentation loss to obtain the total loss used to train the audiovisual segmentation model; in one training iteration of the audiovisual segmentation model, the target semantic loss and the segmentation loss are calculated synchronously and backpropagated uniformly.

[0012] Secondly, an audiovisual segmentation method is also provided, comprising: acquiring a video to be processed; inputting audio signals and images from the video to be processed into an audiovisual segmentation model, wherein the audiovisual segmentation model is a trained audiovisual segmentation model obtained according to the training method described in any of the technical solutions of the first aspect; and outputting a segmentation result based on the trained audiovisual segmentation model.

[0013] Thirdly, an electronic device is also provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the technical solutions provided in the first aspect or the method provided in the second aspect.

[0014] Fourthly, a storage medium is also provided, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, it implements the method described in any of the technical solutions provided in the first aspect or the method provided in the second aspect.

[0015] Compared with existing technologies, the above-mentioned technical solution performs self-attention enhancement and target semantic consistency constraints on audio features based on target semantic labels, enabling audio features to form a preference for target-related semantic content. Then, based on the guidance of enhanced audio features, cross-attention reweighting is performed on unenhanced audio features to obtain target-oriented audio features. This reduces the interference of background noise and acoustic components unrelated to the target on segmentation decisions and enhances the representation ability of audio features for the sound-producing target. Furthermore, by performing bidirectional interactive fusion based on image features and target-oriented audio features, the role of audio modalities in the fusion process can be enhanced, and the influence of image saliency bias on sound source localization can be weakened. In addition, by using a sparse self-attention network to suppress the response of image regions that do not match the target-oriented audio features during image feature fusion, the interference of irrelevant image regions on the segmentation results can be reduced, thereby improving the alignment accuracy and segmentation accuracy between the sound-producing target and the image region.

[0016] Compared with existing technologies, some of the above-mentioned technical solutions further improve the robustness of audio features and the accuracy of subsequent segmentation by performing short-time Fourier transform, effective frequency range truncation, and Mel-domain transform on the audio signal, and dividing the effective frequency components into adjacent frequency bands. This allows for the filtering out of low-frequency noise and high-frequency redundant components while retaining the effective frequency components that characterize the target's vocal characteristics and representing more subtle frequency changes. By inputting the self-attention-enhanced audio features into a fully connected layer to obtain the target semantic prediction result, and calculating the target semantic loss based on cross-entropy, the preference of audio features for target-related semantic content can be further strengthened. By limiting the source relationships of the query matrix, key matrix, and value matrix in the bidirectional interactive fusion process, and by using the intermediate interaction results as input to the sparse self-attention network and as the source of keys and values ​​in subsequent audio feature fusion, the targeting and stability of cross-modal interaction can be improved. By changing the number of channels in the prediction mask map through the fully connected layer and by weighted summing of the target semantic loss and the segmentation loss, a more stable training target can be formed, thereby further improving the accuracy of segmentation prediction. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 A flowchart illustrating an audiovisual segmentation training method provided in this application; Figure 2 A schematic diagram of the module architecture of an audiovisual segmentation system provided in this application; Figure 3 A schematic diagram of the network structure of the audiovisual segmentation method provided in this application; Figure 4 This application provides a schematic diagram for target-oriented audio feature extraction. Figure 5 A schematic diagram of the bidirectional interactive fusion structure provided in this application; Figure 6 A schematic diagram of the sparse self-attention network structure provided in this application. Detailed Implementation

[0018] The technical solution of this application will be further described below with reference to the accompanying drawings. It should be noted that the term "acquisition" generally refers to the process of obtaining data or information, which can be obtained directly through sensors, acquisition devices, or interfaces, or by reading, calling, receiving, or parsing stored data. The term "based on" generally refers to a relationship description that uses certain data, conditions, rules, parameters, or models as input or basis for processing, calculation, derivation, or judgment, and does not exclude the simultaneous processing in combination with other data, conditions, rules, parameters, or models. The term "comprising" is intended to cover non-exclusive inclusion, that is, a solution that includes a series of steps, structures, or features is not necessarily limited to the explicitly listed steps, structures, or features, and may also include other steps, structures, or features not explicitly listed but inherent to the solution. The term "at least" is used to indicate a minimum quantity or minimum degree of limitation, and does not exclude situations where there is more than this minimum quantity or higher than this minimum degree. The terms "first," "second," "third," etc., are only used to distinguish the same or similar objects, steps, or features, and are not used to limit a specific order, priority, or degree of importance; they can be used interchangeably without conflict. The term "input" generally refers to feeding signals, data, or features into a model, module, network, or processing flow; the term "output" generally refers to the results, features, or prediction information given by the model, module, network, or processing flow after processing. Unless otherwise expressly defined, the terms used in this application should be interpreted broadly in their usual sense.

[0019] Furthermore, in this application, the term "target semantic label" is used to characterize the semantic information of the vocal target category and can be used to constrain the semantic preferences of audio features; the term "segmentation label" is used to characterize the pixel-level annotation information of the vocal target in the image. The term "self-attention" refers to an attention mechanism that establishes a relationship between different positions within the same feature; the term "cross-attention" refers to an attention mechanism that establishes a matching relationship between features from two different sources and completes information guidance. The terms "query matrix," "key matrix," and "value matrix" correspond to the feature representations used in the attention mechanism to initiate matching, provide matching basis, and carry weighted converged content, respectively. "ResNet50" is an image feature extractor using a classic convolutional neural network, whose network structure is derived from the ResNet (Residual Network) model, with 50 layers; "VGGish" is an audio feature extractor using a classic convolutional neural network, whose network structure is derived from the VGG (Visual Geometry Group) model, mainly used for feature encoding of the log-Melogram of audio to extract high-dimensional features that can characterize the semantic information of sound. The term "Softmax" refers to the normalized exponential function, which typically refers to a function that maps a set of real numbers to a normalized weight distribution. In this application, it is used to convert the matching results into a weight distribution. The term "ReLU" refers to the Rectified Linear Unit, which represents the linear rectified activation function.

[0020] Figure 1 This is a flowchart illustrating an audiovisual segmentation training method provided in this application. Figure 1 As shown, the method may include the following steps: In step S1, training samples are obtained, each sample including an audio signal, an image, a target semantic label, and a segmentation label, and the audio signal and image are input into the audiovisual segmentation model to be trained; In step S2, audio features and image features are extracted, and self-attention enhancement is performed on the audio features based on the target semantic label to impose a target semantic consistency constraint on the audio features, thereby obtaining enhanced audio features; In step S3, cross-attention reweighting is performed on the unenhanced audio features based on the guidance of the enhanced audio features, thereby obtaining target-oriented audio features; In step S4, bidirectional interactive fusion is performed based on the image features and the target-oriented audio features to obtain fused image features and audio features, wherein a sparse self-attention network is used to suppress the response of image regions that do not match the target-oriented audio features during the image feature fusion process; In step S5, a segmentation prediction result is generated based on the fused image features and the fused audio features, and the target semantic loss and segmentation loss are calculated based on the target semantic label and the segmentation label, respectively, to train the audiovisual segmentation model, thereby obtaining the trained audiovisual segmentation model.

[0021] Based on the above method flow, an audiovisual segmentation method flow also needs to include, for example: Figure 1 As shown in step S6, after acquiring the video to be processed, the audio signal and image in the video to be processed are input into the trained audiovisual segmentation model, and the segmentation result is output.

[0022] Figure 2 This is a schematic diagram of the modular architecture of an audiovisual segmentation system provided in this application. Figure 2 As shown, the audiovisual segmentation system 100 may include a sample acquisition module 110, a feature extraction module 120, an audio enhancement module 130, an audio reweighting module 140, a bidirectional interactive fusion module 150, a sparse self-attention module 160, a prediction training module 170, and an inference output module 180. Specifically, the sample acquisition module 110 acquires training samples, including audio signals, images, target semantic labels, and segmentation labels; the feature extraction module 120 extracts audio features and image features based on the audio signals and images; the audio enhancement module 130 performs self-attention enhancement on the audio features based on the target semantic labels to impose target semantic consistency constraints on the audio features, obtaining enhanced audio features; the audio reweighting module 140 performs cross-attention reweighting on unenhanced audio features based on the guidance of the enhanced audio features, obtaining target-oriented audio features; and the bidirectional interactive fusion module 150 performs self-attention enhancement on the unenhanced audio features based on the image features and... The target-oriented audio features are subjected to bidirectional interactive fusion; the sparse self-attention module 160 is used to suppress the response of image regions that do not match the target-oriented audio features during the image feature fusion process, so as to obtain fused image features; the prediction training module 170 is used to generate segmentation prediction results based on the fused image features and the fused audio features, and calculate the target semantic loss and segmentation loss based on the target semantic label and the segmentation label, respectively, to train the audiovisual segmentation model and obtain the trained audiovisual segmentation model; the inference output module 180 is used to input the audio signal and image in the video to be processed into the trained audiovisual segmentation model and output the segmentation result.

[0023] Figure 3 This is a schematic diagram of the network structure for the audiovisual segmentation method provided in this application. Figure 3 As shown, the input image is processed by ResNet50 to extract multi-scale image features. These multi-scale image features are then concatenated and convolved to form the final image features. Simultaneously, the input audio signal is subjected to target semantic consistency constraints through self-attention enhancement and guided by the enhanced audio features, forming target-oriented audio features. On the image side, multi-scale image features can be further processed through multiple layers of 1×1 convolutions and 3×3 convolutions with different dilation rates (the dilation rate is set to, for example, 10 ... Figure 3 The image features are processed using rate6, rate12, and rate18 layers, average pooling layers, and upsampling to extract and integrate multi-scale contextual information; subsequently, image features are... Target-oriented audio features The input is shared by an interactive fusion process, and the output is a segmentation result. Figure 3 The network structure shown can preserve the multi-scale semantic information of the image while introducing audio information obtained by adapting to the target frequency constraint into the audiovisual segmentation process, thereby improving the segmentation accuracy of the sound target.

[0024] In step S2 above, the extraction of audio features includes the following: A short-time Fourier transform is performed on the audio signal of the input system to convert it into spectral features that can simultaneously represent time and frequency information; the effective range of the frequency domain after the short-time Fourier transform is truncated to filter out low-frequency noise components and high-frequency redundant components, retaining the effective frequency components used to represent the sound characteristics of the sound-producing target in the current audio / video signal; a Mel-domain transform is performed based on the truncated frequency components to generate a Mel spectrogram, thereby extracting audio features based on the generated Mel spectrogram.

[0025] In one embodiment, for continuous-time audio signals Perform short-time Fourier transform In this case, the following formula can be used: Where t represents the center time point of the window function, corresponding to the core moment of the current analysis time frame. This represents the integral dummy variable, used to iterate through the sampling points of the audio signal across the entire time axis. This represents the angular frequency dimension parameter, corresponding to the frequency components of the audio signal. This represents the sliding window function after translation. Through short-time Fourier transform, audio signals that originally existed only in the time domain can be transformed. Transform into a system that includes both time dimension parameter 𝑡 and frequency dimension parameter 𝑡. The time-frequency domain signal is used to achieve a joint representation of the three-dimensional information of the audio signal: time, frequency, and energy. In a preferred embodiment, The Hanning window is used as the sliding window function. The specific expression of the Hanning window function is as follows: Where T represents the window length, i.e. the duration of a single analysis time frame. The Hanning window exhibits symmetrical cosine decay characteristics in the interval [0,T], with its amplitude smoothly converging to zero at both ends. This effectively suppresses spectral leakage after Fourier transform and reduces mutual interference between frequency components of adjacent time frames, thereby ensuring the short-term stationarity of local audio signal analysis.

[0026] In the above embodiments, after the short-time Fourier transform is completed, the frequency domain can be further truncated to filter out low-frequency noise components and high-frequency redundant components, retaining the effective frequency components used to characterize the target sound characteristics. Specifically, the continuous domain angular frequency ω can first be converted into physical frequencies to establish a correspondence between the frequency domain and actual auditory perception; then, low-frequency components below a preset lower limit and high-frequency components above a preset upper limit are removed. Low-frequency components typically contain environmental noise and residual DC offset, contributing little to effective audio features, while high-frequency components often exceed the core frequency range of most speech or commonly used audio, and the human ear has low resolution in this frequency band, resulting in more redundant information. Finally, the truncated effective frequency components are retained as the input basis for subsequent Mel-domain transform. Preferably, the frequency range corresponding to the effective frequency components is 125Hz to 7500Hz. By truncating the effective frequency domain, the amount of data computation can be reduced, processing efficiency improved, and invalid redundant information filtered out, thus enhancing the robustness of audio features.

[0027] In some preferred embodiments, to further characterize more detailed frequency variations, the frequency range corresponding to the effective frequency components can be divided into multiple adjacent frequency bands, and frequency features can be extracted based on each adjacent frequency band. Specifically, dividing the effective frequency components into three adjacent frequency bands can be done as 125Hz–2500Hz, 2500Hz–5000Hz, and 5000Hz–7500Hz. By dividing the effective frequency range into multiple adjacent frequency bands and extracting frequency features separately, the target acoustic characteristics carried by different frequency bands can be more precisely characterized, thereby providing more discriminative audio input for subsequent audio feature enhancement, target semantic consistency constraints, and cross-modal fusion.

[0028] In some embodiments, after the effective frequency components are truncated and divided as described above, 64 Mel filters are used to perform Mel domain mapping on the effective frequency components, so that different Mel frequency channels correspond to the logarithmic energy characteristics of different frequency bands. To meet the engineering requirements of batch processing audio data, the Mel spectral features corresponding to a single audio track can be batch-integrated to obtain a batched Mel spectrogram. Its tensor dimension can be represented as [B, 64, L], where each element... Corresponding to the b-th audio sample, the p-th Mel audio channel, and the... The logarithmic energy characteristics of each time frame are obtained, where B represents the batch dimension, a positive integer corresponding to the number of audio samples processed simultaneously. Those skilled in the art can flexibly adjust this value according to the actual hardware computing power, such as using 32 or 16; the second dimension, 64, represents the Mel frequency dimension, corresponding to the sixty-four Mel filters constructed in this embodiment; and L represents the time dimension, corresponding to the number of time frames after the audio signal is discretized. Therefore, the obtained Mel spectrogram can completely retain the core feature information of the audio signal in the three dimensions of "batch—Mel frequency—time".

[0029] In some embodiments, to improve the robustness of the audiovisual segmentation model to scale changes, viewpoint changes, and illumination perturbations, the original images input during the training phase are... and its pixel-level annotation Perform consistent image enhancement operations to obtain the network input image. With the corresponding supervised label Y. First, the input size space is preset with height H and width W, where W represents the number of pixels in the horizontal direction of the image, and H represents the number of pixels in the vertical direction of the image; image normalization parameters are set, including channel mean ( ) and standard deviation ( This is used for normalization, making the input distribution more uniform, which is beneficial for model convergence. and Random horizontal flipping and random scaling are performed, with a scaling factor s randomly selected from a preset scale set S to scale the image and annotations synchronously. Color perturbations are added, where the perturbation ranges for brightness, contrast, saturation, and hue can be represented as: brightness=0.5, contrast=0.5, saturation=0.5, hue=0.25, respectively. These steps simulate changes in lighting and differences in camera imaging, improving the model's generalization ability. The enhanced image is converted into a tensor, with pixels normalized to the range [0,1]. The annotations are converted into integer tensors for pixel-level supervision. Through the above image enhancement processing, random flipping, random scaling, color perturbation, and random cropping are introduced during the training phase, and a consistent geometric transformation is performed on the image and pixel-level annotations, ultimately generating an input image with fixed size and normalized values. This significantly improves the model's robustness to scale changes, viewpoint shifts, and lighting perturbations.

[0030] Specifically, let the original image to be processed be G, which is either a two-dimensional grayscale image or a three-dimensional color image (using the RGB color space by default), with an image resolution of W×H. For a grayscale image, the pixel value matrix of G is G(x,y) ∈[0,255], x∈[0,W-1], y∈[0,H-1]; for an RGB color image, the pixel value matrix of G is G(x,y,c), c∈{R,G,B}, corresponding to the red, green, and blue color channels respectively, with each channel having a pixel value range of [0,255]. Let the set of all adjustable parameters in the feature enhancement process be Ω, which includes core parameters such as noise reduction coefficients, contrast gain, and edge enhancement weights. Each parameter can be flexibly adjusted according to the quality of the original image, such as noise intensity and blur degree, to ensure that the enhancement effect is adapted to the original image in different scenarios. The final output input image I has the same resolution as the original image G, including maintaining the same W×H, and the pixel value range is uniformly standardized to [0,255] to avoid numerical overflow affecting subsequent processing. The grayscale image output is a two-dimensional pixel matrix I(x,y), and the color image output is a three-dimensional pixel matrix I(x,y,c). The key features of I, such as edges, texture, and brightness distribution, are significantly improved compared to G, and the noise content is lower than the preset threshold.

[0031] Figure 4 This is a schematic diagram illustrating the target-oriented audio feature extraction method provided in this application. (See diagram below.) Figure 4 As shown, the input audio signal is first converted into a spectral representation corresponding to multiple frequency bands. The spectrum of each frequency band is then input into VGGish to extract frequency band features. After that, the multiple frequency band features are concatenated and processed by 1×1 convolution to form unenhanced audio features represented in the real number field. In this context, B represents the batch dimension, and C represents the number of channels or the feature dimension, characterizing the feature expressive power. A larger C indicates richer semantic and texture patterns that can be encoded. W and H represent the resolution of the unenhanced audio feature map in the two spatial dimensions, respectively. Combining the input form of the audio signal after spectral transformation in this application, W and H can correspond to two dimensions on the time-frequency plane, representing the feature distribution in the time and frequency directions, or representing the size of the corresponding feature map after transformation by the feature extraction network. By representing the unenhanced audio features using the above tensor form, the organization of audio features in the batch, channel, and time-frequency spatial dimensions can be described more clearly. In this process, multiple frequency bands correspond to the adjacent frequency band division results within the aforementioned effective frequency range, thereby enabling the sound information in different frequency bands to be modeled separately and uniformly represented in subsequent processing. Figure 4 As shown, this forms unenhanced audio features. Then, a self-attention layer is applied to enhance the audio features, establishing internal correlations between the audio features in the time and frequency dimensions. The self-attention enhancement process can be represented by the following formula: , where d head The dimension parameter represents the attention head, and SA represents the function symbol characterizing the self-attention mechanism. This represents matrix multiplication. , representing the learnable parameters for generating the query, key, and value, respectively, can all be automatically learned during the training phase through backpropagation and loss function optimization. Specifically, the learnable parameters are... Figure 4 The self-attention layer shown can be represented by three sets of learnable linear mapping matrices, used to represent the input features. Mapping from the original channel dimension C to the subspace dimension d required for attention computation. head Generate query Q, key K, and value V respectively, where the query Q vector is calculated in the form of: In the field of deep learning, it represents "what information the current position wants to obtain from other positions," and the calculation form of the key K vector is as follows: This represents "what information each location can provide" and is used for similarity matching with the query vector. The value vector V is calculated in the form of... This is used to carry the information of the actual content that is weighted and aggregated. The above three sets of parameters enable the self-attention layer to be trainable and adaptable based on input features, rather than relying on fixed rules. In the above formula, d... head Commonly used Calculation, where For the number of attention heads, The vector lengths of Q, K, and V are determined, representing the similarity calculation performed by the attention in a certain dimensional space. Their function is to scale the dot product, avoiding the problem of QK becoming larger as the dimensionality increases. T If the numerical range is too large, the output of the softmax function becomes too "sharp," resulting in a smaller or unstable gradient; excessive focus reduces generalization ability, and dividing by... This allows the probability scale to be adjusted to a more suitable range, resulting in a smoother softmax distribution and more stable training. Through the above calculations, global dependencies can be established between audio features across different time frames and frequency components, thereby obtaining enhanced audio features. Compared to directly processing the original audio features, the audio features enhanced by self-attention can more fully aggregate contextual information related to the target vocalization behavior, providing a more stable feature foundation for subsequent semantic constraints and reweighting.

[0032] Furthermore, to enable audio features to form a preference for target-related semantic content as early as possible, target semantic consistency constraints can be imposed on the enhanced audio features based on the target semantic labels. Specifically, the enhanced audio features can be... Inputting into a fully connected (FC) layer yields the target semantic prediction result. This process can be represented by the following formula: ,in, and These represent the weight and bias parameters of the fully connected layer, respectively. Then, based on the target semantic prediction results... The target semantic loss is calculated by cross-entropy between the target semantic label and the target semantic label. This is to encourage enhanced audio features to more closely approximate the semantic distribution of the true target category. Target semantic loss. The formula is expressed as: , in, This represents the number of target categories, specifically the number of target categories present in the current dataset. The category label representing the current object target. This represents the cross-entropy function. This constraint reduces the interference of background noise and target-irrelevant acoustic components on the audio representation, enabling the enhanced audio features to exhibit target-oriented semantic preferences early in training.

[0033] In some embodiments, after applying the target semantic consistency constraint, it may also be based on enhanced audio features. The guidance for unenhanced audio features Perform cross-attention reweighting, including: based on enhanced audio features Generate the first query matrix based on unenhanced audio features. A first key matrix and a first value matrix are generated. First attention weights are obtained based on the matching relationship between the first query matrix and the first key matrix. Then, the first value matrix is ​​weighted and converged based on these first attention weights to obtain the target-oriented audio features. The above process can be represented by the following formula: ,in, Represents the cross-attention function. All of these are learnable parameters for generating queries, keys, and values ​​through cross-attention. The first query matrix mentioned above is... The first key matrix is The first value matrix is The first attention weight is As can be seen from the above equation, in this cross-attention process, the audio features are enhanced. Used to provide target-related query information, without enhanced audio features. This provides key and value information, enabling selective reweighting of the original audio representation based on target semantic preferences. This reweighting process further highlights acoustic segments and temporal positions consistent with the target semantics, suppresses the influence of irrelevant frequency components and temporal segments, and ultimately forms target-oriented audio features more suitable for subsequent cross-modal interactive fusion. .

[0034] Based on the detailed explanation above and Figure 4 As can be seen, in the embodiments of this application, the extraction of target-oriented audio features can be understood as first extracting audio features based on multiple adjacent frequency bands, then enhancing the internal expressive power of the audio features through self-attention enhancement, then strengthening the preference of the audio features for target-related semantic content through target semantic consistency constraints, and further performing cross-attention reweighting on the unenhanced audio features based on the guidance of the enhanced audio features, thereby obtaining the target-oriented audio features. The obtained target-oriented audio features It can be used as audio input in subsequent image-audio bidirectional interactive fusion processes to support accurate segmentation of the sound-producing target.

[0035] Figure 5 A schematic diagram of the bidirectional interactive fusion structure provided in this application. In step S4 above, the bidirectional interactive fusion process includes: image-guided audio feature fusion based on image features to target-oriented audio features; and audio-guided image feature fusion based on target-oriented audio features to image features. Specifically, as... Figure 5 As shown, image features Target-oriented audio features The bidirectional interactive fusion structure, with common input, achieves bidirectional information transfer between the visual and audio modalities through two interconnected interaction paths, thereby obtaining fused image features and fused audio features. Specifically, the audio-guided image feature fusion process includes: generating a second query matrix based on image features; generating a second key matrix and a second value matrix based on target-oriented audio features; obtaining a second attention weight based on the matching relationship between the second query matrix and the second key matrix; and weighted aggregation of the second value matrix based on the second attention weight to obtain an intermediate interaction result, which serves as the input to the sparse self-attention network. The image-guided audio feature fusion process includes: generating a third query matrix based on target-oriented audio features; generating a third key matrix and a third value matrix based on the intermediate interaction result; obtaining a third attention weight based on the matching relationship between the third query matrix and the third key matrix; and weighted aggregation of the third value matrix based on the third attention weight to obtain the fused audio features.

[0036] Figure 6 A schematic diagram of the sparse self-attention network structure provided in this application. Figure 5 The image-side path in the bidirectional interactive fusion structure shown can also be combined with Figure 6 The sparse self-attention layer formed by the sparse self-attention network shown is used for processing. The above-mentioned bidirectional interactive fusion process also includes: converging the output of the sparse self-attention network with the intermediate interaction results, and outputting the fused image features based on the convergence result.

[0037] The following is combined with Figure 5 and Figure 6 The above-mentioned two-way interactive fusion process will be explained in detail.

[0038] First of all, the In the bidirectional interactive fusion of layers, the output of the sparse self-attention network can be represented as The input audio features can be represented as The input image features can be represented as In some embodiments, during the audio-guided image feature fusion process described above, the first... The image features output by the layer audio-guided image decoder are represented as follows In other embodiments, the above process also includes the aforementioned intermediate interaction results. The process is in Figure 5 The element marked with a dashed box is an AGV, which can be represented by the following formula: ,in, All are learnable parameters. That is, based on the first Query corresponding to layer image features The second query matrix obtained through linear mapping, That is, based on the first Keys corresponding to layer audio features The second bond matrix obtained through linear mapping, That is, based on the first Values ​​corresponding to layer audio features The second value matrix obtained through linear mapping is used to derive the second attention weights using the softmax function. As shown in this formula, during this interaction, the visual features from the image are used to provide query information, while the audio features output by the previous layer's image-guided audio decoder provide key and value information, thus enabling the target-oriented audio features to guide the update of image features. Therefore, the intermediate interaction results formed on the visual side not only represent the established cross-modal correspondence but also serve as the source of keys and values ​​in subsequent audio-side interactions, supporting further fusion of audio features.

[0039] In some embodiments, the output of the image-guided audio feature fusion process described above can be represented as the audio features output by the image-guided audio decoder of layer n. The process is in Figure 5 The area marked with a dashed box as VGA can be represented by the following formula: The matrix correspondence in this formula is similar to that of the AGV mentioned above, and will not be elaborated here. As can be seen from this formula, in this interaction process, the audio features of the previous layer are used to provide query information, while the visual features output by the other interaction path, i.e., the intermediate interaction result, are used to provide key and value information. Thus, the audio features are updated based on the cross-modal attention relationship to obtain the first... Layered image-guided audio decoder output audio features Through this process, structural and spatial contextual information from the image modality can be incorporated into the audio feature representation, thereby enhancing the role of audio features in the overall fusion stage.

[0040] In some embodiments, the two-way interactive fusion module 150 may further include a first interactive unit and a second interactive unit, wherein the first interactive unit can be referred to as Figure 5 The AGV shown is constructed to complete the first round of cross-modal interaction between the image side and the target-oriented audio features. The second interaction unit can be constructed by referring to... Figure 5 The VGA shown is used to construct a second round of cross-modal interaction between the audio side and the intermediate interaction result. Through the collaborative processing of the above two interaction units, the hierarchical information transmission and fusion between the visual modality and the audio modality can be realized within the bidirectional interaction fusion module 150. The processing of the first interaction unit can be compared with... Figure 5 This corresponds to the audio-guided image feature fusion process shown. Specifically, the first interaction unit generates a second query matrix based on image features, a second key matrix and a second value matrix based on target-oriented audio features, obtains a second attention weight based on the matching relationship between the second query matrix and the second key matrix, and performs weighted aggregation on the second value matrix based on the second attention weight to obtain an intermediate interaction result. Through this intermediate interaction result, target-related semantic information in the target-oriented audio features can be introduced into the image feature update process, enabling the visual features to initially possess cross-modal representation capabilities corresponding to the current target audio. The processing of the second interaction unit can be compared with... Figure 5The image-guided audio feature fusion process shown corresponds to this. Specifically, the second interaction unit generates a third query matrix based on the target-oriented audio features, generates a third key matrix and a third value matrix based on the intermediate interaction results, obtains a third attention weight based on the matching relationship between the third query matrix and the third key matrix, and performs weighted aggregation on the third value matrix based on the third attention weight to obtain the fused audio features. Thus, the audio side can further utilize the intermediate interaction results formed after the first round of interaction for feature updates, thereby enhancing the alignment between audio features and visual features. In this embodiment, the intermediate interaction results serve as input to the sparse self-attention module 160 to suppress responses from image regions that do not match the target-oriented audio features through the sparse self-attention network; on the other hand, the intermediate interaction results also serve as the source of the third key matrix and the third value matrix in the second interaction unit. That is, the intermediate interaction results simultaneously play a dual role in image-side filtering and audio-side updating. By using the intermediate interaction results simultaneously for image feature optimization and audio feature fusion, not only can the computational overhead caused by the repeated construction of intermediate information be reduced, but the information transmission between the image side and the audio side during the bidirectional interaction process can also be kept consistent, thereby improving the fusion stability and overall segmentation performance of the bidirectional interactive fusion module 150.

[0041] like Figure 6 As shown, the sparse self-attention layer, based on the dense self-attention computation, also introduces a ReLU branch and uses learnable parameters. and (correspond Figure 6 In and The outputs of the Softmax and ReLU branches are weighted to weaken the response of image regions that do not match the audio features pointed to by the current target, while preserving the dense self-attention approach, thanks to the one-sided suppression effect of ReLU. Specifically, the intermediate interaction results are first normalized and linearly mapped to generate queries, keys, and values, which are then fed into the Softmax and ReLU branches respectively. After sparse self-attention processing, they are converged with the intermediate interaction results to output the fused image features. The formula for the above process is expressed as: Where Q, K, and V represent the intermediate interaction results, respectively. The query, key, and value obtained through layer normalization and linear mapping; d head represents the feature dimension of the attention head; P represents the positional encoding or positional bias term. and This represents the learnable weighted parameters corresponding to the Softmax and ReLU branches; This represents a linear mapping function. By introducing the one-sided suppression property of the ReLU branch, the response of image regions that do not match the target audio features can be further weakened. Through this processing, while preserving cross-modal correspondences, interference from mismatched image regions can be further filtered out, improving the discriminative ability of image features.

[0042] In some embodiments, the output of the sparse self-attention layer can be further processed by residual connections, normalization, and / or a feedforward neural network (FFN) before being converged with intermediate interaction results to output fused image features. The above process... Figure 5 The text is marked with a dashed box and refers to SSA, or Sparse Self-Attention. Through the above processing, on the one hand, the cross-modal correspondence between the image and audio already established in the intermediate interaction results can be preserved; on the other hand, interference from irrelevant image regions can be further filtered out, improving the consistency of image features' response to target-oriented audio features, thereby providing more accurate fused image features for subsequent segmentation prediction.

[0043] In the embodiments provided in this application, the training phase generates segmentation prediction results based on the fused image and audio features, and calculates the target semantic loss and segmentation loss based on the target semantic label and segmentation label, respectively, for model training. This includes: changing the number of channels in the prediction mask image through a fully connected layer to obtain the segmentation prediction results; calculating the segmentation loss based on the cross-entropy of the segmentation prediction results and the segmentation labels; and weighted summing the target semantic loss and the segmentation loss to obtain the total loss used to train the audiovisual segmentation model. The total loss in the above process... ,in, and These are all adjustable hyperparameters. By adjusting the hyperparameters, the balance between the impact of the target semantic loss and the segmentation loss can be adjusted. Indicates the target semantic loss. Represents the segmentation loss, and ,in, This represents the cross-entropy function, where N represents the total number of samples. To obtain the prediction result by changing the number of channels in the prediction mask image through a fully connected layer, For segmentation labels. In this embodiment, during one training iteration of the above-mentioned audiovisual segmentation model, the target semantic loss and segmentation loss are calculated simultaneously and backpropagated in a unified manner, thereby achieving end-to-end joint training of classification and segmentation tasks, rather than separate training of each other independently.

[0044] In some embodiments, this application also provides an electronic device. The electronic device may include a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor may be one or more of a central processing unit, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. The memory may include volatile memory and / or non-volatile memory. The processor executes the steps of the aforementioned audiovisual segmentation method by calling the computer program stored in the memory, including: acquiring training samples, extracting audio features and image features, performing self-attention enhancement on the audio features based on target semantic labels to impose target semantic consistency constraints, performing cross-attention reweighting on the unenhanced audio features based on the guidance of the enhanced audio features, performing bidirectional interactive fusion based on image features and target-oriented audio features, and using a sparse self-attention network to suppress the response of image regions that do not match the target-oriented audio features during image feature fusion, thereby generating a segmentation prediction result and completing model training; or, in the inference stage, inputting the audio signal and image from the video to be processed into the trained audiovisual segmentation model and outputting the segmentation result. Thus, the aforementioned audiovisual segmentation method can be implemented by an electronic device through software, hardware, or a combination of both.

[0045] In some embodiments, this application also provides a storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned audiovisual segmentation method. The computer-readable storage medium can be a disk, optical disk, read-only memory, random access memory, flash memory, solid-state drive, mobile memory card, or other tangible medium capable of storing program code. By storing the program code for implementing the aforementioned audiovisual segmentation method in a computer-readable storage medium, when an electronic device calls and executes the computer program, it can complete processes such as training sample acquisition, audio feature extraction, target semantic consistency constraint, target-oriented audio feature generation, bidirectional interactive fusion, segmentation prediction result generation, and model training and inference output, thereby realizing the technical solution corresponding to the aforementioned audiovisual segmentation method.

[0046] Based on the above embodiments, this application conducted comparative experiments on the VPO dataset and the AVSBench-semantic dataset. The following are the experimental setup and results provided in this application. In the VPO dataset experiment, based on the audiovisual segmentation method and system provided in the above embodiments, ResNet-50 was used as the image feature extraction skeleton, and VGGish was used as the audio feature extractor. The training period was set to 80. During training, the SGD optimizer was used with an initial learning rate of 0.005, momentum of 0.9, batch size of 12, and a learning rate decay rate of 5e-4. For all datasets, the dimension of the image input features was [3, 512, 512], indicating 3 channels and 512 dimensions. The audio feature dimension was [2, 128], where 2 represents stereo and 128 represents audio content features. The metrics used in the VPO series experiments were as follows: , FDR. In this embodiment, Where N represents the number of categories, This indicates that it has been correctly classified into a category. The number of pixels, Indicates that it was incorrectly predicted as The number of pixels that belong to one category but actually belong to another. This refers to the true category as However, the number of pixels not predicted as belonging to this category; ,in Indicates accuracy rate. Recall rate, in this experiment The evaluation metric is 0.3. In the AVSBench-semantic data experiment, the training epoch was set to 60. During training, the SGD optimizer was used with an initial learning rate of 0.005, momentum of 0.9, batch size of 16, and learning rate decay of 5e-4. For all datasets, the dimension of the image input features is [3, 512, 512], indicating that the number of channels is 3 and the length and width are 512. The dimension of the audio features is [T, 128], where T=5 represents the number of video frames extracted, and the first frame of each second is taken from a 5-second video. The evaluation metric used is... , And the average of the two.

[0047] Based on the above experimental setup, the experimental results for TPAVI, AVSegFormer, CAVP, and the method of this application on the single-source target dataset VPO-SS are as follows: TPAVI's mIoU is 52.75, The value is 69.54, and the FDR is 22.83; the mIoU of AVSegFormer is 57.55. The value was 73.03, and the FDR was 19.76; the mIoU of CAVP was 62.31. The mIoU of the method in this application is 78.46, and the FDR is 13.56; the mIoU of the method in this application is 69.70. The mIoU is 79.33 and the FDR is 11.15. Therefore, on the VPO-SS dataset, the proposed method improves the mIoU by 7.39% and reduces the FDR by 2.41% compared to the baseline model CAVP, indicating that in single-source scenarios, the proposed method can more accurately segment the sound source target while reducing the error segmentation rate. Combining the model structure of the proposed method with the baseline model structure, it can be inferred that this improvement mainly benefits from the selection and utilization of audio frequency information, suppressing noise and redundant components in various frequency information.

[0048] On the multi-source target dataset VPO-MS, the experimental results of TPAVI, AVSegFormer, CAVP, and the method in this application are as follows: TPAVI's mIoU is 54.30, The value is 71.95, and the FDR is 22.45; the mIoU of AVSegFormer is 58.33. The value was 74.28, and the FDR was 22.13; the mIoU of CAVP was 64.31. The mIoU of the method in this application is 78.92, and the FDR is 18.67; the mIoU of the method in this application is 68.83. The mIoU is 81.79 and the FDR is 15.43. It can be seen that in multi-source scenarios, the method in this application still achieves good performance in both mIoU and FDR. The method outperforms TPAVI, AVSegFormer, and CAVP in three metrics: accuracy, frequency, and frequency response (FDR), indicating that the proposed method not only improves segmentation accuracy but also maintains good robustness under multi-source interference conditions.

[0049] On the synonymous multi-source dataset VPO-MSMI, the experimental results for TPAVI, AVSegFormer, CAVP, and the method in this application are as follows: TPAVI's mIoU is 51.73, The value is 68.85, and the FDR is 26.75; the mIoU of AVSegFormer is 54.22. The value was 70.39, and the FDR was 25.51; the mIoU of CAVP was 60.36. The value is 75.60, and the FDR is 22.12; the mIoU of the method in this application is 62.17. The mean squared efficiency (mIoU) is 77.70, and the free-radius density (FDR) is 17.99. It is evident that, in the more complex scenario of synonymous multiple sound sources, the proposed method still achieves higher mIoU and FDR compared to CAVP. Meanwhile, the FDR further decreased, indicating that the method of this application can still distinguish targets well and suppress erroneous predictions when multiple similar sound-emitting targets coexist.

[0050] Furthermore, on the AVSBench-semantic dataset, the experimental results of TPAVI, AVSAC, BAVS, AVSegFormer, SelM, CAVP, and the method in this application are as follows: the average value of TPAVI is 22.69. It is 20.18. The average value for AVSAC is 25.20; the average value for AVSAC is 27.55. It is 25.40. The average value is 29.72; the average value of BAVS is 27.15. It is 24.68. The average value is 29.63; the average value for AVSegFormer is 27.12. It is 24.93. The average value of SelM is 29.30; the average value of SelM is 34.50. It is 31.98. The average value was 37.22; the average value of CAVP was 32.38. It is 30.37. The average value is 35.29; the average value of the method in this application is 34.60. It is 31.80. The score is 37.50. Therefore, the proposed method outperforms TPAVI, AVSAC, BAVS, AVSegFormer, and CAVP on the AVSBench-semantic dataset, and improves upon CAVP's average score by 2.22. An improvement of 2.21, which is also slightly better than the average of SelM. The metrics demonstrate that the proposed method is not only applicable to audio-guided image segmentation tasks, but also has strong applicability in audio-guided video segmentation semantic scenarios. A comprehensive comparison of the model structure of the proposed method with existing models reveals that the proposed method helps align audio and image modalities and reduces the gap between different modalities through audio frequency enhancement, adaptive constraints on target features, and bidirectional audio-video interaction fusion.

[0051] In summary, the method proposed in this application demonstrates superior segmentation performance compared to various existing methods in single-source, multi-source, synonymous multi-source, and audiovisual semantic segmentation scenarios. This indicates that the method proposed in this application has good effectiveness and stability in enhancing audio semantic preference, suppressing interference from irrelevant frequency components, and improving cross-modal alignment.

[0052] In summary, this application provides a training method, an audiovisual segmentation method, a system, an electronic device, and a storage medium. By performing self-attention enhancement on audio features and imposing target semantic consistency constraints, the audio features are made to form a preference for target-related semantic content. Then, based on the guidance of the enhanced audio features, cross-attention reweighting is performed on the unenhanced audio features to obtain target-oriented audio features. Finally, bidirectional interactive fusion is performed based on image features and target-oriented audio features. During the image feature fusion process, a sparse self-attention network is used to suppress the response of image regions that do not match the target-oriented audio features, thereby improving the accuracy of target localization and segmentation.

[0053] The above embodiments are merely illustrative of the technical concept and features of the present invention, intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and should not be construed as limiting the scope of protection of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A training method for training an audiovisual segmentation model, characterized in that, include: Acquire training samples, each sample including audio signal, image, target semantic label and segmentation label, and input the audio signal and image into the audiovisual segmentation model to be trained; Audio and image features are extracted, and self-attention enhancement is performed on the audio features based on the target semantic label to impose a target semantic consistency constraint on the audio features, thereby obtaining enhanced audio features; Guided by the enhanced audio features, cross-attention reweighting is performed on the unenhanced audio features to obtain target-oriented audio features. The cross-attention reweighting uses the enhanced audio features to provide target-related query information and the unenhanced audio features to provide key and value information. Based on the image features and the target-oriented audio features, a bidirectional interactive fusion is performed to obtain fused image features and audio features. During the bidirectional interactive fusion process, an intermediate interactive result is formed. In the image feature fusion process, a sparse self-attention network is used to suppress the response of image regions that do not match the target-oriented audio features. The intermediate interactive result serves as the input of the sparse self-attention network and as the source of keys and values ​​in the audio feature fusion process. Segmentation prediction results are generated based on the fused image features and fused audio features. Target semantic loss and segmentation loss are calculated based on the target semantic label and segmentation label, respectively, to train the audiovisual segmentation model and obtain the trained audiovisual segmentation model.

2. The training method according to claim 1, characterized in that, The extraction of the audio features includes: Perform a short-time Fourier transform on the audio signal; The effective range of the frequency domain after the short-time Fourier transform is truncated and divided into multiple adjacent frequency bands. When extracting the audio features, the extraction is performed based on each of the multiple adjacent frequency bands. A Mel domain transform is performed on the truncated and segmented frequency components to generate a Mel spectrogram, and the audio features are extracted based on the Mel spectrogram.

3. The training method according to claim 1, characterized in that, The method of guiding the unenhanced audio features based on the enhanced audio features to perform cross-attention reweighting on the unenhanced audio features includes: generating a first query matrix based on the enhanced audio features, generating a first key matrix and a first value matrix based on the unenhanced audio features, obtaining a first attention weight based on the matching relationship between the first query matrix and the first key matrix, and weighting and converging the first value matrix based on the first attention weight to obtain the target-oriented audio features.

4. The training method according to claim 1, characterized in that, The bidirectional interactive fusion includes image-guided audio feature fusion based on the image features and image-guided image feature fusion based on the target-oriented audio features; and audio-guided image feature fusion based on the target-oriented audio features and image features.

5. The training method according to claim 4, characterized in that, The image feature fusion includes: generating a second query matrix based on the image features; generating a second key matrix and a second value matrix based on the target-oriented audio features; obtaining a second attention weight based on the matching relationship between the second query matrix and the second key matrix; and weighting and converging the second value matrix based on the second attention weight to obtain an intermediate interaction result. The output of the sparse self-attention network is combined with the intermediate interaction results, and the fused image features are output based on the combined results.

6. The training method according to claim 5, characterized in that, The audio feature fusion includes: generating a third query matrix based on the target-oriented audio features; generating a third key matrix and a third value matrix based on the intermediate interaction results; obtaining a third attention weight based on the matching relationship between the third query matrix and the third key matrix; and weighting and converging the third value matrix based on the third attention weight to obtain the fused audio features.

7. The training method according to claim 1, characterized in that, The calculation of the target semantic loss includes: inputting the self-attention-enhanced audio features into a fully connected layer to obtain the target semantic prediction result, and calculating the target semantic loss based on the cross-entropy between the target semantic prediction result and the target semantic label, so that the audio features form a preference for target-related semantic content; The calculation of the segmentation loss includes: based on the fused image features and audio features, changing the number of channels in the prediction mask image through a fully connected layer to obtain the segmentation prediction result, and calculating the segmentation loss based on the cross-entropy of the segmentation prediction result and the segmentation label; The target semantic loss and the segmentation loss are weighted and summed to obtain the total loss used to train the audiovisual segmentation model; In one training iteration of the audiovisual segmentation model, the target semantic loss and the segmentation loss are calculated synchronously and backpropagated in a unified manner.

8. An audiovisual segmentation method, characterized in that, include: The video to be processed is obtained, and the audio signal and image in the video to be processed are input into the audiovisual segmentation model, wherein the audiovisual segmentation model is a trained audiovisual segmentation model obtained by the training method of the audiovisual segmentation model according to any one of claims 1 to 7. Based on the trained audiovisual segmentation model, the segmentation result is output.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7 or the method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7 or the method of claim 8.

Images