Vision-centered video sounding object segmentation method

Abstract

The invention discloses a vision-centered video sounding object segmentation method, and relates to the technical field of artificial intelligence and computer vision, and the method comprises the steps: firstly obtaining audio features and multi-scale visual features corresponding to an audio and a video; inputting the first-level feature with the highest resolution in the multi-scale visual features into a prototype prompt type query generation module to obtain a visual derivation query; inputting the generated visual derivation query into an iterative audio and video interaction module, and performing iterative interaction with audio features and multi-scale visual features so as to gradually optimize the information expression capability of the visual derivation query; finally, the optimized visual derivation query is used to generate a final prediction result, including a pixel-level segmentation mask of the sounding object of each frame and an audio event category corresponding to each mask. According to the method, multi-modal data containing video and audio information can be analyzed, pixel-level accurate segmentation of an actual sounding object in a video frame is realized, and identification of a sound source type is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a method for segmenting sound-emitting objects in a video based on vision. Background Art

[0002] With the explosive growth of multimedia content and the continuous improvement of computer vision and audio signal processing capabilities, research on the multimodal fusion of vision and audio has become a hot topic. Traditional segmentation methods based solely on vision cannot effectively distinguish between objects that actually make sounds and those that do not. When localization relies solely on audio signals (such as simple sound localization methods), it is difficult to accurately locate the corresponding visual area in the video frame because the audio signal often contains a mixture of multiple sound sources.

[0003] Currently, many audio-visual segmentation algorithms use an audio-centric design, which uses audio features as the primary query in the network or directly integrates audio features into the query vector to guide segmentation. Although such methods can capture the global semantic information of audio, they also face two major problems:

[0004] (1) Perceptual ambiguity: Audio in real scenes is often mixed together. Different sound sources may interfere with each other, and even noise sources outside the picture may appear. This makes it easy for audio-centric queries to make false detections or missed detections during segmentation.

[0005] (2) Loss of visual details: Audio features are usually difficult to carry details such as precise spatial position and shape contours; and when subsequently cross-modally fused with visual features, if visual information is introduced too late, the mask prediction accuracy will often be insufficient.

[0006] Therefore, how to simultaneously take into account the role of audio in determining whether a sound is made and the advantages of vision in fine-grained segmentation and target differentiation has become an important issue that needs to be addressed in the task of video sound object segmentation. Summary of the Invention

[0007] In order to overcome the above-mentioned defects in the prior art, the present invention provides a vision-centric video sound object segmentation method that can analyze multimodal data containing video and audio information, achieve pixel-level precise segmentation of actual sound-making objects in video frames, and identify the sound source type.

[0008] To achieve the above object, the present invention adopts the following technical solutions, including:

[0009] A vision-centric video sound object segmentation method comprising:

[0010] S1, input the audio and video into the audio encoder and visual encoder respectively to obtain the corresponding audio features and multi-scale visual features;

[0011] S2, input the first-level features with the highest resolution among the multi-scale visual features into the prototype hint query generation module to obtain the visual derived query;

[0012] S3: Input the visually derived query into the iterative audio-video interaction module, which iteratively interacts with the audio features and multi-scale visual features in turn to gradually optimize the information expression capability of the visually derived query and obtain the optimized visually derived query;

[0013] In S4, the optimized visually derived query is used to generate the final prediction results, including the segmentation masks of the sound-making objects in the video frame and the audio event category corresponding to each mask.

[0014] Preferably, in step S1, the video frame is input into the image encoder to extract multi-scale visual features, which are recorded as in represents the visual features of the i-th stage, H i 、W i 、 V i The height, width, and number of channels of V2 are the highest among the multi-scale visual features. I The spatial resolution is the lowest;

[0015] Perform short-time Fourier transform on the audio to obtain time-frequency representation, apply Mel filter to obtain the corresponding Mel frequency, and input the Mel spectrum into the audio encoder to extract audio features, which is recorded as S is the feature size of the Mel spectrum graph, C a is the number of channels of audio features, and T is the number of video frames corresponding to the audio.

[0016] Preferably, in step S2, the prototype prompt query generation module introduces a set of audio event prototypes, denoted as Used to provide audio event category information for query, K is the total number of audio event categories, C h The number of channels of the audio event prototype. The specific process of generating the prototype hint query is as follows:

[0017] S21, performs dimensionality reduction transformation on the highest resolution visual feature V2 in the multi-scale visual feature, and reduces the number of channels of the visual feature V2 from Reduced to C h , get the visual features after dimensionality reduction The visual features V after dimensionality reduction h Perform shape adjustment to obtain adjusted visual features Using a multi-layer perceptron network, H2W2 is regarded as the input channel dimension of the multi-layer perceptron, and the adjusted visual features V' h Aggregate spatial information to obtain visual embedding features It includes a set of N visual embeddings, where N represents the maximum number of sound-making objects that may exist in the video, and the number of channels of each visual embedding is C. h ;

[0018] S22, the audio event prototype P is used as the audio category prior, and the cross attention is combined with the visual embedding feature V e Interact and obtain the visual embedding features after interaction The calculation method is:

[0019]

[0020] in, and is the transformation matrix weight of query, key and value in cross attention; Softmax is the activation function; the visual embedding feature after interaction Includes a set of N interactive visual embeddings;

[0021] S23, introduces Gumbel-Softmax technology to embed the visual features after interaction The pixel context is grouped and calculated as follows:

[0022]

[0023] in, and Represents the linear transformation weights of query and key in cross attention; represents the noise term sampled independently and identically from the Gumbel(0,1) distribution; is the allocation matrix; the argmax function is used on the allocation matrix R in the dimension N to select the best matching index of each pixel feature in the visual embedding after N interactions from the allocation matrix R; the onehot function converts the index of each pixel feature into a one-hot encoding; the sg function indicates the gradient stop to ensure that the gradient propagation is not affected during back propagation, so that the entire optimization process remains differentiable; the final allocation matrix is ​​obtained Represents a hard assignment relationship between pixel features and visual embeddings, that is, each pixel is clearly divided into a specific visual embedding group;

[0024] S24, using the final allocation matrix The visual features V after dimensionality reduction h Aggregate pixel context and convert V hAggregate onto N interacted visual embeddings to get the visually derived query V q , calculated as:

[0025]

[0026] in, and W o are linear projection weight matrices respectively; Norm is the normalization function, which means Divide the H2W2 dimension by the sum of the dimension values ​​to achieve Normalization of The visually derived query is the final output of the prototype hint query generation module.

[0027] Preferably, before training begins, the audio event prototype is a set of randomly initialized learnable parameters, and the prototype-audio contrast loss is designed in the prototype-prompt query generation module The specific calculation is as follows:

[0028] The audio features corresponding to the current frame Perform linear transformation and then compress it using global average pooling to obtain compact audio features Thus, the audio features corresponding to the current frame Mapped to the same space as the audio event prototype;

[0029] calculate The inner product between the audio event prototype P and the predicted matching matrix is ​​obtained The matching matrix is ​​used to reflect the degree of matching between the current audio and the audio event prototype of each audio event category. The larger the value, the higher the possibility that the audio event category appears in the current audio.

[0030] The true label of the matching matrix M * Obtained from the dataset, this label marks the audio event category actually contained in the current audio;

[0031] Prototype - Audio Contrast Loss For M and M * The binary cross entropy loss between:

[0032]

[0033] Preferably, in step S3, the iterative audio and video interaction module includes at least one layer, each layer includes an audio information acquisition module and a visual information enhancement module connected in sequence, and the visually derived query interacts with the audio features and the multi-scale visual features respectively through the audio information acquisition module and the visual information enhancement module; the audio information acquisition module and the visual information enhancement module are both composed of a decoder, and the decoder is composed of a multi-head cross-attention layer, a multi-head self-attention layer, and a feedforward network connected in sequence;

[0034] In the audio information acquisition module, the visually derived query interacts with the audio features corresponding to the current frame through multi-head cross-attention, where the audio features serve as keys and values ​​in the cross-attention.

[0035] In the visual information enhancement module, the visual derived query output by the audio information acquisition module after interacting with the audio features interacts one by one with the multi-scale visual features of the current frame through multi-head cross-attention; among them, the multi-scale visual features serve as keys and values ​​in the cross-attention in turn.

[0036] Preferably, the iterative audio and video interaction module includes multiple decoders, and among two adjacent decoders, the output of the previous decoder is used as the input of the next decoder.

[0037] Preferably, for the optimized visually derived query, a masking head and a classification head are used to generate segmentation mask predictions of the sound-emitting object and audio event category predictions corresponding to each mask, respectively.

[0038] Preferably, during the training of the model, the total loss function for:

[0039]

[0040] in, For classification loss, cross entropy loss is used; It is the mask loss, which is composed of binary cross entropy loss and Dice loss; The prototype-audio contrastive loss in the prototype-cued query generation module; λ cls ,λ mask and λ pac is the corresponding weight.

[0041] The present invention also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the method for segmenting a video sound object based on vision is implemented.

[0042] The present invention also provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements the vision-centric video sound object segmentation method.

[0043] The advantages of the present invention are:

[0044] (1) The present invention aims to overcome the shortcomings of existing audio-centric video sound object segmentation methods in processing mixed audio, invisible sound sources, and object fine contour extraction, such as perceptual ambiguity and loss of visual details. A new transformer network framework with a vision-centric approach in query design and network structure is proposed, which enables the model to distinguish different sound targets in the same audio while obtaining rich visual contour details, thereby significantly improving the accuracy and robustness of segmentation. The present invention can analyze multimodal data containing video and audio information, achieve pixel-level accurate segmentation of actual sound objects in video frames, and identify sound source types.

[0045] (2) The present invention proposes a vision-centric transformer network (VCT), which iteratively acquires audio and visual information by using visually derived query vectors generated by visual features, overcomes the limitations of audio-centric methods, and achieves more accurate and robust sound object segmentation.

[0046] (3) Vision-derived query vectors are generated directly from visual features, so that the segmentation process focuses on real visible objects, which can significantly alleviate the modal interference and ambiguity problems of traditional audio queries in mixed sound source scenarios.

[0047] (4) When generating visually derived queries, the Prototype Prompted Query Generation module uses a small number of learnable audio event prototypes to guide visual features to perceive potential sound source categories in advance, thereby improving the matching degree of the target audio signal. In addition, the module combines pixel grouping to allow queries to aggregate information from different image regions, making different visually derived queries more distinguishable.

[0048] (5) In the Iterative Audio-Visual Interaction module, the visually derived query interacts with the audio features and visual features of different resolutions in multiple rounds, first aggregating the corresponding audio semantic information and then refining finer visual details for accurate pixel-level segmentation and category recognition of the sound-emitting objects.

[0049] (6) Experiments on a public video sound object segmentation dataset show that the present invention has achieved the current best segmentation performance in single-source, multi-source, and multi-scene benchmarks with semantic labels, with high robustness and high precision, and has high application value.

[0050] (7) Visually derived queries can more naturally distinguish different sounding objects in mixed audio scenes, effectively reduce misclassification or omission, and suppress audio perception ambiguity.

[0051] (8) Rich visual spatial information is established at the beginning, without having to wait until the end to introduce visual features, so that the contour of the segmentation mask is more accurate and the visual details are preserved.

[0052] (9) Especially when there are off-screen sound sources or environmental noise in the audio, the present invention can still rely on visual information to ensure the positioning and recognition of the actual visible sound-making object, and the method is more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall framework diagram of the method of the present invention.

[0054] Figure 2 A framework diagram of the module for prototyping hinted queries. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Depend on Figure 1 As shown, the present invention provides a method for video sound object segmentation based on vision, which can segment and identify sound objects in input video and corresponding audio. Where T is the number of frames, H and W are the height and width of the video frame, and the corresponding audio clip Video sound object segmentation aims to predict the segmentation mask of the sound-making object Here, K represents the number of audio event categories. In category-agnostic settings (such as semi-supervised single-source segmentation and fully supervised multi-source segmentation), K = 1, meaning the model only needs to segment the sounding object without distinguishing the specific audio event category.

[0057] The present invention provides a method for segmenting a video sound object based on vision, comprising the following steps:

[0058] S1, input multiple frames in the audio clip and video clip into an audio encoder and a visual encoder respectively to obtain corresponding audio features and multi-scale visual features.

[0059] S2, in order to ensure that the visually derived query contains rich visual details at the beginning, the first-level features with the highest resolution among the multi-scale visual features are input into the prototype hint query generation module to obtain the visually derived query.

[0060] In S3, the generated visually derived query is input into the iterative audio-visual interaction module, which iteratively interacts with the audio features and multi-scale visual features in turn to gradually optimize the information expression capability of the visually derived query.

[0061] In S4, the optimized visually derived query is used to generate the final prediction results, including the pixel-level segmentation mask of the sound-making object in each frame, and the audio event category corresponding to each mask.

[0062] The specific process of step S1 is as follows:

[0063] S11, extraction of multi-scale visual features.

[0064] For input video For each frame (video frame), a visual encoder (image encoder, such as ResNet, SwinTransformer, etc.) is used to extract multi-scale visual features, which are recorded as in Represents the visual features from the i-th stage, height H i =H / 2 i , width W i =W / 2 i , Indicates the number of channels at this stage. Among the multi-scale visual features, V2 has the highest spatial resolution and V5 has the lowest spatial resolution.

[0065] In the field of computer vision, multi-scale visual features refer to feature information extracted by analyzing and extracting visual data such as images or video frames at different scales (or resolutions). Images exhibit different details and structural information at different scales. For example, at larger scales (low spatial resolution), the overall image outline and the approximate location of major objects can be seen; at smaller scales (high spatial resolution), finer details such as object texture and edges can be observed. Multi-scale visual features attempt to capture this diverse information at different scales to more comprehensively and accurately describe image content.

[0066] Multi-scale visual features can be extracted using a pyramid structure or convolutional neural networks. In a pyramid structure, an image is constructed into a series of versions of different resolutions, with the bottom high-resolution image gradually upsampled to lower resolution images. Features are extracted at each resolution layer to obtain multi-scale visual features. In a CNN, multi-scale feature extraction is achieved through convolution kernels of different sizes and different pooling operations. Smaller convolution kernels can capture local image details, while larger convolution kernels can capture broader contextual information. Pooling operations can reduce the resolution of features, thereby extracting features at different scales.

[0067] S12, audio feature extraction.

[0068] First, the input audio clip is converted to mono format, resulting in Where L depends on the actual duration of the audio. To ensure the consistency of the input data in the time dimension, the audio is resampled to 16kHz. Perform a short-time Fourier transform to calculate its time-frequency representation, and apply a Mel filter to obtain the corresponding Mel spectrum. Where T represents the number of video frames corresponding to the audio. After completing the time-frequency conversion, the Mel spectrum Input audio encoder (such as VGGish, etc.) to extract audio features. Different from the original VGGish, this invention removes the final compression layer to obtain audio features Where S represents the feature size of the Mel-spectrogram, C a Represents the number of channels of audio features.

[0069] In step S2, the goal of prototypical query generation is to generate visually derived queries that not only have rich visual details but also can integrate audio semantic information, so as to more accurately obtain multimodal features in the subsequent audio and video interaction process.

[0070] Depend on Figure 2 As shown, the specific process of step S2 is as follows:

[0071] S21, visual embedding aggregation:

[0072] First, spatial information aggregation is performed on the highest resolution visual feature V2 from the multi-scale visual features to obtain a set of initial visual embeddings. Since V2 has a high resolution, direct use leads to high computational complexity, so it is first subjected to dimensionality reduction transformation, and its number of channels is reduced from Reduced to C h , get the visual feature V after dimensionality reduction h :

[0073] V h =Conv 1×1 (δ(Conv 3×3 (δ(Conv 1×1 (V2)))))

[0074] Among them, δ represents the ReLU activation function; Conv 1×1 Represents 1×1 convolution, Conv 3×3 Represents a 3×3 convolution.

[0075] After obtaining the visual feature V after dimensionality reduction h After that, the shape is further changed from Adjust to Get the adjusted visual features This is equivalent to V h Flattening in the spatial dimension allows each channel to cover the spatial information of the entire feature map.

[0076] Subsequently, a multi-layer perceptron network consisting of three fully connected layers is used, and H2W2 is regarded as the input channel dimension of the multi-layer perceptron to perform the adjustment on the visual features V' h Further spatial information aggregation is performed to obtain visual embedding features

[0077] V e =Reshape(MLP(Reshape(V' h )))

[0078] Among them, the Reshape function is used to adjust the number of rows, columns, and dimensions of the matrix; the output feature shape of the multi-layer perceptron MLP is Output visual embedding features It includes a set of N visual embeddings, where N represents the maximum number of sound-making objects that may exist in the video, and the number of channels of each visual embedding is C. h .

[0079] Through the above operations, the generated visual embedding feature V e It covers rich spatial details and has strong global perception capabilities, thus laying the foundation for subsequent visually derived queries.

[0080] S22, audio event prototype prompt:

[0081] In order to make the visual derived query more accurate in obtaining relevant audio information in the subsequent iterative audio and video interaction module, the prototype-prompted query generation module introduces a set of audio event prototypes (AudioEvent Prototypes) in the query generation process, denoted as It is used to provide the possible audio event category information in the scene for the query, K is the total number of audio event categories, C h is the number of channels of the audio event prototype. The audio event prototype is used as the prior of the audio event category, and is embedded in the visual feature V through cross attention. e Interact and obtain the visual embedding features after interaction

[0082]

[0083] in, and is the transformation matrix weight of query, key and value in cross attention; the output of cross attention is the visual embedding feature after interaction It consists of a set of N interactive visual embeddings.

[0084] In this embodiment, before training begins, the audio event prototype is a set of randomly initialized learnable parameters. Since the audio event prototypes are randomly initialized in the initial stage, in order to ensure that they can effectively learn the semantic information of different audio events, a prototype-audio contrast loss is designed in the prototype-cued query generation module. This enables the audio event prototype to learn the semantic information of different audio event categories from audio features.

[0085] Specifically, in order to convert the audio features corresponding to the current frame Mapped to the same space as the audio event prototype, first A t Perform linear transformation and then compress it using global average pooling to obtain compact audio features Afterwards, calculate The inner product between the audio event prototype P and the predicted matching matrix is ​​obtained This matrix reflects the degree of matching between the current audio and the audio event prototype of each audio event category. The larger the value, the higher the possibility of the audio event category appearing in the current audio. In order to guide the learning of audio event prototypes, the true label M of the matching matrix is * Obtained from the dataset, this label marks the audio event category information actually contained in the current audio. It is a binary matrix, 1 represents the current audio contains a certain audio event category, and 0 represents it does not contain. Prototype-audio contrast loss For M and M * The binary cross entropy loss between:

[0086]

[0087] By optimizing the prototype-audio contrast loss It can shorten the distance between the audio features and the category prototypes that actually appear in the audio, and at the same time increase the distance between them and the category prototypes that do not appear, so that the audio event prototype P will not remain in a randomly initialized state, but will be able to learn meaningful semantic information through audio features and accurately reflect the semantic characteristics of different audio events.

[0088] S23, pixel context grouping:

[0089] In order to generate more discriminative visual derivative queries that can focus on different visual areas, the prototype-hinted query generation module introduces the Gumbel-Softmax technology to embed the interactive visual features through a hard assignment method. The pixel contexts of are grouped and the assignment process is guaranteed to be always differentiable during the training phase, thus achieving end-to-end optimization of the module. The specific calculation method is:

[0090]

[0091] in, and represents the linear transformation weights of query and key in cross attention, represents the noise term sampled independently and identically from the Gumbel(0,1) distribution. The purpose of using the argmax function in the dimension N is to select the best matching index of each pixel feature in the visual embedding after N interactions. The onehot function converts the index of each pixel feature into a one-hot encoding. The sg function stands for gradient stop to ensure that the backpropagation does not affect the gradient propagation, making the entire optimization process still differentiable. The final allocation matrix is ​​obtained It represents the hard assignment relationship between pixel features and visual embeddings, that is, each pixel is clearly divided into a specific visual embedding group.

[0092] S24, visually derived query:

[0093] Using the final allocation matrix The visual features V after dimensionality reduction h Aggregate the pixel context onto N interacted visual embeddings:

[0094]

[0095] in, and W o They are linear projection weight matrices, Norm represents Divide the H2W2 dimension by the sum of the dimension values ​​to achieve Normalization of The visually derived query is the final output of the prototype-cued query generation module. This visually derived query not only inherits the spatial information of the visual embedding, but also combines the category information provided by the audio event prototype. It has clear regional distinction capabilities during the initialization phase, making the subsequent audio and video feature interaction more accurate, thereby improving the overall performance of the model in the sound object segmentation task.

[0096] In step S3, the visually derived query is input into the iterative audio-video interaction module, and iteratively interacts with the audio features and the multi-scale visual features in turn to gradually optimize the information expression capability of the visually derived query and obtain the optimized visually derived query.

[0097] The iterative audio-video interaction module includes at least one layer, each layer includes an audio information acquisition module and a visual information enhancement module connected in sequence, and the visually derived query interacts with the audio features and the multi-scale visual features respectively through the audio information acquisition module and the visual information enhancement module; the audio information acquisition module and the visual information enhancement module are both composed of a decoder, and the decoder is composed of a multi-head cross-attention (MHCA) layer, a multi-head self-attention (MHSA) layer and a feed-forward network (FFN) connected in sequence.

[0098] In this embodiment, the audio information acquisition module includes a decoder for interacting visually derived queries with audio features; the visual information enhancement module includes three decoders for interacting visually derived queries with multi-scale visual features (V5, V4, V3).

[0099] First, in the audio information acquisition module, the visually derived query of each video frame is combined with the audio feature A corresponding to the current frame through the decoder’s multi-head cross attention. t Interact, where the audio feature A t As the key and value in the cross-attention, multi-head cross-attention enables the visual-derived query to obtain the corresponding audio information in the visual area it represents, thereby determining whether there is a sound object in the visual area represented by the visual-derived query and further inferring its corresponding audio category.

[0100] Subsequently, in the visual information enhancement module, the visually derived query output by the audio information acquisition module, after interacting with the audio features, interacts one by one with the multi-scale visual features of the current frame, thereby further enriching the visual details of the query. In the multi-head cross-attention of each decoder in the visual information enhancement module, V5, V4, and V3 are used as keys and values ​​respectively. The visual information enhancement module enables the visually derived query to extract information from visual features of different scales, thereby supplementing the visual area represented by the visually derived query with finer-grained visual details, thereby generating a more accurate segmentation mask.

[0101] The iterative audio and video interaction module includes multiple decoders. To ensure that the attention mechanism can focus on potential sound-generating areas, the output of the previous decoder is used as the input of the next decoder in two adjacent decoders, thereby constraining the multi-head cross attention to focus only on possible sound-generating object areas.

[0102] Specifically, in the iterative audio and video interaction module of this embodiment, the definition is a basic interaction unit, which is used to represent the interaction sequence between visually derived queries and audio features and multi-scale visual features. In the iterative audio and video interaction process, the interaction unit It is repeated twice, and an additional audio information acquisition module is added at the end, that is, the iterative audio and video interaction module includes 9 decoders, and the structure of the iterative audio and video interaction module is, in order, an audio information acquisition module (1 decoder), a visual information enhancement module (3 decoders), an audio information acquisition module (1 decoder), a visual information enhancement module (3 decoders), and an audio information acquisition module (1 decoder), thereby ensuring that the visually derived query can fully aggregate the audio information before outputting the final prediction, thereby improving the model's perception of the sound-emitting object. Through this iterative interaction mechanism, the visually derived query can gradually obtain its corresponding sound information and continuously fuse fine visual features, so that the visually derived query can not only accurately distinguish different sound-emitting objects, but also effectively alleviate the perceptual ambiguity problem caused by audio mixing. Ultimately, this hierarchical and progressive information fusion method enables the vision-centered video sound object segmentation method proposed in the present invention to have stronger recognition ability and robustness.

[0103] In step S4, for the optimized visually derived query, a masking header and a classification head are used to generate segmentation mask predictions of the sound-emitting object and audio event category predictions corresponding to each mask, respectively.

[0104] In the process of model training, the present invention adopts three loss functions, namely classification loss Mask loss and the prototype-audio contrast loss proposed in the prototype-hinted query generation module To optimize the learning effect of the model. Among them, the classification loss Cross entropy loss is used. Mask loss It is composed of binary cross entropy loss and Dice loss. The Dice loss function is mainly used to deal with the situation where the foreground area is small, ensuring that the model can still maintain good segmentation accuracy when processing small targets. Therefore, the total loss function of the present invention in the training process of the model is The definition is as follows:

[0105]

[0106] Among them, λ cls ,λ mask and λ pac are the weights of these loss functions.

[0107] The model training process of the present invention is as follows:

[0108] (1) Obtain samples, construct a data set, and perform training initialization settings;

[0109] The sample is audio or video, and the label of the sample is the segmentation mask of all sound objects contained in the audio or video and the audio event category corresponding to each mask. Define the current number of training times as d, and let the model parameter of the dth training be ε d ; Set the total number of training times to epoch; the dataset is divided into a training set and a validation set, and the initial best accuracy of the validation set is recorded as 0.

[0110] (2) First, multiple frames from the audio and video clips are fed into an audio encoder and a visual encoder, respectively, to obtain the corresponding audio features and multi-scale visual features. Then, to ensure that the visually derived query initially contains rich visual details, the first-level features with the highest resolution in the multi-scale visual features are fed into the prototype-cued query generation module to obtain the visually derived query. The generated visually derived query is then fed into the iterative audio-video interaction module, where it iterative interactions with the audio features and multi-scale visual features are performed to gradually optimize the information expression capability of the query. Finally, the optimized visually derived query is used to generate the final prediction results, including the pixel-level segmentation mask of the sound-emitting object in each frame, and the audio event category corresponding to each mask.

[0111] (3) The pixel-level segmentation mask in the prediction result is defined as Where K represents the number of audio event categories. In the category-independent setting (such as semi-supervised single-source segmentation setting and fully supervised multi-source segmentation setting), K = 1, that is, the model only needs to segment the object that makes the sound without distinguishing the specific audio event category. The audio event category corresponding to each mask in the prediction result is defined as and Calculate the mask loss separately with its true label and classification loss Compared with prototype-audio contrast loss Calculate the total loss together according to Perform back propagation and update the model parameters to obtain the segmentation model for this training.

[0112] (4) Run the segmentation model trained this time on the validation set. If the average accuracy on the validation set is greater than the existing best accuracy, update the best model parameters to the model parameters obtained from this training, and update the best accuracy to the average accuracy on the validation set; otherwise, do not update the best model parameters and best accuracy.

[0113] (5) Determine whether the number of training times d reaches the total number of training epochs. If so, end the training process and output the optimal model parameters; otherwise, perform the next training according to steps (2) and (3), and update the number of training times d = d + 1.

[0114] The model usage process of the present invention is as follows:

[0115] After model training is complete, the optimal model parameters obtained from training are loaded, and audio and video data are read. Multiple frames from the audio and video clips are input into an audio encoder and a visual encoder, respectively, to obtain corresponding audio features and multi-scale visual features. Subsequently, to ensure that the visually derived query initially contains rich visual details, the highest-resolution first-level features from the multi-scale visual features are input into the prototype-cued query generation module to obtain the visually derived query. The generated visually derived query is then input into the iterative audio and video interaction module, which iteratively interacts with the audio features and multi-scale visual features to gradually optimize the information expression capability of the query. Finally, the optimized visually derived query is used to generate the final prediction results, including the pixel-level segmentation mask of the sound-emitting object in each frame, and the audio event category corresponding to each mask.

[0116] Example 1

[0117] The present invention conducted experiments on three benchmark settings of the Audio-Visual Segmentation (AVS-Bench) dataset, including: (1) semi-supervised Single Sound Source Segmentation (abbreviated as S4). In this setting, the sound-making object in the video clip is a single object and remains unchanged throughout the entire video; during the training process, only the segmentation mask annotation of the first frame of the video is provided as a supervisory signal for model training, and the segmentation mask of all T frames is provided in the test set for result verification. Semi-supervised single sound source segmentation includes a total of 4932 videos, of which the training set, validation set and test set include 3452, 740 and 740 videos respectively. The sound-making objects cover 23 different categories, and the total number of annotated frames is 10852. In this setting, the segmentation mask only contains two categories: foreground and background, and there is no need to distinguish between different categories of sounds. (2) fully-supervised Multiple Sound Source Segmentation (MS3). In this setting, the sound-emitting objects will change dynamically over time; during training and testing, segmentation masks of all T frames in the video are provided for supervision or algorithm verification. Fully supervised multi-source segmentation includes a total of 424 videos, of which the training set, validation set and test set include 296, 64 and 64 videos respectively. The sound-emitting objects cover 23 different categories, and the total number of annotated frames is 2120. In this setting, the segmentation mask only includes two categories: foreground and background, and there is no need to distinguish between different categories of sounds. (3) Fully supervised audio-visual semantic segmentation (AVSS). This setting is different from fully supervised multi-source segmentation. The sound-emitting objects will change dynamically over time, but the difference is that the sound category of the sound-emitting objects needs to be output additionally. The training and test datasets for this setup include data from semi-supervised single-source segmentation and fully-supervised multi-source segmentation. Further data was collected, bringing the total dataset size to 12,356 videos. The training set, validation set, and test set include 8,498, 1,304, and 1,554 videos, respectively. The sound-producing objects cover a total of 70 semantic categories, and a total of 82,972 annotated frames. In this setup, each frame is annotated with the semantic category of the sound-producing object.

[0118] The segmentation method proposed in this invention uses Swin-B as the visual encoder and the video frame width and height is 384x384. It achieves average intersection-over-union (IoU) / F scores of 86.2 / 93.4, 67.6 / 81.4 and 51.2 / 55.5 on the S4, MS3 and AVSS subsets of the AVSBench dataset, respectively, exceeding the previous optimal method.

[0119] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vision-centric video sound object segmentation method, characterized in that: include: S1, input the audio and video into the audio encoder and visual encoder respectively to obtain the corresponding audio features and multi-scale visual features; S2, input the first-level features with the highest resolution among the multi-scale visual features into the prototype hint query generation module to obtain the visual derived query; S3: Input the visually derived query into the iterative audio-video interaction module, which iteratively interacts with the audio features and multi-scale visual features in turn to gradually optimize the information expression capability of the visually derived query and obtain the optimized visually derived query; In S4, the optimized visually derived query is used to generate the final prediction results, including the segmentation masks of the sound-making objects in the video frame and the audio event category corresponding to each mask.

2. The method for segmenting sound objects in a video based on vision according to claim 1, wherein: In step S1, the video frame is input into the image encoder to extract multi-scale visual features, which are recorded as in represents the visual features of the i-th stage, H i 、W i 、 V i The height, width, and number of channels of V2 are the highest among the multi-scale visual features. I The spatial resolution is the lowest; Perform short-time Fourier transform on the audio to obtain time-frequency representation, apply Mel filter to obtain the corresponding Mel frequency, and input the Mel spectrum into the audio encoder to extract audio features, which is recorded as S is the feature size of the Mel spectrum graph, C a is the number of channels of audio features, and T is the number of video frames corresponding to the audio.

3. The method for segmenting sound objects in a video based on vision according to claim 2, wherein: In step S2, the prototype-prompted query generation module introduces a set of audio event prototypes, denoted as Used to provide audio event category information for query, K is the total number of audio event categories, C h The number of channels of the audio event prototype. The specific process of generating the prototype hint query is as follows: S21, performs dimensionality reduction transformation on the highest resolution visual feature V2 in the multi-scale visual feature, and reduces the number of channels of the visual feature V2 from Reduced to C h , get the visual features after dimensionality reduction The visual features V after dimensionality reduction h Perform shape adjustment to obtain adjusted visual features Using a multi-layer perceptron network, H2W2 is regarded as the input channel dimension of the multi-layer perceptron, and the adjusted visual features V' h Aggregate spatial information to obtain visual embedding features It includes a set of N visual embeddings, where N represents the maximum number of sound-making objects that may exist in the video, and the number of channels of each visual embedding is C. h ; S22, the audio event prototype P is used as the audio category prior, and the cross attention is combined with the visual embedding feature V e Interact and obtain the visual embedding features after interaction The calculation method is: in, and is the transformation matrix weight of query, key and value in cross attention; Softmax is the activation function; the visual embedding feature after interaction Includes a set of N interactive visual embeddings; S23, introduces Gumbel-Softmax technology to embed the visual features after interaction The pixel context is grouped and calculated as follows: in, and Represents the linear transformation weights of query and key in cross attention; represents the noise term sampled independently and identically from the Gumbel(0,1) distribution; is the allocation matrix; the argmax function is used on the allocation matrix R in the dimension N to select the best matching index of each pixel feature in the visual embedding after N interactions from the allocation matrix R; the onehot function converts the index of each pixel feature into a one-hot encoding; the sg function indicates the gradient stop to ensure that the gradient propagation is not affected during back propagation, so that the entire optimization process remains differentiable; the final allocation matrix is ​​obtained Represents a hard assignment relationship between pixel features and visual embeddings, that is, each pixel is clearly divided into a specific visual embedding group; S24, using the final allocation matrix The visual features V after dimensionality reduction h Aggregate pixel context and convert V h Aggregate onto N interacted visual embeddings to get the visually derived query V q , calculated as: in, and W o are linear projection weight matrices respectively; Norm is the normalization function, which means Divide the H2W2 dimension by the sum of the dimension values ​​to achieve Normalization of The visually derived query is the final output of the prototype hint query generation module.

4. The method for segmenting sound objects in a video based on vision according to claim 3, wherein: Before training begins, the audio event prototype is a set of randomly initialized learnable parameters, and the prototype-audio contrast loss is designed in the prototype-prompt query generation module The specific calculation is as follows: The audio features corresponding to the current frame Perform linear transformation and then compress it using global average pooling to obtain compact audio features Thus, the audio features corresponding to the current frame Mapped to the same space as the audio event prototype; calculate The inner product between the audio event prototype P and the predicted matching matrix is ​​obtained The matching matrix is ​​used to reflect the degree of matching between the current audio and the audio event prototype of each audio event category. The larger the value, the higher the possibility that the audio event category appears in the current audio. The true label of the matching matrix M * Obtained from the dataset, this label marks the audio event category actually contained in the current audio; Prototype - Audio Contrast Loss For M and M * The binary cross entropy loss between:

5. The method for segmenting sound objects in a video based on vision according to claim 1, wherein: In step S3, the iterative audio and video interaction module includes at least one layer, each layer includes an audio information acquisition module and a visual information enhancement module connected in sequence, and the visually derived query interacts with the audio features and the multi-scale visual features in sequence through the audio information acquisition module and the visual information enhancement module respectively; The audio information acquisition module and the visual information enhancement module are both composed of a decoder, which is composed of a multi-head cross attention layer, a multi-head self-attention layer and a feedforward network connected in sequence; In the audio information acquisition module, the visually derived query interacts with the audio features corresponding to the current frame through multi-head cross-attention, where the audio features serve as keys and values ​​in the cross-attention. In the visual information enhancement module, the visual derived query output by the audio information acquisition module after interacting with the audio features interacts one by one with the multi-scale visual features of the current frame through multi-head cross-attention; among them, the multi-scale visual features serve as keys and values ​​in the cross-attention in turn.

6. The method for segmenting sound-emitting objects in a video based on vision according to claim 5, characterized in that: The iterative audio and video interaction module includes multiple decoders. Among two adjacent decoders, the output of the previous decoder is used as the input of the next decoder.

7. The method for segmenting sound objects in a video based on vision according to claim 1, wherein: For the optimized visually derived query, a masking head and a classification head are used to generate segmentation mask predictions of the sound-emitting object and the audio event category prediction corresponding to each mask, respectively.

8. The method for segmenting sound objects in a video based on vision according to claim 4, wherein: During the training of the model, the total loss function for: in, For classification loss, cross entropy loss is used; It is the mask loss, which is composed of binary cross entropy loss and Dice loss; The prototype-audio contrastive loss in the prototype-cued query generation module; λ cls ,λ mask and λ pac is the corresponding weight.

9. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements a vision-centric video sound object segmentation method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements a vision-centric video sound object segmentation method according to any one of claims 1 to 8.

Images