A video and audio source separation method based on visual and auditory multimodality

Abstract

This invention discloses a video audio source separation method based on audiovisual multimodal perception, characterized by the following steps: Step 1, extracting features from the visual and auditory information of the video to obtain visual labels and auditory features; Step 2, feeding the visual labels and auditory features into a multi-instance multi-label network to obtain the association value between the auditory features and the visual labels; Step 3, feeding the video and the association value into an audio source separation network to obtain the individual sound of each object. In Step 2, the training of the multi-instance multi-label network includes the following steps: Step 2-1, constructing an initial multi-instance multi-label network; Step 2-2, feeding all audio feature vectors obtained from a video as a training set into the multi-instance multi-label network for training, obtaining the association feature map between each audio feature vector and all visual objects.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and audio signal processing, and in particular relates to a video and audio source separation method based on visual and auditory multimodality. Background Art

[0002] Video is the most important and primary media object for conveying information, and understanding scenes and events is essentially a multimodal experience. We perceive information in videos by watching and listening. A key challenge is that in real videos, object sounds are not observed as separate entities but as a single audio channel with all frequencies mixed together. Audio source separation, that is, correctly separating the sounds of many objects mixed together in a video, remains a difficult problem for natural data outside of laboratory environments, despite extensive research in the field of signal processing.

[0003] In recent years, with the continuous development of deep learning, the blind audio separation task has raised challenges similar to image segmentation, because all sounds overlap in the input signal. In particular, the excellent performance of convolutional neural networks in learning the representation features of audio-visual integration has enabled the sound source separation task to be solved end-to-end. However, in reality, due to various reasons, many videos are mixed with noise or even mixed with the sounds of many objects, making it difficult for people to clearly hear the sound belonging to a certain object and to obtain information accurately and timely. It is difficult for general sound source separation models to have strong generalization and high-precision separation performance on these difficult samples. Summary of the invention

[0004] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a video sound source separation method based on visual and auditory multimodality.

[0005] The present invention provides a video sound source separation method based on visual and auditory multimodality, which has the following characteristics and comprises the following steps: step 1, extracting features of visual information and auditory information of a video to obtain visual labels and auditory features; step 2, putting the visual labels and auditory features into a multi-instance multi-label network to obtain association values ​​of the auditory features and the visual labels; step 3, putting the video and the association value into a sound source separation network to obtain a separate sound for each object, wherein in step 2, the training of the multi-instance multi-label network comprises the following steps:

[0006] Step 2-1, construct an initial multi-instance multi-label network; Step 2-2, put all the audio feature vectors obtained in a video into the multi-instance multi-label network as a training set for training, and obtain the associated feature map of each audio feature vector and all visual objects.

[0007] In the video and audio source separation method based on visual and auditory multimodality provided by the present invention, it can also have the following characteristics: wherein, step 1 includes the following steps: step 1-1, first obtain the spectrum diagram through short-time Fourier transform, and then obtain all audio vectors through non-negative matrix decomposition; step 1-2, classify and predict the objects appearing in the video frame through the ResNet-152 network pre-trained by ImageNet, obtain the probability of each object corresponding to a certain classification, and retain the classification labels with probability values ​​above the set threshold.

[0008] In the video sound source separation method based on audio-visual multimodality provided by the present invention, it can also have the following characteristics: wherein, step 2-2 includes the following steps: step 2-2-1, inputting the audio vectors of each video in the training set into the multi-entity multi-label model in turn for one iteration to obtain the output value of the multi-entity multi-label model; step 2-2-2, using the output value of the multi-entity multi-label model to calculate the multi-label hinge loss and back-propagate the error to update the parameters; step 2-2-3, repeating steps 2-2-1 to 2-2-2 until the difference between the predicted correlation value of the correct classification and the incorrect classification is greater than the threshold 1, the training is completed, and the multi-entity multi-label model network is obtained.

[0009] In the video and audio source separation method based on visual and auditory multimodality provided by the present invention, it can also have the following characteristics: wherein, step 3 includes the following steps: step 3-1, performing short-time Fourier transform on the mixed audio to obtain a spectrum diagram; step 3-2, obtaining the audio vector belonging to each object through the correlation value; step 3-3, generating the original audio of each object through inverse short-time Fourier transform, taking the audio vector of each object as a reference.

[0010] In the video and audio source separation method based on audio-visual multimodality provided by the present invention, it can also have the following characteristics: wherein, the multi-entity multi-label model includes an input layer, a fully connected layer, a normalization layer, a RELU, a 1x1 convolutional layer, a normalization layer, a RELU and two maximum pooling layers arranged in sequence.

[0011] Functions and Effects of the Invention

[0012] According to the video sound source separation method based on visual and auditory multimodality involved in the present invention, the separation steps are: step 1, feature extraction of visual information and auditory information of the video to obtain visual labels and auditory features; step 2, putting the visual labels and auditory features into a multi-instance multi-label network to obtain the association value of the auditory features and the visual labels; step 3, putting the video and the association value into the sound source separation network to obtain a separate sound for each object, wherein in step 2, the training of the multi-instance multi-label network includes the following steps: step 2-1, constructing an initial multi-instance multi-label network; step 2-2, putting all audio feature vectors obtained in a video as a training set into the multi-instance multi-label network for training, and obtaining an association feature map of each audio feature vector and all visual objects.

[0013] Therefore, the video sound source separation method based on audio-visual multimodality of the present invention utilizes a large amount of unannotated videos to discover the potential sound representation of each object, uses the most advanced image recognition network to infer the objects present in each unlabeled video, and performs non-negative matrix decomposition on the audio channel of each video to recover its set of frequency basis vectors. At this point, it is not clear which audio basis is used with which visible objects. In order to restore this connection, a neural network for multi-instance multi-label learning is constructed in the present invention, which maps the audio library to the distribution of detected visual objects. From this audio basis object association network, the present invention extracts the audio basis vectors linked to each visual object, thereby generating its prototype spectrogram.

[0014] Finally, given a video, the present invention uses the learned audio basis of each object to guide audio source separation, achieving better results than the current most advanced sound source separation algorithm, with high generalization and stronger noise reduction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a video and audio source separation method based on visual and auditory multimodality in an embodiment of the present invention;

[0016] Figure 2 is a flow chart of the training steps of a multi-entity multi-label network for calculating audio-visual association information in an embodiment of the present invention;

[0017] Figure 3 is a model structure diagram of a multi-entity multi-label network for calculating audio-visual association information in an embodiment of the present invention; and

[0018] Figure 4 It is a model structure diagram of the sound source separation network in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate a method for separating video and audio sources based on visual and auditory multimodality of the present invention.

[0020] In this embodiment, a video and audio source separation method based on visual and auditory multimodality is provided.

[0021] Figure 1 It is a flowchart of a method for separating object sounds in a video based on deep learning and audio-visual integration in an embodiment of the present invention.

[0022] like Figure 1 As shown, the video and audio source separation method based on visual and auditory multimodality involved in this embodiment includes the following steps:

[0023] Step S1, extracting features from the visual information and auditory information of the video to obtain visual labels and auditory features.

[0024] The specific implementation of step S1 is:

[0025] Step S1-1, first obtain a spectrum diagram through short-time Fourier transform, and then obtain all audio vectors through non-negative matrix decomposition.

[0026] In step S1-2, the objects appearing in the video frame are classified and predicted using the ResNet-152 network pre-trained on ImageNet to obtain the probability that each object corresponds to a certain classification, and the classification labels whose probability values ​​are above the set threshold are retained.

[0027] Step S2, putting the visual label and the auditory feature into a multi-instance multi-label network to obtain the association value between the auditory feature and the visual label. The specific implementation method of step S2 is as follows:

[0028] Step S2-1, constructing an initial multi-instance multi-label network.

[0029] Step S2-2: All audio feature vectors obtained in a video are put into a multi-instance multi-label network as a training set for training, and a correlation feature map of each audio feature vector and all visual objects is obtained.

[0030] Figure 2 4 is a flowchart of the training steps of a multi-entity multi-label network for calculating audio-visual association information in an embodiment of the present invention.

[0031] Figure 3 4 is a model structure diagram of a multi-entity multi-label network for calculating audio-visual association information in an embodiment of the present invention.

[0032] like Figure 2-3 As shown, the training steps of the multi-entity multi-label network in step S2-2 are:

[0033] Step S2-2-1, input the audio vectors of each video in the training set into the multi-entity multi-label model in turn for one iteration to obtain the output value of the multi-entity multi-label model, wherein the multi-entity multi-label model includes an input layer, a fully connected layer, a normalization layer, a RELU, a 1x1 convolutional layer, a normalization layer, a RELU and two maximum pooling layers which are set in sequence.

[0034] In step S2-2-2, the output value of the multi-entity multi-label model is used to calculate the multi-label hinge loss and the error is back-propagated to update the parameters.

[0035] Step S2-2-3, repeat steps S2-2-1 to S2-2-2 until the difference between the predicted association value of the correct classification and the incorrect classification is greater than the threshold 1, the training is completed, and a multi-entity multi-label model network is obtained.

[0036] Step S3, putting the video and the associated value into the sound source separation network to obtain the individual sound of each object.

[0037] Step S3-1, performing short-time Fourier transform on the mixed audio to obtain a spectrum diagram.

[0038] Step S3-2, obtaining the audio vector belonging to each object through the association value.

[0039] Step S3-3, generating the original audio of each object by inverse short-time Fourier transform, taking the audio vector of each object as a reference.

[0040] Figure 4 It is a model structure diagram of the sound source separation network in this embodiment.

[0041] Functions and Effects of the Embodiments

[0042] According to the video sound source separation method based on visual and auditory multimodality involved in this embodiment, the separation steps are: Step 1, feature extraction of visual information and auditory information of the video to obtain visual labels and auditory features; Step 2, putting the visual labels and auditory features into a multi-instance multi-label network to obtain the association value of the auditory features and the visual labels; Step 3, putting the video and the association value into the sound source separation network to obtain a separate sound for each object, wherein in step 2, the training of the multi-instance multi-label network includes the following steps: Step 2-1, constructing an initial multi-instance multi-label network; Step 2-2, putting all audio feature vectors obtained in a video as a training set into the multi-instance multi-label network for training, and obtaining an association feature map of each audio feature vector and all visual objects.

[0043] Therefore, according to the method for separating the sound of objects in videos based on deep learning audio-visual combination of the present embodiment, a large number of unannotated videos are used to discover the potential sound representation of each object, the most advanced image recognition network is used to infer the objects present in each unlabeled video, and the audio channel of each video is subjected to non-negative matrix decomposition to restore its set of frequency basis vectors. At this point, it is not clear which audio basis is used with which visible objects. In order to restore this connection, a neural network for multi-instance multi-label learning is constructed in the present embodiment, which maps the audio library to the distribution of detected visual objects. From this audio basis object association network, the present embodiment extracts the audio basis vectors linked to each visual object, thereby generating its prototype spectrogram.

[0044] Finally, given a video, this embodiment uses the learned audio basis of each object to guide audio source separation, achieving better results than the current most advanced sound source separation algorithm, with high generalization and stronger noise reduction effects.

[0045] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. A video and sound source separation method based on visual and auditory multimodality, characterized in that: The following steps are involved: Step 1: extract features from the visual information and auditory information of the video to obtain visual labels and auditory features; Step 2, putting the visual label and the auditory feature into a multi-instance multi-label network to obtain the association value between the auditory feature and the visual label; Step 3: Put the video and the associated value into a sound source separation network to obtain a separate sound for each object. Wherein, in step 2, the training of the multi-instance multi-label network includes the following steps: Step 2-1, construct an initial multi-instance multi-label network; Step 2-2: All audio feature vectors obtained in a video are put into the multi-instance multi-label network as a training set for training, and a correlation feature map of each audio feature vector and all visual objects is obtained.

2. The video and sound source separation method based on visual and auditory multimodality according to claim 1 is characterized in that: in, Step 1 includes the following steps: Step 1-1, first obtain the spectrum through short-time Fourier transform, and then obtain all audio vectors through non-negative matrix decomposition; Step 1-2: Use the ImageNet pre-trained ResNet-152 network to classify and predict the objects appearing in the video frame, obtain the probability of each object corresponding to a certain classification, and retain the classification labels with probability values ​​above the set threshold.

3. The video and audio source separation method based on visual and auditory multimodality according to claim 1, Features: in, Step 2-2 includes the following steps: Step 2-2-1, inputting the audio vectors of each video in the training set into the multi-instance multi-label model in sequence for one iteration to obtain the output value of the multi-entity multi-label model; Step 2-2-2, using the output value of the multi-instance multi-label model to calculate the multi-label hinge loss and back-propagating the error to update the parameters; Step 2-2-3, repeat step 2-2-2 until the difference between the predicted association value of the correct classification and the incorrect classification is greater than the threshold 1, the training is completed, and a multi-instance multi-label model network is obtained.

4. The video and sound source separation method based on visual and auditory multimodality according to claim 1 is characterized in that: in, Step 3 includes the following steps: Step 3-1, performing short-time Fourier transform on the mixed audio to obtain a spectrum diagram; Step 3-2, obtaining an audio vector belonging to each object through the association value; Step 3-3, generating the original audio of each object by inverse short-time Fourier transform and taking the audio vector of each object as a reference.

5. The video and sound source separation method based on visual and auditory multimodality according to claim 1 is characterized in that: The multi-instance multi-label model includes an input layer, a fully connected layer, a normalization layer, a RELU, a 1x1 convolutional layer, a normalization layer, a RELU, and two maximum pooling layers arranged in sequence.

Images