Method for separating sound source from video

Abstract

The invention provides a method for separating a sound source from a video. The method is composed of a training stage and a testing stage. According to the method, the method includes constructing asound source separation model composed of a visual target detection network, a sound feature extraction network and a sound separation network in a training stage; and selecting two different types ofvideos from the training data to mix the audios of the videos, and training a sound source separation model to enable the sound source separation model to accurately separate original audios corresponding to the two videos from the mixed audio. And in the testing stage, a test video is acquired and then input into the trained sound source separation model, the model detects all visual targets inthe video, and sounds corresponding to the visual targets are separated from the original audio. According to the invention, the sound source can be separated from the target object level, all targetobjects appearing in the video can be detected and automatically matched with the separated corresponding sound, the relation between each visual target object and the separated sound is established,and the application prospect is wide.

Description

technical field [0001] The invention relates to a method for separating sound sources from videos, belonging to the field of cross-modal learning. Background technique [0002] In recent years, technologies such as smart wearable devices, smart homes, and smart service robots have developed rapidly, which require real-time processing of video, audio and other data in real scenes and further use of the processing results in subsequent behaviors. Among them, it is a very important task to separate the sound of each audio source from the mixed audio containing multiple audio sources. For example, when a person gives a voice command to an intelligent service robot, the environment may also contain the sounds of household appliances such as telephone ringtones, alarm clocks, and TVs. At this time, the intelligent robot needs to separate the human voice from the obtained mixed audio to correctly Identifies the instructions given to it by a person. The task of sound source separa...

AI Technical Summary

Problems solved by technology

The sound separation results of these methods are not suitable for applications such as intelligent robots in real scenes. When sound separation is performed at the pixel level, although the sound produced by each pixel can be obtained, the connection between adjacent pixels cannot be established. Isolated pixels have no practical significance in real-world scenarios. For example, a robot can only know the sound signal from a certain pixel in the current picture but does not know that this pixel is a part of the object of the alarm clock.

Similarly, when performing sound separation at the image segmentation region level, the robot can only know the sound signal generated in a certain region but cannot know which objects are actually contained in the region in the current picture, which makes it impossible for the robot to understand the source of the separated sound. Actual Meaning of Representation

Images