A method of locating sound source from video

Abstract

The invention proposes a method for locating a sound source from a video, which belongs to the field of cross-modal learning. In the training phase, the method obtains the training sample video and performs preprocessing, constructs a neural network composed of a fully connected layer and a localization network, and uses the preprocessed training samples to train the neural network for localization of the sound source , to obtain the trained sound source localization neural network. In the test phase, the test video is obtained and preprocessed, and then the trained sound source localization neural network is input to calculate the similarity, and the similarity is used to further synchronize the sound and video images and the synchronized sound source positioning, so as to solve the out-of-sync video The sound source localization problem. The invention can automatically discover the corresponding relationship between each object and sound in the video picture, has high positioning accuracy, high position accuracy, and has high application value.

Description

technical field [0001] The invention proposes a method for locating a sound source from a video, which belongs to the field of cross-modal learning. Background technique [0002] In recent years, with the popularity of the Internet and television, people are faced with more and more video clips. Videos contain rich sounds and images, and finding connections among them is meaningful in many ways, such as making human-machine interactions more friendly. It is becoming more and more important to automatically discover the corresponding relationship between various objects and sounds in the video screen, so as to help people quickly understand the part of the pronunciation in the video. The robot can also determine the location of the target in many scenarios such as rescue by locating the sound source in the video. [0003] At present, most of the existing work on video sound source positioning is at the pixel level, using convolutional neural networks to learn the relationsh...

AI Technical Summary

Problems solved by technology

[0003] At present, most of the existing work on video sound source positioning is at the pixel level, using convolutional neural networks to learn the relationship between sound and different positions in the picture, and using heat maps to mark possible sounds in the original image Partly, the edge of this method is blurred, the positioning accuracy is not high, and there is still positioning information in video frames where the sound and picture are not synchronized

Images