Method of locating sound source from video

Abstract

The invention provides a method for locating a sound source from a video, and belongs to the field of cross-modal learning. According to the method, a training sample video is acquired in a training stage and preprocessed, a neural network composed of a full connection layer and a sound source positioning neural network composed of a positioning network are constructed, the sound source positioning neural network is trained by using a preprocessed training sample, and the trained sound source positioning neural network is obtained. In the test stage, a test video is obtained and preprocessed,then the test video is input into the trained sound source positioning neural network, the similarity is calculated, synchronization of sound and video images and sound source positioning after synchronization are further carried out through the similarity, and therefore the problem of sound source positioning of the asynchronous video is solved. The method can automatically find the correspondingrelation between each object and the sound in the video picture, is high in positioning accuracy, is high in position precision, and is very high in application value.

Description

technical field [0001] The invention proposes a method for locating a sound source from a video, which belongs to the field of cross-modal learning. Background technique [0002] In recent years, with the popularity of the Internet and television, people are faced with more and more video clips. Videos contain rich sounds and images, and finding connections among them is meaningful in many ways, such as making human-machine interactions more friendly. It is becoming more and more important to automatically discover the corresponding relationship between various objects and sounds in the video screen, so as to help people quickly understand the part of the pronunciation in the video. The robot can also determine the location of the target in many scenarios such as rescue by locating the sound source in the video. [0003] At present, most of the existing work on video sound source positioning is at the pixel level, using convolutional neural networks to learn the relationsh...

AI Technical Summary

Problems solved by technology

[0003] At present, most of the existing work on video sound source positioning is at the pixel level, using convolutional neural networks to learn the relationship between sound and different positions in the picture, and using heat maps to mark possible sounds in the original image Partly, the edge of this method is blurred, the positioning accuracy is not high, and there is still positioning information in video frames where the sound and picture are not synchronized

Images