Method and system for sound source localization and sound source separation based on dual consistent network

Abstract

The invention discloses a method and system for sound source localization and sound source separation based on a dual consistent network, and belongs to the field of image-audio multimodality. It mainly includes the following steps: 1) Obtain audio and video data sets, select a pair of videos belonging to different sound domains, extract corresponding single-source audio and image information, and calculate the mixed audio. 2) Perform feature encoding on audio and image respectively to obtain audio and image features. 3) Send the mixed audio and image features to the sound source separation module of the dual consistent network to separate the single-source audio. 4) Send the image and corresponding audio features to the sound source localization module of the dual consensus network to obtain the sounding object in the image. Compared with the traditional method in the task of sound source localization and sound source separation, the present invention regards the two tasks as dual tasks, and uses the same architecture to complete them simultaneously, and utilizes the characteristics of the two tasks to enhance the performance of each other during the training process, Finally, the effect was improved on both tasks.

Description

technical field [0001] The invention relates to the image-audio multimodal field, in particular to a method for sound source localization and sound source separation based on a dual consistent network. Background technique [0002] Vision and hearing are important ways for human beings to perceive the world. We can recognize and separate the sounds from various objects, and at the same time find objects that emit sounds in complex scenes. Having such a powerful perception is what we need to make subsequent complex decisions. Foundation. Therefore, enabling machines to have the ability to separate and locate sound sources is the only way to realize artificial intelligence. [0003] Many current studies mainly focus on two separate tasks, namely sound source localization and visually guided sound separation. Although they have achieved certain results, there are still some unsolved problems: [0004] 1) In the current vision-guided sound separation model, it is necessary to ...

AI Technical Summary

Problems solved by technology

[0004] 1) In the current vision-guided sound separation model, it is necessary to use a specific image to query the sound corresponding to the object in it, but when there are multiple objects in the image, the model will not know which object to separate the sound corresponding to, resulting in poor performance

[0005] 2) At present, most of the models correspond to two tasks, which cannot be processed at the same time with a set of architectures. When the audio needs to be positioned and separated at the same time, the direct superimposition model will be very complicated and the calculation speed is slow

Images