A multi-modal speech separation method and system

Abstract

This disclosure proposes a multi-modal speech separation method and system, including: receiving the mixed voice of the object to be recognized and the facial visual information of the object to be recognized; using the Dlib library to perform face detection to obtain the number of speakers; and processing the above information Obtain the complex language spectrogram and the face image of the speaker and transmit it to the multi-modal speech separation model, and dynamically adjust the structure of the model according to the number of speakers, wherein the multi-modal speech separation model uses complex numbers in the training process Domain ideal ratio masking is used as the training target, which is defined as the ratio between the clean sound spectrogram and the mixed sound spectrogram in the complex domain, which consists of real and imaginary parts and contains the amplitude and phase information of the sound; The multimodal speech separation model outputs the time-frequency mask corresponding to the number of faces; the output mask is multiplied with the spectrogram of the mixed voice to obtain the spectrogram of the clean voice, and the spectrogram of the clean voice is made The short-time Fourier inverse transform is calculated to obtain the time-domain signal of the clean sound, thereby completing the speech separation. The disclosed model is more suitable for most application scenarios.

Description

technical field [0001] The present disclosure belongs to the technical field of speech separation, and in particular, relates to a multimodal speech separation method and system. Background technique [0002] In life, it is often necessary to come into contact with a variety of mixed sounds, among which the mixed sound of people is the most common. In a mixed environment, people have the ability to focus on one person's voice while ignoring others' voices and ambient noise, a phenomenon known as the cocktail party effect. Due to the powerful sound signal processing capabilities of the human auditory system, it is easy to separate mixed sounds. With the intelligentization of life, voice separation technology has played an important role in various voice interaction devices. However, for computers, how to efficiently realize voice separation has always been a difficult problem. [0003] At present, speech separation technology has a very wide range of applications. For examp...

AI Technical Summary

Problems solved by technology

[0013] To sum up, only one feature extractor is used to extract the sound of the full frequency band in the above literature, that is, the feature extractor used on each frequency is the same, and the effect of obtaining data is not good enough. In addition, the existing speaker The number of speakers has been fixed when designing the network parameters, that is, the model is static, and a fixed number of speakers must be used in the process of training and testing, and the number of inputs cannot be flexibly changed. Therefore, the existing speech separation technology needs to be further improved. improvement of

Images