Complex scene voice recognition method and device based on multiple modes

Abstract

The invention discloses a complex scene voice recognition method based on multiple modes. The method comprises the following steps: synchronously collecting an audio signal, a lip image signal and a facial electromyogram signal corresponding to voice input if a collected lip image of a user is detected to change, determining multi-source data features of the signals in a space domain and a time domain, and coding and modeling the multi-source data features by using a speech recognition model to obtain common information of different modal expression contents and to obtain multi-modal speech information, and synthesizing a text by using a language model. The invention further discloses a complex scene voice recognition device based on multiple modes. The device comprises a data acquisitionmodule, a feature extraction module, a coding and decoding module, a text synthesis module and an interaction module. According to the invention, efficient, accurate and robust voice recognition in complex scene environments with vocal cord damage, high noise, high closure, high privacy requirements and the like is realized, and more reliable voice interaction technology and system are provided for complex man-machine interaction scenes.

Description

technical field [0001] The present invention relates to the technical field of speech recognition, in particular to a multimodal fusion-based collaborative interactive speech recognition method and device in a complex scene. Background technique [0002] Voice interaction is one of the most commonly used and direct communication methods between people. Speech recognition technology based on sound medium sprouted in the period of machine translation research in the 1950s. In recent years, with the development of artificial neural networks and machine learning algorithms, acoustic models based on deep learning have gradually been adopted in speech recognition. Speech recognition technology has made remarkable progress in recent years, and has been widely used in industry, communication, medical and other fields, and opened a new era of intelligent speech recognition and interaction. [0003] The traditional speech recognition technology that relies on the sound medium cannot...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem of limited application of single-modal speech recognition technology in complex scenes, the present invention proposes a multi-modal complex scene speech recognition method and device

Images