Cross-modal multi-feature fusion audio and video speech recognition method and system

Abstract

The invention relates to an audio and video speech recognition technology, and provides a cross-modal multi-feature fusion audio and video speech recognition method and system in consideration of thesituation that speech interaction is easily affected by complex environmental noise, and facial motion information is acquired through a video and is relatively stable in an actual robot application environment. According to the method, speech information, visual information and visual motion information are fused through an attention mechanism, and the speech content expressed by a user is acquired more accurately by using the relevance among different modes, so that the speech recognition precision under the condition of complex background noise is improved, the speech recognition performance in human-computer interaction is improved, and the problem of low pure-speech recognition accuracy in a noise environment is effectively solved.

Description

technical field [0001] The invention relates to audio-video speech recognition technology, in particular to a cross-modal multi-feature fusion audio-video speech recognition method and system. Background technique [0002] The purpose of Automatic Speech Recognition (ASR) technology is to enable machines to "understand" human speech, and convert human speech information into readable text information, which is the key technology for realizing human-computer speech interaction. Among the various expressions of human beings, language contains the most abundant and precise information. With the gradual development of deep learning, the speech recognition rate in a quiet environment is higher than 95%, which has surpassed the recognition accuracy of humans. [0003] However, in the practical application of human-computer interaction, complex background noise will have a great impact on speech quality and speech intelligibility, seriously affecting speech recognition performance...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, considering that in the actual robot application environment, voice interaction is easily affected by complex environmental noise, and facial motion information is acquired through video and is relatively stable. The present invention provides an audio-video speech recognition method and system for cross-modal multi-feature fusion. The present invention fuses speech information, visual information and visual motion information through an attention mechanism, and utilizes the correlation between different modalities to achieve more Accurately acquire the voice content expressed by the user, improve the accuracy of voice recognition under complex background noise conditions, improve the performance of voice recognition in human-computer interaction, and effectively overcome the problem of low accuracy of pure voice recognition in noisy environments

Images