Lip language recognition-based method for improving speech comprehension degree of patient with severe hearing impairment

Abstract

The invention discloses a lip language recognition-based method for improving the speech comprehension degree of a patient with severe hearing impairment. The method comprises the steps: collecting alip motion image sequence from a real environment through image collection equipment, and enabling the lip motion image sequence to serve as an input feature of a deep neural network; constructing a visual modal voice endpoint detection method based on deep learning, and determining the position of a voice segment under the condition of a low signal-to-noise ratio; constructing a deep learning model based on a three-dimensional convolution-residual network-bidirectional GRU structure as a baseline model; constructing a lip language recognition model based on spatio-temporal information features on the basis of the baseline model; and training a network model by using the cross entropy loss, and identifying the speaking content according to the trained lip language identification model. According to the method, fine-grained features and time domain key frames of the lip language image are captured through space-time information feedback, so that the adaptability to the lip language features in a complex environment is improved, the lip language recognition performance is improved, the language understanding ability of a patient suffering from severe hearing impairment is improved, and the method has a good application prospect.

Description

Technical field[0001]The invention belongs to the field of speech comprehension, and in particular relates to a method for improving speech comprehension of patients with severe hearing impairment based on lip recognition.Background technique[0002]For patients with severe hearing impairment or higher, due to their severe psychological or physical hearing impairment, they can hardly understand the language content expressed by other speakers or even perceive the sound. Although hearing aids can partially improve the auditory perception of patients, their practical effects are limited for patients with severe hearing impairment. Although the auditory cochlea can improve the speech perception of patients with severe hearing impairment, there are certain risks because the auditory cochlea requires surgery.[0003]Moreover, in a complex actual environment, speech signals are often accompanied by various types of noise and instantaneous interference, especially under conditions of low signa...

AI Technical Summary

Problems solved by technology

[0010] Purpose of the invention: Aiming at the problem that speech hearing aid methods in the prior art have limited effect on patients with severe hearing impairment, the present invention discloses a method for improving speech comprehension of patients with severe hearing impairment based on lip recognition. By introducing spatial information fe

Images