Lip language recognition method fusing channel attention and selective feature fusion mechanism

Abstract

The invention discloses a lip language recognition method fusing a channel attention and selective feature fusion mechanism, and the method comprises the steps: 1, downloading a data set GRID for training a model, and carrying out the preprocessing of the data set; 2, building a lip language recognition network, and selecting a proper objective function to optimize model parameters; 3, evaluating the effect of the model by adopting corresponding evaluation indexes; and 4, performing lip language recognition on the video by using the trained model. According to the method, the stacked 3D convolutional neural network, the selective feature fusion network and the bidirectional GRU network are used for encoding the input video frame, a channel attention mechanism is added between every two 3D convolutional layers, and finally, the CTC decoder is used for generating the output text, so that the lip region features of a speaker can be better learned, and a more accurate lip reading effect is realized.

Description

technical field [0001] The invention belongs to the technical field of computer machine learning and artificial intelligence, and mainly relates to a lip language recognition method of a deep neural network. Background technique [0002] Lip language plays a vital role in human communication and speech comprehension, yet according to research, humans are poor at lip reading. Good lip language recognition technology can be a supplement to audio-based speech recognition, which can be used to improve hearing aids, improve the acquisition of language information in silent, safe, and noisy environments, etc., and has great practicality, so it has become an increasingly concerned topic. field. Before the advent of deep learning, most of the work on lip reading was based on hand-designed feature learning, which was computationally intensive and less accurate. Deep learning methods that have been developed in recent years are used to extract static features of the speaker's lip re...

AI Technical Summary

Problems solved by technology

However, there are relatively few studies on the sentence-level lip recognition method for complete sentence sequence prediction, and the existing models do not fully extract the features of the lip area, the accuracy of lip recognition is still low, and there is room for improvement.

Images