Lip language recognition method based on generative adversarial network and time convolutional network

Abstract

The invention discloses a lip language recognition method based on a generative adversarial network and a time convolutional network. According to the method, firstly, a lip deflection angle is judged through a ResNet angle classifier, then, a GAN two-stage converter is used for lip correction, and finally, input into TCN is carried out for feature recognition and classification to generate a lip language recognition result. According to the method, the influence of uncertainty of illumination intensity, illumination angle, recognition angle, speaker identity and the like in an actual environment, which cannot be solved by a traditional convolution model, on lip feature extraction is overcome, and the accuracy of lip language recognition is remarkably improved. According to the method, dense multi-angle lip change original data is designed, continuity of images of a single camera is achieved, continuity of lip images in an observation range is achieved to the maximum extent, the problem that an existing multi-angle model cannot process continuously-changing lip images in an actual environment is effectively solved, therefore, the lip language recognition precision is improved.

Description

technical field [0001] The invention belongs to the field of artificial intelligence and deep learning, in particular to a lip language recognition method based on a generative confrontation network and a temporal convolution network. Background technique [0002] With the development of science and technology and the improvement of hardware manufacturing level, the amount of information that computers can process is also increasing exponentially, which makes artificial intelligence technology based on deep learning enter a stage of rapid development, and artificial intelligence technology has become more and more It has been widely used in people's daily life, subtly changing people's production and lifestyle, and has become one of the indispensable and important technologies in human society. The application scenarios of artificial intelligence technology cover all aspects of production and life, including speech recognition, intelligent medical care, machine vision, intel...

AI Technical Summary

Problems solved by technology

However, the lip language recognition task deals with a longer continuous image sequence, the context is more closely connected, and the span of the time dimension is larger, and the accuracy of the attention mechanism in the lip language recognition task still needs to be improved.

Another major difficulty in the lip language recognition task is that lip features are often affected by angle, lighting, and speaker identity, and feature extraction faces great uncertainty

Most of the recognition models use a feature extractor based on Residual Network (ResNet), which works well in laboratory conditions, but does not perform well when applied directly in the actual environment.

Images