Cross-modal lip language recognition method

Abstract

The invention provides a cross-modal lip language recognition method. The cross-modal lip language recognition method comprises the steps of S1, data preprocessing: acquiring a lip region of video data and a feature matrix of audio data; s2, model training: sequentially carrying out the steps of speaker recognition task training, cross-modal contrast learning, model parameter and lip language characteristic standardization and the like until the model converges; and S3, model deployment: only inputting a to-be-recognized non-training data video sequence, obtaining the lip shape characteristics of a speaker by using a visual recognition branch, standardizing the lip language characteristics, and finally performing mapping from the lip language characteristics to characters. According to the method, visual features with better distinguishability can be extracted on the premise of not additionally manually annotating data, the generalization and robustness of feature extraction are better, the method can be used across speakers, and a group of model parameters do not need to be independently trained for each category of samples.

Description

technical field [0001] The invention relates to the field of recognition, in particular to a cross-modal lip language recognition method. Background technique [0002] Lip language recognition is a visual language recognition technology, which mainly uses the lip movement information in the video, combined with the language recognition technology of language prior knowledge and context information. Lip recognition plays an important role in both language understanding and communication, and is often used when effective audio information is not available. It also has extremely high application value and can be applied to the treatment of speech-impaired patients, the field of security, military equipment and human-computer interaction. [0003] The limitation of traditional lip recognition methods is that they only focus on video input information, and cannot learn better visually separable features without the guidance of additional experience knowledge. Therefore, these m...

AI Technical Summary

Problems solved by technology

[0003] The limitation of traditional lip recognition methods is that they only focus on video input information, and cannot learn better visually separable features without additional experience and knowledge guidance.

Therefore, these methods usually rely on a large amount of accurately labeled data, however, the cost of obtaining labeled data in real life is prohibitively high

Images