An adversarial double-contrast self-supervised learning method for cross-modal lip reading

Abstract

The present invention proposes an adversarial double-contrast self-supervised learning method for cross-modal lip reading, which includes a visual encoder, an audio encoder, two multi-scale temporal convolutional networks with average pooling, an identity discriminator, and a model state classifier. The method learns effective visual representations by combining audio-visual synchrony-based dual contrastive learning, identity adversarial training, and modality adversarial training. In double-contrastive learning, the noise-contrastive estimate is used as the training objective to distinguish real samples from noise samples. In the adversarial training, an identity discriminator and a modality classifier are proposed for audio-visual representation, the identity discriminator is used to distinguish whether the input visual features have a common identity, and the modality classifier is to predict whether the input features belong to the visual modality or not. The audio modality is then utilized for adversarial training using a momentum gradient reversal layer.

Description

technical field [0001] The invention belongs to the field of image processing, and in particular relates to an adversarial double-contrast self-supervised learning method for cross-modal lip reading. Background technique [0002] Supervised deep learning has made revolutionary progress in many fields such as image classification, object detection and segmentation, speech recognition, machine translation, etc. Although supervised learning has made remarkable progress in the past few years, its success largely relies on large amounts of human-annotated training data. However, for some specific tasks, such as lip reading, the cost of annotation can be prohibitively expensive. In recent years, self-supervised learning has attracted increasing attention due to its high labeling efficiency and good generalization ability. Self-supervised learning methods have shown great potential in natural language processing, computer vision, and cross-modal representation learning. [0003]...

AI Technical Summary

Problems solved by technology

Previous studies in this area have attempted pairwise comparison strategies to bring visual embeddings closer to corresponding audio embeddings and further away from non-corresponding audio embeddings. Two-contrast learning requires manual selection of negative samples, and its effect largely depends on the effectiveness of negative samples; second, representation learning only relies on synchronized audio-video data pairs, and other self-supervised signals, such as speaker-related information and modality information, can also be used to optimize the quality of learned representations, but these self-monitoring signals are usually ignored in previous work

Images