A lip reading method for unseen speakers
By introducing prior knowledge and adversarial learning strategies into the lip-reading method, and dynamically adjusting the neural network parameters, an end-to-end system is constructed, which solves the problem of lip-reading of unfamiliar speakers and improves the generalization ability on small-scale datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU RES INST OF XIAN UNIV OF ELECTRONIC SCI & TECH
- Filing Date
- 2022-09-20
- Publication Date
- 2026-07-03
Smart Images
Figure CN115497136B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of lip reading technology, and more specifically to a lip reading method applicable to speakers who have never been seen before. Background Technology
[0002] Lip reading is a visual language recognition technology that primarily utilizes lip movement information from videos, combined with prior language knowledge and contextual information. It is often used when effective audio or text information is unavailable. Its applications are extremely valuable, including video understanding, security, military equipment, human-computer interaction, and the treatment of speech disorders.
[0003] On limited datasets, traditional lip-reading methods can only recognize the visual lip features of speakers used during model training or speakers who are very similar to them. They cannot accurately capture key lip information from unfamiliar or unseen speakers, and collecting massive amounts of real-world lip-reading data is impractical. This method leverages prior knowledge of the facial and lip relationships between speakers to guide the neural network feature extractor. It specifically adapts efficient feature extraction methods to the speakers currently input into the neural network. With the assistance of adversarial learning strategies, the neural network can extract rich, speaker-indistinguishable lip-reading features, enabling the proposed method to be extended to data with unfamiliar or unseen speakers, greatly alleviating its dependence on the size of the training set. Summary of the Invention
[0004] The purpose of this invention is to provide a lip-reading method suitable for speakers who have never been seen before, and to solve the following technical problems:
[0005] Existing lip reading methods can only recognize the visual lip features of speakers used during model training or speakers who are very similar to them, and cannot accurately obtain key lip information of unfamiliar or unseen speakers.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] A lip-reading method suitable for speakers not previously seen includes the following steps:
[0008] S1. Data Preprocessing:
[0009] Perform face recognition on videos or images obtained after frame extraction, and crop the area containing the lips;
[0010] S2, Model Training:
[0011] S21. Model prior knowledge on face / lip data, update module parameters through unsupervised and self-supervised learning algorithms, and obtain feature expressions and related relationships of different speakers;
[0012] S22. The video data is sent to the lip reading recognition module, which contains a dynamic feature extractor that dynamically acquires and combines basic features based on the input speaker representation obtained in S21, and trains the neural network to perform lip reading recognition on the input speaker.
[0013] S23. The intermediate features of lip reading recognition are fed into the speaker classification module. Under the adversarial learning strategy, the module distinguishes the intermediate features belonging to different speakers. At the same time, the lip reading recognition module is encouraged to ignore the speaker's personality features and only learn the lip reading related features. During the training phase, the lip reading recognition and speaker classification modules alternately update the parameters.
[0014] S3, Model Deployment:
[0015] Input a video sequence of an unfamiliar / unseen speaker (not training data). The prior knowledge module provides the speaker's feature representation and guides the dynamic feature extractor to dynamically acquire and combine basic features and perform lip reading.
[0016] As a further aspect of the present invention: repeat S22-S23 until the loss function value no longer decreases in multiple consecutive training rounds after the learning rate decays, i.e., the model converges.
[0017] As a further aspect of the present invention: the model includes a prior knowledge modeling module, a lip-reading recognition module, and a speaker classification module;
[0018] The prior knowledge modeling module is used to obtain the feature expressions and related relationships of different speakers;
[0019] The lip reading recognition module is used to convert lip reading features into text output;
[0020] The speaker classification module is used to distinguish the intermediate features of different speakers, and the lip reading recognition module is encouraged to ignore the speaker's individual characteristics and only learn lip reading related features.
[0021] As a further aspect of the present invention: the lip reading recognition module includes a dynamic feature extractor, which is used to dynamically acquire and combine basic features.
[0022] As a further aspect of the present invention, the model input-output relationship specifically includes the following steps:
[0023] Image of the speaker's face / lips to be identified → Prior knowledge modeling module → Speaker feature representation;
[0024] Feature representation of the video sequence to be identified & feature representation of the speaker to be identified → lip reading module (dynamic feature extractor) → lip reading module (the rest) → output text;
[0025] Feature representation of the video sequence to be identified & feature representation of the speaker to be identified → lip reading module (intermediate features) → speaker classification module → adversarial learning.
[0026] As a further aspect of the present invention: the connectionist temporal classification loss function and the cross-entropy loss are used to constrain the lip reading module and the adversarial learning module respectively, and the reconstruction loss is used to assist in the modeling of prior knowledge.
[0027] The beneficial effects of this invention are:
[0028] Compared to the traditional two-stage lip-reading recognition method that first performs feature engineering and then constructs a classifier, this application uses deep learning to build an end-to-end lip-reading recognition system, eliminating the need to train a separate set of model parameters for each category of samples. Dynamically adjusting network parameters improves the generalization and robustness of the extracted features, making them applicable to unfamiliar / unseen speakers. This alleviates the dependence on the size of the training set, allowing for strong generalization even when trained on small datasets.
[0029] By employing unsupervised learning, this method enables the neural network responsible for modeling prior knowledge to learn the relationships between speakers from a large number of face and lip images without requiring manually labeled data. Through dynamically adjusting network parameters, this method enhances the feature extraction capability of a simple end-to-end neural network for unfamiliar / unseen speakers, significantly expanding the generalization ability of lip-reading neural networks. Attached Figure Description
[0030] The invention will now be further described with reference to the accompanying drawings.
[0031] Figure 1 This is a schematic diagram of a lip-reading recognition method applicable to speakers who have never been seen before, according to the present invention.
[0032] Figure 2 This is a schematic diagram of the overall process of the present invention;
[0033] Figure 3 This is a flowchart illustrating the prior knowledge modeling module of the present invention;
[0034] Figure 4 This is a flowchart illustrating the speaker classification module of the present invention;
[0035] Figure 5 This is a schematic diagram of the lip reading recognition module of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] This invention discloses a lip-reading recognition method suitable for speakers who have never been seen before. The core idea of the algorithm is to utilize prior knowledge to dynamically adjust the parameters of the neural network to better adapt to lip-reading data from unfamiliar speakers. This reduces the dependence on massive training data and enables strong generalization ability even when trained on small datasets.
[0038] Please see Figure 1-5 As shown, a lip-reading recognition method suitable for speakers who have never been seen includes the following steps:
[0039] S1. Data Preprocessing:
[0040] Perform face recognition on videos or images obtained after frame extraction, and crop the area containing the lips;
[0041] S2, Model Training:
[0042] S21. Model prior knowledge on face / lip data, update module parameters through unsupervised and self-supervised learning algorithms, and obtain feature expressions and related relationships of different speakers;
[0043] S22. The video data is sent to the lip reading recognition module, which contains a dynamic feature extractor that dynamically acquires and combines basic features based on the input speaker representation obtained in S21, and trains the neural network to perform lip reading recognition on the input speaker.
[0044] S23. The intermediate features of lip reading recognition are fed into the speaker classification module. Under the adversarial learning strategy, the module distinguishes the intermediate features belonging to different speakers. At the same time, the lip reading recognition module is encouraged to ignore the speaker's personality features and only learn the lip reading related features. During the training phase, the lip reading recognition and speaker classification modules alternately update the parameters.
[0045] Repeat steps S22-S23 until the loss function value no longer decreases in multiple training rounds after the learning rate decays, indicating that the model has converged.
[0046] S3, Model Deployment:
[0047] Input a video sequence of an unfamiliar / unseen speaker (not training data). The prior knowledge module provides the speaker's feature representation and guides the dynamic feature extractor to dynamically acquire and combine basic features and perform lip reading.
[0048] The model includes a prior knowledge modeling module, a lip reading recognition module, and a speaker classification module; the lip reading recognition module is further divided into dynamic feature extraction and the remaining parts.
[0049] The prior knowledge modeling module is used to obtain the feature expressions and related relationships of different speakers, and generate speaker feature expressions to guide the lip reading recognition module to extract the current speaker's features in a targeted manner.
[0050] The lip-reading recognition module is used to convert lip-reading features into text output; the lip-reading recognition module contains a dynamic feature extractor, which is used to dynamically acquire and combine basic features;
[0051] The speaker classification module is used to distinguish the intermediate features of different speakers, and the lip reading recognition module is encouraged to ignore the speaker's personality features and only learn lip reading related features. Furthermore, the adversarial learning strategy helps the lip reading recognition module obtain speaker-independent lip reading features.
[0052] The specific steps involved in the model's input-output relationship are as follows:
[0053] Image of the speaker's face / lips to be identified → Prior knowledge modeling module → Speaker feature representation;
[0054] Feature representation of the video sequence to be identified & feature representation of the speaker to be identified → lip reading module (dynamic feature extractor) → lip reading module (the rest) → output text;
[0055] Feature representation of the video sequence to be identified & feature representation of the speaker to be identified → lip reading module (intermediate features) → speaker classification module → adversarial learning.
[0056] The prior knowledge modeling module can acquire the feature representations and related relationships of different speakers; the lip-reading main module includes a lip-reading recognition module (dynamic feature extraction) and a lip-reading recognition module (the rest). The lip-reading main module is the main structure responsible for lip-reading recognition. The lip-reading recognition module (dynamic feature extraction) represents the dynamic lip-reading feature extractor, which receives the representation of the current speaker and dynamically adjusts the neural network parameters contained in the feature extractor based on this. The lip-reading recognition module (the rest) includes some necessary structures in the other lip-reading process, such as recurrent neural networks for time series modeling and multilayer perceptrons for classification; the speaker classification module classifies the intermediate features of lip-reading recognition and determines whether the features clearly belong to a certain speaker. Under the adversarial learning strategy, the speaker classifier can encourage the lip-reading recognition structure to learn lip-reading features that do not distinguish between speakers.
[0057] In the prior knowledge modeling module
[0058] Autoencoders (prior knowledge modeling modules) and similar technologies can utilize unsupervised learning to model prior knowledge. Specifically, we train the reconstruction capability of the prior knowledge modeling module, expecting it to possess two capabilities: (1) reducing the input information to a representative speaker representation; and (2) ensuring the acquired speaker representation has sufficient information to be restored to the original input. The obtained representative speaker representation provides guidance for dynamically extracting lip-reading features. Figure 3 The diagram illustrates the training process of the prior knowledge modeling module. The prior knowledge modeling module first compresses the speaker data to be identified to obtain a speaker representation, and then reconstructs the input information from this representation as much as possible through data reconstruction to ensure that the obtained representation contains important original information.
[0059] In the lip-reading main module
[0060] The lip-reading recognition module can be divided into two parts:
[0061] 1) Lip reading module (dynamic feature extraction): In order to enable the neural network to dynamically adjust the network parameters for different speakers, this module contains a set of basic feature extractors, which are responsible for acquiring basic visual features. Then, based on the speaker representation provided by the prior knowledge modeling module, the basic features are adjusted to provide information for the current speaker to the other lip reading components.
[0062] 2) Lip reading module (the rest): This includes a recurrent neural network for time series modeling, a multilayer perceptron for classification, etc. This part of the structure converts the speaker's specific lip reading information into text output.
[0063] Since the input and output text lengths of lip-reading videos differ, leading to alignment issues, we use the Connectionist Temporal Classification loss function to constrain the lip-reading recognition module. For example... Figure 5 The diagram illustrates the training process for lip-reading recognition. The lip-reading recognition module dynamically extracts short-term features from the input speaker data based on speaker representation, then establishes long-term contextual relationships and maps them to text.
[0064] In the speaker classification module,
[0065] To obtain more robust lip-reading features from unfamiliar / unseen speakers, we employ an adversarial learning strategy in our algorithm. By alternately training the speaker classifier and the lip-reading module, the lip-reading module is encouraged to ignore speaker personality traits and learn only lip-reading-related features. Specifically, within the adversarial learning framework, the speaker classification module identifies which speaker a middle lip feature belongs to, while the lip-reading module is encouraged to learn lip-reading information that does not include speaker personality traits. This makes it less likely that the middle feature will be easily identified as belonging to a particular speaker; in other words, the speaker classification module and the lip-reading module compete against each other on the speaker separability of the middle feature. Figure 4 The diagram illustrates a training process for an adversarial learning strategy. After dynamically extracting short-term features, the speaker classifier receives intermediate features from the lip-reading module, determines the speaker to which the feature belongs, and then, based on the obtained information, encourages the lip-reading module to ignore speaker personality features and learn only lip-related features. This process is repeated alternately during training.
[0066] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.
Claims
1. A lip-reading recognition method suitable for speakers never seen before, characterized in that, Includes the following steps: S1. Data Preprocessing: Perform face recognition on videos or images obtained after frame extraction, and crop the area containing the lips; S2, Model Training: S21. Prior knowledge modeling is performed on the face / lip data, and the module parameters are updated through unsupervised and self-supervised learning algorithms to obtain the feature expressions and related relationships of different speakers; wherein, the feature expression of the speaker is obtained by inputting the face or lip image of the speaker to be identified into the prior knowledge modeling module; S22. The video data is sent to the lip reading recognition module, which contains a dynamic feature extractor that dynamically acquires and combines basic features based on the input speaker representation obtained in S21, and trains the neural network to perform lip reading recognition on the input speaker. S23. Input the intermediate features of lip reading recognition into the speaker classification module. Under the adversarial learning strategy, the speaker classification module distinguishes the personality features of different speakers, while the lip reading recognition module ignores the speaker's personality features and only learns lip-related features. During the training process, the lip reading recognition module and the speaker classification module alternately update the parameters. During training, the model is optimized by reconstructing the loss function, the CTC loss function, and the cross-entropy loss function; S3, Model Deployment: The input video sequence of an unfamiliar / unseen speaker is used as input. The prior knowledge module provides the feature representation of the input speaker and guides the dynamic feature extractor to dynamically acquire and combine basic features and perform lip reading recognition. The input video sequence of an unfamiliar / unseen speaker is non-training data.
2. The lip-reading recognition method for speakers never seen before, as described in claim 1, is characterized in that... Repeat steps S22-S23 until the loss function value no longer decreases in multiple training rounds after the learning rate decays, indicating that the model has converged.
3. The lip-reading recognition method for speakers never seen before, as described in claim 1, is characterized in that... The model includes a prior knowledge modeling module, a lip-reading recognition module, and a speaker classification module; The prior knowledge modeling module is used to obtain the feature expressions and related relationships of different speakers; The lip reading recognition module is used to convert lip reading features into text output; The speaker classification module is used to distinguish the intermediate features of different speakers, and the lip reading recognition module is encouraged to ignore the speaker's individual characteristics and only learn lip reading related features.
4. The lip-reading recognition method for speakers never seen before, as described in claim 3, is characterized in that... The lip reading module includes a dynamic feature extractor, which is used to dynamically acquire and combine basic features.
5. A lip-reading recognition method applicable to speakers never seen before, as described in claim 3, is characterized in that... The specific steps involved in the model's input-output relationship are as follows: Input the face or lip image of the speaker to be identified into the prior knowledge modeling module, and output the speaker feature representation; The speaker's features are input into the lip-reading module, intermediate features are extracted, and the text is output. The intermediate features are input into the speaker classification module for adversarial learning.