Chinese lip language recognition improving method and device based on pre-training language model

Abstract

The invention discloses a Chinese lip language recognition improving method and device based on a pre-training language model, and the method comprises the steps: constructing an error correction data set, and expanding the scale of the data set through a homonym enhancement and confusion matrix data enhancement method; the error correction data set is preprocessed, characters are mapped into corresponding tokens through a word segmentation model, and then the tokens are mapped into word embedding through an embedding layer. Performing feature extraction on the error correction sentence through a pre-training language model; according to the characteristics output by the pre-training language model, the position of an error character is output through the linear error prediction layer, and the linear error correction layer corrects the error character; training an error correction network based on an error detection loss function and an error correction loss function, and using a filling strategy in the training process to improve the error correction accuracy of the model; and improving a lip language recognition result based on the trained lip language error correction network. The method effectively reduces the character error rate of autoregression and non-autoregression models, and is easy to migrate to other models.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition, and relates to a Chinese lip reading recognition and error correction method and device, and a storage medium. Background Art

[0002] Visual speech recognition (VSR), also known as lip reading, refers to the process of transcribing speech from silent videos, involving three steps: lip localization, visual feature extraction, and sequence modeling. In particular, sequence modeling decodes the lip feature sequence generated by a deep neural network (DNN) backbone into text, ensuring semantic coherence in the transcription, and is therefore considered the most challenging module in lip reading.

[0003] From the perspective of the decoding paradigm, current sequence modeling methods can be divided into two main categories: autoregressive (AR) methods and non-autoregressive (NAR) methods. Taking advantage of the Transformer's advantages in sequence modeling, the latest lip reading models adopt the autoregressive paradigm, namely Generative Pre-trained Transformer (GPT), where tokens are generated sequentially. Despite its high recognition accuracy, GPT suffers from shortcomings in computational efficiency. Based on the independence assumption between output tokens, NAR methods are proposed to generate the entire sequence in parallel. In order to improve accuracy, some methods have to sacrifice the performance advantages brought by parallel reasoning. This is because they simply transfer non-autoregressive methods from the fields of machine translation and speech recognition without specific considerations for the lip reading task. Nevertheless, both AR and NAR models are limited by the decoder capacity and the size of the lip reading dataset. As a result, these models lack sufficient knowledge of human language, resulting in inaccurate predictions that often deviate from the language norm.

[0004] With the rise of large language models in the field of natural language processing, many works have tried to use pre-trained language models to improve the accuracy of lip reading recognition. Some of them use pre-trained language models as decoders to benefit lip reading recognition from their pre-training strategies, and some use pre-trained language models as teacher models for knowledge distillation to improve the language modeling capabilities of lip reading models.

[0005] However, the above method requires a lot of time for fine-tuning and cannot achieve good results due to the limitations of training data and the gap between audio and video modalities and text modalities. Summary of the invention

[0006] To solve the above problems, the present invention provides a method and device for improving Chinese lip reading recognition based on a pre-trained language model. According to the problems and possible causes of current lip reading recognition, the present invention proposes a method for correcting lip reading recognition results using a pre-trained language model. The method is independent of the structure of the lip reading recognition model and can be used as a universal plug-in, which has been proven to be effective on both AR and NAR models. The technical solution of the present invention is:

[0007] The first aspect of the present invention relates to a method for improving Chinese lip reading recognition based on a pre-trained language model, comprising the following steps:

[0008] Step 1: Build an error correction dataset and expand the dataset size through homophone enhancement and confusion matrix data enhancement methods;

[0009] Step 2: Preprocess the error correction data set, map the characters into corresponding tokens through the word segmentation model, and then map them into word embeddings through the embedding layer.

[0010] Step 3, extracting features of the error correction sentences through the pre-trained language model;

[0011] Step 4: The features output by the pre-trained language model are used to output the positions of the erroneous characters through the linear error prediction layer, and the linear error correction layer corrects the erroneous characters.

[0012] Step 5: Train the error correction network based on the error detection loss function and the error correction loss function, and use the padding strategy to improve the error correction accuracy of the model during the training process;

[0013] Step 6: Improve the lip reading recognition results based on the trained lip reading error correction network.

[0014] Furthermore, the step 1 comprises:

[0015] Step 1-1: Based on the labels of the CMLR dataset and the news release data, a corrected text set Y is constructed. The result of lip reading constitutes the original text X. Based on X and Y, an error index G is generated, {(X1, Y1, G1)…(X N ,Y N ,G N )} constitute an error correction data set;

[0016] Step 1-2, using the CMLR recognition results, statistical analysis is performed to create a confusion matrix. This matrix allows mapping X i →Y i Generate input X from labeled text corpus Y. Use confusion matrix data augmentation to expand the error correction dataset;

[0017] In steps 1-3, existing homophone tools are used to replace the original correction text Y with homophones, taking into account common dialect pronunciation errors, such as confusion between zh and z, eng and en, f and h, l and n, etc. Homophone data enhancement is used to expand the error correction dataset.

[0018] In steps 1-4, the recognition results of the CMLR test set are used as the test set of the error correction dataset, and the remaining data are randomly divided into training and validation sets.

[0019] Furthermore, in step 2, the input text is tokenized using the tokenizer of the pre-trained language model, and each character is mapped to a corresponding token. In the embedding layer, the input sequence token is mapped to a word embedding of a hidden size, and then added to the position embedding and the segment embedding to obtain the input sequence of the encoder;

[0020] Furthermore, in step 3, MacBERT is selected as a pre-trained language model for text feature extraction, and a similar Transformer encoder structure is adopted, which is composed of a series of identical modules, each of which contains a multi-head self-attention operation followed by a feedforward network. After this series of operations, the input text features can be obtained.

[0021] Further, the step 4 comprises:

[0022] Step 4-1, use a simple linear prediction layer to implement the detection network. Given the output of the pre-trained language model H = [h1, ...h n ] and the output error correction sequence G = [g1, ...g n ], for each character c in the text sequence S i , there is a corresponding error probability p i :

[0023] p i =P(g i =1|X)=Sigmoid(Wh i +b) (1)

[0024] Where W and b are the parameters of the detection network. After the error detection network, predict whether each character has an error;

[0025] Step 4-2, error correction requires predicting the wrong token as the correct token, which is the same as the correction task of the pre-training task "Masked Language Model" in MacBERT. Therefore, the same classification head as the pre-training task is used to correct the wrong characters. In this module, the output of the pre-trained language model H = [h1, ...h n ] passes through the linear projection layer, GELU activation function and LayerNorm layer in sequence. Then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S 1：N′ =[c′1,…,c′ N ].

[0026] Further, the step 5 comprises:

[0027] Step 5-1: Due to the limitation of the length of the input sentence sequence, the pre-trained model cannot flexibly map the tokens to different positions, so a filling strategy is adopted to solve the index correspondence difference. Special tokens from the word segmenter are used to fill the input sentence, so that the error correction network can map the corrected tokens to different positions in the filled sentence;

[0028] Step 5-2, fine-tune based on the provided pre-trained MacBERT model. The fine-tuning dataset consists of the original text X, the corrected text Y, and the error index G, represented as {(X1, Y1, G1)...(X N , Y N , G N )}.

[0029]

[0030]

[0031] The loss function of the error correction module consists of error detection loss and error correction loss. is the loss function of the error detection network. is the loss function of the error correction network. is the loss function of the error correction module, and ω is the weight of the error detection network loss.

[0032]

[0033] Further, the step 6 comprises:

[0034] Step 6-1: The lip reading recognition model uses a series of video frames F 1：t =[f1,…,f T ] as input, where x i ∈R H ×W×1 represents the preprocessed grayscale image frame. It generates a character sequence S 1：N =[c1,…,c N ] as a transcription, where c i ∈D v , D v is a dictionary of size v.

[0035] Step 6-1, transcribe the original 1：N As the input of the correction model, generate the corrected version S′ 1：N =[c′1,…,c′ N ].

[0036] The second aspect of the present invention relates to a Chinese lip reading recognition improvement device based on a pre-trained language model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the Chinese lip reading recognition improvement method based on a pre-trained language model of the present invention.

[0037] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the Chinese lip reading recognition improvement method based on a pre-trained language model of the present invention.

[0038] The present invention utilizes a pre-trained language model to enhance the language modeling capability of a speech recognition model, combines an error detection branch, and selects a pre-training strategy to improve the model's ability to detect and correct erroneous labels. In addition, the present invention designs a padding strategy and a data augmentation method to cope with the characteristics of visual speech recognition errors. Experimental results on the CMLR dataset show that Lip-CoNet reduces the character error rate (CER) of the autoregressive (AR) model by 0.58% (12.7%→12.12%) and the CER of the non-autoregressive (NAR) model by 1.86% (16.54%→14.68%). It is worth noting that the introduction of Lip-CoNet only leads to a slight increase in inference time and is easy to migrate to other models.

[0039] The beneficial effect of the present invention is that the error correction network proposed by the present invention is implemented on the existing lip reading recognition model to effectively reduce the character error rate (CER) of the autoregressive (AR) and non-autoregressive (NAR) models. Specifically, the CER of the AR model is reduced by 0.58%, while the CER of the NAR model is reduced by 1.86%, and the reasoning time is slightly increased. The method is applicable to AR and NAR models and is easy to migrate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 a and Figure 1 b is an analysis and example of the error type, where: Figure 1 a is a schematic diagram of the analysis process, Figure 1b is an example diagram. Homophone errors are measured by the pronunciation error rate (CERp). CERp is calculated by calculating the edit distance of incorrect phrase pairs in pinyin. If CERp <= 0.5, it is considered a homophone error. When an error occurs within a sentence, the number of words in the sentence will change. The ratio of the difference in the number of words is used to measure the error within the sentence.

[0042] Figure 2 a is a schematic diagram of the fine-tuning data preparation process; Figure 2 b is a schematic diagram of the error correction network architecture. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is explained and described clearly and completely below in conjunction with the accompanying drawings.

[0044] Example 1

[0045] A method for improving Chinese lip reading recognition based on a pre-trained language model comprises the following steps:

[0046] Step 1: Build an error correction dataset based on Figure 1 The error type shown, such as Figure 2 As shown in a, the dataset size is expanded by homophone enhancement and confusion matrix data enhancement methods;

[0047] Step 1-1, based on the labels of the CMLR dataset and the news release data, the corrected text set Y is formed, the result of lip reading constitutes the original text X, and the error index G is generated according to X and Y, {(X1, Y1, G1)...(X N , Y N , G N )} constitute an error correction data set;

[0048] Step 1-2, using the CMLR recognition results, statistical analysis is performed to create a confusion matrix. This matrix allows mapping X i →Y i Generate input X from labeled text corpus Y. Use confusion matrix data augmentation to expand the error correction dataset;

[0049] In steps 1-3, existing homophone tools are used to replace the original correction text Y with homophones, taking into account common dialect pronunciation errors, such as confusion between zh and z, eng and en, f and h, l and n, etc. Homophone data enhancement is used to expand the error correction dataset.

[0050] In steps 1-4, the recognition results of the CMLR test set are used as the test set of the error correction dataset, and the remaining data are randomly divided into training and validation sets.

[0051] Step 2: Preprocess the error correction data set, map the characters into corresponding tokens through the word segmentation model, and then map them into word embeddings through the embedding layer.

[0052] In step 2, the input text is tokenized using the tokenizer of the pre-trained language model, and each character is mapped to a corresponding token. In the embedding layer, the input sequence tokens are mapped to word embeddings of hidden size, which are then added to the position embedding and segment embedding to obtain the input sequence of the encoder;

[0053] Step 3, extracting features of the error correction sentences through the pre-trained language model;

[0054] In step 3, MacBERT is selected as the pre-trained language model for text feature extraction, using a similar Transformer encoder structure, which consists of a series of identical modules, each of which contains a multi-head self-attention operation followed by a feedforward network. After this series of operations, the input text features can be obtained.

[0055] Step 4, such as Figure 2 As shown in b, the features output by the pre-trained language model output the position of the erroneous character through the linear error prediction layer, and the linear error correction layer corrects the erroneous character;

[0056] Step 4-1, use a simple linear prediction layer to implement the detection network. Given the output of the pre-trained language model H = [h1, ...h n ] and the output error correction sequence G = [g1, ...g n ], for each character c in the text sequence S i , there is a corresponding error probability p i :

[0057] p i =P(g i =1|X)=Sigmoid(Wh i +b) (1)

[0058] Where W and b are the parameters of the detection network. After the error detection network, predict whether each character has an error;

[0059] Step 4-2, error correction requires predicting the wrong token as the correct token, which is the same as the correction task of the pre-training task "Masked Language Model" in MacBERT. Therefore, the same classification head as the pre-training task is used to correct the wrong characters. In this module, the output of the pre-trained language model H = [h1, ...h n] passes through the linear projection layer, GELU activation function and LayerNorm layer in sequence. Then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S 1：N′ =[c′1,…,c′ N ].

[0060] Step 5: Train the error correction network based on the error detection loss function and the error correction loss function, and use the padding strategy to improve the error correction accuracy of the model during the training process;

[0061] Step 5-1, such as Figure 2 As shown in a, due to the limitation of the length of the input sentence sequence, the pre-trained model cannot flexibly map the tokens to different positions, and a filling strategy is adopted to solve the difference in index correspondence. Special tokens from the word segmenter are used to fill the input sentence, so that the error correction network can map the corrected tokens to different positions in the filled sentence;

[0062] Step 5-2, fine-tune based on the provided pre-trained MacBERT model. The fine-tuning dataset consists of the original text X, the corrected text Y, and the error index G, represented as {(X1, Y1, G1)...(X N , Y N , G N )}.

[0063]

[0064]

[0065] like Figure 2 As shown in (b), the loss function of the error correction module consists of error detection loss and error correction loss. is the loss function of the error detection network. is the loss function of the error correction network. is the loss function of the error correction module, and ω is the weight of the error detection network loss.

[0066]

[0067] Step 6: Improve the lip reading recognition results based on the trained lip reading error correction network.

[0068] Step 6-1: The lip reading recognition model uses a series of video frames F 1：t =[f1,…,f T ] as input, where x i ∈R H ×W×1 represents the preprocessed grayscale image frame. It generates a character sequence S 1：N =[c1,…,cN ] as a transcription, where c i ∈D v , D v is a dictionary of size v.

[0069] Step 6-1, transcribe the original 1：N As the input of the correction model, generate the corrected version S′ 1：N =[c′1,…,c′ N ].

[0070] Example 2

[0071] The present embodiment relates to a Chinese lip reading recognition improvement device based on a pre-trained language model, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the Chinese lip reading recognition improvement method based on a pre-trained language model of Example 1.

[0072] Example 3

[0073] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the Chinese lip reading recognition improvement method based on a pre-trained language model of Embodiment 1 is implemented.

[0074] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for improving Chinese lip reading recognition based on a pre-trained language model, comprising the following steps: Step 1: Build an error correction dataset and expand the dataset size through the data enhancement method in step 1; Step 2: Preprocess the error correction data set, map the characters to corresponding tokens through the word segmentation model, and then map them to word embeddings through the embedding layer; Step 3, extracting features of the error correction sentences through the pre-trained language model; Step 4: The features output by the pre-trained language model are used to output the positions of the erroneous characters through the linear error prediction layer, and the linear error correction layer corrects the erroneous characters. Step 5: Train the error correction network based on the error detection loss function and the error correction loss function, and use the padding strategy to improve the error correction accuracy of the model during the training process; Step 6: Improve the lip reading recognition results based on the trained lip reading error correction network.

2. According to claim 1, a method for improving Chinese lip reading recognition based on a pre-trained language model is characterized in that: The step 1 comprises: Step 1-1: Based on the labels of the CMLR dataset and the news release data, a corrected text set Y is constructed. The result of lip reading constitutes the original text X. Based on X and Y, an error index G is generated, {(X1, Y1, G1)…(X N ,Y N ,G N )} constitute an error correction data set; Step 1-2, using the CMLR recognition results, statistical analysis is performed to create a confusion matrix; this matrix allows mapping X i →Y i Generate input X from labeled text corpus Y; use confusion matrix data augmentation to expand the error correction dataset; Steps 1-3: Use existing homophone tools to replace the original correction text Y with homophones, taking into account common dialect pronunciation errors, such as confusion between zh and z, eng and en, f and h, l and n, etc.; use homophone data enhancement to expand the correction dataset; In steps 1-4, the recognition results of the CMLR test set are used as the test set of the error correction dataset, and the remaining data are randomly divided into training and validation sets.

3. According to claim 1, a method for improving Chinese lip reading recognition based on a pre-trained language model, characterized in that: In step 2, the input text is segmented using the tokenizer of the pre-trained language model, and each character is mapped to a corresponding token; in the embedding layer, the input sequence token is mapped to a word embedding of a hidden size, and then added to the position embedding and the segment embedding to obtain the input sequence of the encoder.

4. According to claim 1, a method for improving Chinese lip reading recognition based on a pre-trained language model, characterized in that: In the step 3, MacBERT is selected as the pre-trained language model for text feature extraction, and a similar Transformer encoder structure is adopted, which is composed of a series of identical modules, each of which contains a multi-head self-attention operation followed by a feedforward network; after this series of operations, the input text features can be obtained.

5. According to claim 1, a method for improving Chinese lip reading recognition based on a pre-trained language model, characterized in that: The step 4 comprises: Step 4-1, use a simple linear prediction layer to implement the detection network; given the output of the pre-trained language model H = [h1,…h n ] and the output error correction sequence G = [g1,…g n ], for each character c in the text sequence S i , there is a corresponding error probability p i : p i =P(g i =1|X)=Sigmoid(Wh i +b) (1) Where W and b are the parameters of the detection network; after the error detection network, predict whether each character has an error; Step 4-2, error correction requires predicting the wrong token as the correct token, which is the same as the correction task of the pre-training task "Masked Language Model" in MacBERT; therefore, the same classification head as the pre-training task is used to correct the wrong characters; in this module, the output H of the pre-trained language model is [h1,…h n ] passes through the linear projection layer, GELU activation function and LayerNorm layer in sequence; then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S 1∶N′ =[c′1,…,c′ N ].

6. According to claim 1, a method for improving Chinese lip reading recognition based on a pre-trained language model, characterized in that: The step 5 comprises: Step 5-1: Due to the limitation of the length of the input sentence sequence, the pre-trained model cannot flexibly map the tokens to different positions. A filling strategy is adopted to solve the difference in index correspondence. The input sentence is filled with special tokens from the tokenizer, so that the error correction network can map the corrected tokens to different positions in the filled sentence. Step 5-2, fine-tune based on the provided pre-trained MacBERT model; the fine-tuning dataset consists of the original text X, the corrected text Y, and the error index G, represented as {(X1,Y1,G1)…(X N ,Y N ,G N )}; The loss function of the error correction module consists of error detection loss and error correction loss; is the loss function of the error detection network; is the loss function of the error correction network; is the loss function of the error correction module, and ω is the weight of the error detection network loss:

7. The method for improving Chinese lip reading recognition based on a pre-trained language model according to claim 1, characterized in that: The step 6 comprises: Step 6-1: The lip reading recognition model uses a series of video frames F 1:t =[f1,…,f T ] as input, where x i ∈R H×W×1 represents the preprocessed grayscale image frame; it generates a character sequence S 1:N =[c1,…,c N ] as a transcription, where c i ∈D v , D v is a dictionary of size v; Step 6-2, transcribe the original 1:N As the input of the correction model, generate the corrected version S′ 1:N =[c′1,…,c′ N ].

8. A Chinese lip reading recognition improvement device based on a pre-trained language model, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the Chinese lip reading recognition improvement method based on a pre-trained language model as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the Chinese lip reading recognition improvement method based on a pre-trained language model described in any one of claims 1-7 is implemented.

Images