Chinese lip speech recognition enhancement method and device based on pre-trained language model

Abstract

The application discloses a Chinese lip-speech recognition improvement method and device based on a pre-trained language model, which comprises the following steps: constructing a correction data set, expanding the data set scale through homonym enhancement and confusion matrix data enhancement methods; preprocessing the correction data set, mapping characters into corresponding tokens through a word segmentation model, and then mapping the tokens into word embeddings through an embedding layer; extracting features of the corrected sentences through a pre-trained language model; outputting the positions of error characters through a linear error prediction layer from the features output by the pre-trained language model; correcting the error characters through a linear error correction layer; training a correction network based on an error detection loss function and an error correction loss function, and using a padding strategy to improve the correction accuracy of the model during the training process; and improving the lip-speech recognition result based on the trained lip-speech correction network. The application effectively reduces the character error rate of the autoregressive and non-autoregressive models, and is easy to migrate to other models.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition, and relates to a Chinese lip speech recognition and correction method and device and a storage medium. BACKGROUND

[0002] Visual speech recognition (VSR), also known as lip reading, refers to the process of transcribing speech from silent videos, involving three steps: lip localization, visual feature extraction, and sequence modeling. In particular, sequence modeling decodes the sequence of lip features generated by a deep neural network (DNN) backbone into text, ensuring semantic coherence in transcription, and is therefore considered the most challenging module in lip reading.

[0003] From the perspective of decoding paradigm, current sequence modeling methods are mainly divided into two categories: autoregressive (AR) methods and non-autoregressive (NAR) methods. Taking advantage of the Transformer in sequence modeling, the latest lip reading models adopt the autoregressive paradigm, i.e., generating pre-trained transformers (GPT), in which tokens are sequentially generated. Although the recognition accuracy is high, GPT has deficiencies in computational efficiency. Based on the independent assumption between output tokens, NAR methods are proposed to generate the entire sequence in parallel. In order to improve accuracy, some methods have to sacrifice the performance advantage brought by parallel reasoning. This is because they simply transfer non-autoregressive methods from machine translation and speech recognition fields, lacking specific considerations for the lip reading task. Despite this, both AR and NAR models are limited by the decoder's ability and the size of the lip reading dataset. Therefore, these models lack sufficient human language knowledge, leading to inaccurate predictions that often deviate from language norms.

[0004] With the rise of large language models in the field of natural language processing, there are many works trying to use pre-trained language models to improve the accuracy of lip speech recognition. Some of them use pre-trained language models as decoders to benefit from their pre-training strategies, and some use pre-trained language models as teacher models for knowledge distillation to improve the language modeling ability of the lip speech model.

[0005] However, the above methods require a lot of time for fine-tuning, are limited by the data for training and the gap between audio-visual modalities and text modalities, and cannot achieve good results. SUMMARY

[0006] To solve the above problems, the application provides a Chinese lip speech recognition improvement method and device based on a pre-trained language model. According to the problems and possible causes of current lip speech recognition, the application proposes a method of correcting lip speech recognition results using a pre-trained language model. This method is independent of the structure of the lip speech recognition model and can be used as a general plug-in, which has been proven to be effective on both AR and NAR models. The technical solution of the application is:

[0007] The first aspect of the present application relates to a Chinese lip speech recognition improvement method based on a pre-trained language model, comprising the following steps:

[0008] Step 1, build a correction data set, expand the data set size through homophone enhancement and confusion matrix data enhancement method;

[0009] Step 2, pre-process the correction data set, map the characters to corresponding tokens through a word segmentation model, and then map them to word embeddings through an embedding layer.

[0010] Step 3, extract features from the corrected sentence through a pre-trained language model;

[0011] Step 4, the features output by the pre-trained language model output the position of the error character through a linear error prediction layer, and the linear error correction layer corrects the error character;

[0012] Step 5, train the error correction network based on the error detection loss function and the error correction loss function, and use the padding strategy to improve the error correction accuracy of the model during training;

[0013] Step 6, improve the lip speech recognition result based on the trained lip speech error correction network.

[0014] Further, the step 1 comprises:

[0015] Step 1-1, based on the label of the CMLR data set and the news script data, build a correction text set Y, the result of reading lips constitutes the original text X, generate error index G according to X, Y, {(X1, Y1, G1)…(X N ,Y N ,G N} constitute a correction data set;

[0016] Step 1-2, use the CMLR recognition result to perform statistical analysis to create a confusion matrix. The matrix allows the generation of input X from the label text corpus Y by mapping X i →Y i Use confusion matrix data enhancement to expand the error correction data set;

[0017] Step 1-3, use existing homophone tools to replace the original correction text Y with homophones, consider common dialect pronunciation errors, such as confusion between zh and z, eng and en, f and h, l and n, etc. Use homophone data enhancement to expand the error correction data set.

[0018] Step 1-4, the recognition result of the CMLR test set is used as the test set of the error correction data set, and the remaining data is randomly divided into training set and validation set.

[0019] Further, in step 2, the tokenizer of the pre-trained language model is used to segment the input text, and each character is mapped to the corresponding token. In the embedding layer, the input sequence token is mapped to the word embedding of the hidden size, and then added with the position embedding and segment embedding to obtain the input sequence of the encoder;

[0020] Further, in step 3, MacBERT is selected as the pre-trained language model for text feature extraction, which adopts a similar Transformer encoder structure composed of a series of identical modules, each module containing a multi-head self-attention operation followed by a feedforward network. After this series of operations, the input text features can be obtained.

[0021] Further, step 4 includes:

[0022] Step 4-1, a simple linear prediction layer is used to implement the detection network. Given the output H = [h1, … h n ] of the pre-trained language model and the output error correction sequence G = [g1, … g n ], for each character c i in the text sequence S, there is a corresponding error probability p i :

[0023] p i = P(g i = 1 | X) = Sigmoid(Wh i +b) (1)

[0024] where W and b are the parameters of the detection network. After the error detection network, it predicts whether each character has an error or not;

[0025] Step 4-2, error correction needs to predict the wrong token as the correct token, which is the same as the correction task of the pre-training task "masked language model" in MacBERT. Therefore, the same classification head as the pre-training task is used for error character correction. In this module, the output H = [h1, … h n ] of the pre-trained language model is sequentially passed through a linear projection layer, a GELU activation function, and a LayerNorm layer. Then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S 1：N′ = [c'1, …, c' N ].

[0026] Further, step 5 includes:

[0027] Step 5-1, due to the length limit of the input sentence sequence, the pre-trained model cannot flexibly map the token to different positions, so a padding strategy is adopted to solve the difference of index correspondence. Special tokens from the tokenizer are used to pad the input sentence, so that the correction network can map the corrected token to different positions in the padded sentence;

[0028] Step 5-2, based on the provided pre-trained MacBERT model, fine-tuning is performed. The fine-tuning dataset consists of original text X, corrected text Y and error index G, denoted as {(X1, Y1, G1)...(X N , Y N , G N )}.

[0029]

[0030]

[0031] The loss function of the correction module is composed of error detection loss and error correction loss. is the loss function of the error detection network. is the loss function of the error correction network. is the loss function of the correction module, and ω is the weight of the error detection network loss.

[0032]

[0033] Further, the step 6 includes:

[0034] Step 6-1, the lip reading model takes a series of video frames F 1：t =[f1, …, f T ] as input, where x i ∈R H ×W×1 represents the preprocessed grayscale image frame. It generates a character sequence S 1：N =[c1, …, c N ] as transcription, where c i ∈D v , D v is a dictionary of size v.

[0035] Step 6-1, taking the original transcription S 1：N as the input of the correction model, the corrected version S′ 1：N =[c′1, …, c′ N ] is generated.

[0036] The second aspect of the present application relates to a pre-trained language model-based Chinese lip speech recognition improvement device, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the pre-trained language model-based Chinese lip speech recognition improvement method of the present application.

[0037] The third aspect of the present application relates to a computer-readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the pre-trained language model-based Chinese lip speech recognition improvement method of the present application.

[0038] The present application enhances the language modeling capability of the speech recognition model by using the pre-trained language model, combines the error detection branch, and selects the pre-training strategy to improve the ability of the model to detect and correct the error label. In addition, the present application designs a padding strategy and a data enhancement method to deal with the characteristics of visual speech recognition errors. The experimental results on the CMLR dataset show that Lip-CoNet reduces the character error rate (CER) of the autoregressive (AR) model by 0.58% (12.7% to 12.12%), and the CER of the non-autoregressive (NAR) model by 1.86% (16.54% to 14.68%). Notably, the introduction of Lip-CoNet only leads to a slight increase in inference time and is easy to migrate to other models.

[0039] The present application has the beneficial effect that the error correction network proposed in the present application effectively reduces the character error rate (CER) of the autoregressive (AR) and non-autoregressive (NAR) models on the existing lip speech recognition model. Specifically, the CER of the AR model is reduced by 0.58%, and the CER of the NAR model is reduced by 1.86%, with a slight increase in inference time. This method is applicable to AR and NAR models and is easy to migrate. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0041] Figure 1 a and Figure 1 b is the analysis and example of error type, wherein, Figure 1 a is an analysis flowchart, Figure 1b is an example graph. Homophone errors are measured by the pronunciation error rate (CERp). CERp is calculated by computing the edit distance of incorrect word pairs in the pinyin. If CERp <= 0.5, it is considered a homophone error. When errors occur within a sentence, the number of words in the sentence will change. The ratio of the difference in the number of words is used to measure the error within the sentence.

[0042] Figure 2 a is a fine-tuning data preparation process schematic diagram; Figure 2 b is a schematic diagram of the error correction network architecture. DETAILED DESCRIPTION

[0043] The technical solutions of the present application will be clearly and completely explained and described below in combination with the drawings.

[0044] Example 1

[0045] A Chinese lip speech recognition improvement method based on a pre-trained language model, comprising the following steps:

[0046] Step 1, build an error correction dataset, according to Figure 1 the error types shown in the figure, such as Figure 2 a shown by the homophone enhancement and confusion matrix data enhancement method to expand the size of the dataset;

[0047] Step 1-1, based on the labels of the CMLR dataset and the news script data, build a correction text set Y, the result of reading lips constitutes the original text X, according to X, Y generate error index G, {(X1, Y1, G1)...(X N , Y N , G N} constitute an error correction dataset;

[0048] Step 1-2, use the CMLR recognition result to perform statistical analysis to create a confusion matrix. The matrix allows the generation of input X from the labeled text corpus Y by mapping X i →Y i Use confusion matrix data augmentation to expand the error correction dataset;

[0049] Step 1-3, use existing homophone tools to replace the original correction text Y with homophones, taking into account common dialect pronunciation errors, such as confusion between zh and z, eng and en, f and h, l and n, etc. Use homophone data augmentation to expand the error correction dataset.

[0050] Step 1-4, the recognition result of the CMLR test set is used as the test set of the error correction data set, and the remaining data is randomly divided into a training set and a validation set.

[0051] Step 2, the pre-processing of the error correction dataset, mapping characters to corresponding tokens by a tokenizer model, and then mapping to word embeddings by an embedding layer.

[0052] In step 2, the tokenizer of the pre-trained language model is used to tokenize the input text, mapping each character to a corresponding token. In the embedding layer, the input sequence token is mapped to a word embedding of hidden size, and then added to the position embedding and segment embedding to obtain the input sequence of the encoder;

[0053] Step 3, feature extraction of the error correction sentence by a pre-trained language model;

[0054] In step 3, MacBERT is selected as the pre-trained language model for text feature extraction, which adopts a similar Transformer encoder structure composed of a series of identical modules, each containing a multi-head self-attention operation followed by a feed-forward network. After this series of operations, the input text features can be obtained.

[0055] Step 4, as shown in Figure 2 b, the features output by the pre-trained language model are output through a linear error prediction layer to predict the position of the error character, and a linear error correction layer is used to correct the error character;

[0056] Step 4-1, a simple linear prediction layer is used to implement the detection network. Given the output H = [h1, … h n ] of the pre-trained language model and the output error correction sequence G = [g1, … g n ], for each character c i in the text sequence S, there is a corresponding error probability p i :

[0057] p i = P(g i = 1 | X) = Sigmoid(Wh i +b) (1)

[0058] Where W and b are the parameters of the detection network. After the error detection network, it predicts whether each character has an error or not;

[0059] Step 4-2, error correction requires predicting the error token as the correct token, which is the same as the correction task of the pre-training task "masked language model" in MacBERT. Therefore, the same classification head as the pre-training task is used for error character correction. In this module, the output H = [h1, … h nThe input is sequentially passed through a linear projection layer, a GELU activation function, and a LayerNorm layer. Then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S. 1：N′ = [c′1,…,c′ N ].

[0060] Step 5: Train the error correction network based on the error detection loss function and the error correction loss function, and use a padding strategy during training to improve the error correction accuracy of the model;

[0061] Step 5-1, as follows Figure 2 As shown in Figure a, due to the limitation of the length of the input sentence sequence, the pre-trained model cannot flexibly map the tokens to different positions. A padding strategy is adopted to address the differences in index correspondence. Special tokens from the tokenizer are used to pad the input sentence, enabling the error correction network to map the corrected tokens to different positions within the padded sentence.

[0062] Step 5-2: Fine-tuning based on the provided pre-trained MacBERT model. The fine-tuning dataset consists of the original text X, the corrected text Y, and the error index G, represented as {(X1, Y1, G1)...(X...Y1)}. N Y N G N )}.

[0063]

[0064]

[0065] like Figure 2 As shown in b, the loss function of the error correction module consists of error detection loss and error correction loss. It is the loss function of the error detection network. It is the loss function of the error correction network. ω is the loss function of the error correction module, while ω is the weight of the error detection network loss.

[0066]

[0067] Step 6: Improve the lip reading results based on the trained lip reading error correction network.

[0068] Step 6-1, the lip-reading model uses a series of video frames F 1：t = [f1, ..., f T ] as input, where x i ∈R H ×W×1 This represents a preprocessed grayscale image frame. It generates a character sequence S. 1：N = [c1, ..., cN ] as a transcription, where c i ∈ D v , D v is a dictionary of size v.

[0069] Step 6-1, input the original transcription S 1：N as a correction model, generate a corrected version S' 1：N = [c'1,..., c' N ].

[0070] Embodiment 2

[0071] The embodiment relates to a Chinese lip speech recognition improvement device based on a pre-trained language model, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the Chinese lip speech recognition improvement method based on the pre-trained language model in the embodiment 1.

[0072] Embodiment 3

[0073] The embodiment relates to a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the Chinese lip speech recognition improvement method based on the pre-trained language model in the embodiment 1.

[0074] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A pre-trained language model-based Chinese lip speech recognition improvement method, comprising the following steps: Step 1, constructing a correction data set, expanding the data set size through the data enhancement method in step 1; Step 2, preprocessing the correction data set, mapping characters to corresponding tokens through a word segmentation model, and then mapping to word embeddings through an embedding layer; Step 3, extracting features from the corrected sentence through a pre-trained language model to obtain input text features; MacBERT is selected as the pre-trained language model for text feature extraction, which adopts a Transformer encoder structure composed of multiple identical modules, each module containing a multi-head self-attention operation followed by a feedforward network; Step 4, the features output by the pre-trained language model output the positions of the error characters through a linear error prediction layer, and the linear error correction layer corrects the error characters; Step 5, training the error correction network based on the error detection loss function and the error correction loss function, and using a padding strategy to improve the error correction accuracy of the model during training; specifically including: Step 5-1, due to the limitation of the sequence length of the input sentence, the pre-trained model cannot flexibly map tokens to different positions, so a padding strategy is adopted to solve the difference in index correspondence; special tokens from the tokenizer are used to fill the input sentence, so that the error correction network can map the corrected tokens to different positions in the filled sentence; Step 5-2, fine-tuning based on the provided pre-trained MacBERT model; the fine-tuning dataset consists of original text X, corrected text Y and error index G, denoted as {(X1, Y1, G1)…(X N ,Y N ,G N )}; The loss function of the error correction module is composed of an error detection loss and an error correction loss; is the loss function of the error detection network; is the loss function of the error correction network; is the loss function of the error correction module, and ω is the weight of the error detection network loss: Step 6, improving the lip speech recognition result based on the trained lip speech error correction network.

2. The method of claim 1, wherein the pre-trained language model is a Chinese BERT model. The step 1 includes: Step 1-1, build a correction text set Y based on the label and news release data set of CMLR data set, the result of lip reading constitutes the original text X, generate error index G according to X, Y, {(X1, Y1, G1)…(X N ,Y N ,G N )} constitutes a correction data set; Step 1-2, statistical analysis to create a confusion matrix using CMLR results; this matrix allows to map X i → Y i Generate input X from a corpus of labeled text Y; augment the error correction dataset with confusion matrix data augmentation; Step 1-3, using an existing homonym tool to replace the original corrected text Y with homonyms; using homonym data augmentation to expand the error correction data set; Step 1-4, the recognition results of the CMLR test set are used as the test set of the error correction data set, and the remaining data is randomly divided into a training set and a validation set.

3. The method of claim 1, wherein the pre-trained language model is a Chinese BERT model. In step 2, the tokenizer of the pre-trained language model is used to segment the input text, and each character is mapped to a corresponding token; in the embedding layer, the input sequence token is mapped to a hidden size word embedding, and then added with a position embedding and a segment embedding to obtain the input sequence of the encoder.

4. The method of claim 1, wherein the pre-trained language model is a Chinese BERT model. The step 4 includes: Step 4-1, implement the detection network using a simple linear prediction layer; given the output of a pre-trained language model H = [h1, … h n ] and the error-corrected sequence of outputs G = [g1, … g n ], for each character c i in the text sequence S, there is a corresponding error probability p i : p i = P(g i = 1 | X) = Sigmoid(Wh i + b) (1) Where W and b are the parameters of the detection network; after passing through the error detection network, it is predicted whether each character has an error; Step 4-2, error correction needs to correct the wrong label to the correct label, which is the same as the correction task of the pre-training task "masked language model" in MacBERT; therefore, the same classification head as the pre-training task is used for error character correction; the output H = [h1,…h n ] of the pre-trained language model sequentially passes through a linear projection layer, a GELU activation function and a LayerNorm layer; then, in the final linear layer, the input is projected from the hidden features to the vocabulary size to obtain the final correct character sequence S 1∶N′ = [c'1,…,c' N ].

5. The method of claim 1, wherein the pre-trained language model is a Chinese BERT model. The step 6 includes: Step 6-1, the lip-reading model uses a series of video frames F 1:t =[f1,…,f T ] as input, where x i ∈R H×W×1 This represents a preprocessed grayscale image frame; it generates a character sequence S. 1:n =[c1,…,c N As transcription, where c i ∈D v D v It is a dictionary of size v; Step 6-2, the original transcript S 1:N As a correction model input, generate a corrected version S' 1:N = [c'1,..., C' N ].

6. The device for improving Chinese lip speech recognition based on a pre-trained language model, characterized in that, A memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the pre-trained language model-based Chinese lip speech recognition improvement method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, A program is stored thereon, which is executed by a processor to implement the pre-trained language model-based Chinese lip speech recognition improvement method of any one of claims 1-5.

Images