Lip speech recognition method and system based on subspace sparse attention mechanism and medium

Abstract

The present application relates to a kind of lip reading method based on subspace sparse attention mechanism, system and medium, method includes: obtaining lip region image sequence, based on the lip region image sequence extraction obtains lip feature sequence;The lip feature sequence is input to the preset training complete phoneme sequence extraction model, obtains the pronunciation phoneme sequence corresponding to the lip feature sequence;The pronunciation phoneme sequence is input to the sentence inference model with the subspace sparse self-attention mechanism built in and obtains target sentence sequence.The present application enhances the context information by constructing a special attention mechanism, realizes the prediction long sentence sequence in a forward operation, so that the inference rate and accuracy are greatly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, and in particular to a lip-reading method, system, and computer-readable storage medium based on a subspace sparse attention mechanism. Background Technology

[0002] Lip reading (also known as lip-reading) is the process of interpreting and analyzing a speaker's lip movements to derive their spoken content. It aims to supplement auditory information with visual information, enabling accurate comprehension even in cases of hearing impairment. The process typically involves inputting silent video and outputting either speech or text. Lip reading technology plays a crucial role in assisting the deaf and mute, providing daily living services, and public security.

[0003] The development of deep learning has laid a solid foundation for lip-reading technology. However, as the length of input videos increases, deep models become difficult to train, often requiring specific feature learning methods. The inference process is computationally complex and time-consuming, and also involves the optimization of a large number of parameters in the model. Therefore, with the surge in demand for lip-reading and the increasing length of recognition videos, improving the recognition speed and accuracy has become a critical task for lip-reading. Summary of the Invention

[0004] In view of this, it is necessary to provide a lip reading method, system and medium based on subspace sparse attention mechanism to solve the problem that the training process is difficult to converge quickly due to the excessive length of sentence sequences.

[0005] On one hand, this invention provides a lip-reading recognition method based on a subspace sparse attention mechanism, comprising:

[0006] Obtain a sequence of lip region images, and extract a sequence of lip features based on the sequence of lip region images;

[0007] The lip feature sequence is input into a pre-set, well-trained phoneme sequence extraction model to obtain a phoneme sequence corresponding to the lip feature sequence.

[0008] The phoneme sequence is input into a sentence reasoning model with a subspace sparse self-attention mechanism to obtain the target sentence sequence.

[0009] In some possible implementations, determining the fully trained phoneme sequence extraction model includes:

[0010] Initialize the LSTM model, use the lip feature sequence corresponding to the lip region image sample as the training sample, input the training sample into the LSTM model, and obtain the prediction result of the phoneme sequence.

[0011] Based on the training samples and the prediction results, the value of the LSTM model loss function is obtained;

[0012] Based on the value of the loss function of the LSTM model, the fully trained phoneme sequence extraction model is obtained.

[0013] In some possible implementations, the sentence reasoning model with a subspace sparse self-attention mechanism includes a sentence reasoning network module, a language model determination module, and a reasoning sentence sequence module; the step of inputting the phoneme sequence into the sentence reasoning model with the subspace sparse self-attention mechanism to obtain the target sentence sequence includes:

[0014] The phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transition sentence subsequences.

[0015] All the transition sentence subsequences are input into the language model determination module, and the perplexity value of all the transition sentence subsequences is calculated based on the perplexity.

[0016] The transition sentence subsequence with the smallest perplexity value is selected based on the inference sentence sequence module to obtain the predicted target sentence sequence.

[0017] In some possible implementations, the sentence reasoning network module includes multiple sentence subsequence reasoning submodules, each of which includes a masked multi-head self-attention mechanism module and a feedforward network module; the phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transitional sentence subsequences, including:

[0018] The masked multi-head self-attention mechanism module and the feedforward network module in each sentence subsequence inference submodule convert the vectors corresponding to the phoneme sequences into the corresponding sentence subsequences.

[0019] In some possible implementations, the masked multi-head self-attention mechanism module and the feedforward network module in the inference sub-module of each sentence subsequence transform the vector corresponding to the phoneme sequence into the corresponding sentence subsequence, including:

[0020] The phoneme sequence is processed by a masked multi-head self-attention mechanism module to obtain the first vector;

[0021] The second vector is obtained by multiplying the first vector and the vector corresponding to the pronunciation factor sequence;

[0022] The second vector layer is normalized and input into the feedforward neural network module for dimensionality reduction to obtain the third vector;

[0023] Multiplying the third vector and the second vector and then performing layer normalization yields the complete sequence of transition sentences.

[0024] In some possible implementations, the module execution process of the masked multi-head self-attention mechanism module includes:

[0025] The input vector is transformed linearly to obtain three matrices: query Q, key K, and value V.

[0026] The three matrices are convolved to obtain the corresponding word vectors;

[0027] Perform a dimensionality reduction operation on the word vector of the key K and the word vector of the value V;

[0028] The word vectors of the query Q, the key K, and the value V are calculated to obtain the output vector.

[0029] In some possible implementations, all the transition sentence subsequences are input into the language model decision module, including:

[0030] The word corresponding to the sentence subsequence is determined based on whether the phoneme sequence corresponding to the sentence subsequence includes a subset:

[0031] If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches a word, then output the word;

[0032] If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches multiple words, then output the word with the largest expected value.

[0033] If the phoneme sequence corresponding to the sentence subsequence contains multiple subsets, then the perplexity value of the sentence subsequence is calculated based on the perplexity.

[0034] In some possible implementations, the perplexity value of the sentence subsequence is calculated based on the perplexity, including:

[0035] The words corresponding to the first subset and the words corresponding to the second subset of the sentence subsequence are matched to obtain a new subset, and a first perplexity value is calculated between them based on the perplexity, to obtain the word combination with the lowest preset number of words with the lowest first perplexity value;

[0036] The words corresponding to the new subset are matched with the words corresponding to the next subset to obtain a new subset, and a second perplexity value is calculated between them. The word combination with the lowest preset number of words with the second perplexity value is obtained. This process is repeated until all subsets are matched to obtain all target sentence sequences.

[0037] On the other hand, the present invention also provides a lip reading system based on a subspace sparse attention mechanism, including a microprocessor and a memory interconnected, characterized in that the microprocessor is programmed or configured to execute the steps of the lip reading method based on the subspace sparse attention mechanism described in any of the above implementations.

[0038] On the other hand, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps in the lip-reading method based on the subspace sparse attention mechanism described in any of the above implementations.

[0039] The beneficial effects of the above embodiments are as follows: The lip-reading method based on a subspace sparse attention mechanism provided by this invention first acquires a sequence of lip region images, extracts a lip feature sequence based on the lip region image sequence, then inputs the lip feature sequence into a well-trained phoneme sequence extraction model to obtain a sequence of pronounced phonemes, and finally inputs the pronounced phoneme sequence into a sentence inference model with a subspace sparse self-attention mechanism to obtain the target sentence sequence. This invention utilizes a special attention mechanism to enhance contextual information, enabling the prediction of long sentence sequences in a single forward operation, thereby greatly improving the inference speed and accuracy, and providing technical support for related applications. Attached Figure Description

[0040] Figure 1 A flowchart illustrating an embodiment of the lip-reading method based on a subspace sparse attention mechanism provided by the present invention.

[0041] Figure 2 This is a module structure diagram of an embodiment of the masked multi-head self-attention mechanism module provided by the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0043] It should be understood that the illustrative drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0044] In the description of the embodiments of the present invention, "and / or" describes the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0045] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0046] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0047] Figure 1 This is a schematic flowchart of an embodiment of the lip-reading method based on a subspace sparse attention mechanism provided by the present invention, as shown below. Figure 1 As shown, lip-reading methods based on subspace sparse attention mechanisms include:

[0048] S101. Obtain the lip region image sequence, and extract the lip feature sequence based on the lip region image sequence;

[0049] S102. Input the lip feature sequence into a preset, well-trained phoneme sequence extraction model to obtain a phoneme sequence corresponding to the lip feature sequence.

[0050] S103. Input the phoneme sequence into a sentence reasoning model with a subspace sparse self-attention mechanism to obtain the target sentence sequence.

[0051] Compared with existing technologies, the lip-reading method based on a subspace sparse attention mechanism provided in this invention first acquires a sequence of lip region images, extracts a lip feature sequence from the lip region image sequence, then inputs the lip feature sequence into a well-trained phoneme sequence extraction model to obtain a sequence of pronounced phonemes, and finally inputs the pronounced phoneme sequence into a sentence inference model equipped with a subspace sparse self-attention mechanism to obtain the target sentence sequence. By utilizing a special attention mechanism to enhance contextual information, it enables the prediction of long sentence sequences in a single forward operation, thereby significantly improving inference speed and accuracy, and providing technical support for related applications.

[0052] It is understandable that once the target sentence sequence is obtained, the purpose is to convert the vocal features of the lip region into text, thereby achieving lip reading recognition through text recognition.

[0053] It should be noted that in step S101, obtaining the lip region image sequence and extracting the lip feature sequence based on the lip region image sequence includes:

[0054] The acquired video sequence data is cropped to a preset length and adjusted to a preset frame rate.

[0055] Face detection is performed on each frame of the image in sequence. Then, key point detection is performed on the detected face images. Based on the obtained face key point annotation information, the position of the corner of the mouth in each face image is determined and cropped to obtain the corresponding lip region image. Multiple lip region images constitute a lip region image sequence.

[0056] The offset and rotation factor of the lip region image after being aligned with the preset standard image are calculated to obtain the lip feature vector corresponding to each lip region image. Then, they are concatenated in sequence to obtain the corresponding lip feature sequence.

[0057] It should be noted that Long Short-Term Memory (LSTM) networks are a type of temporal recurrent neural network specifically designed to address the long-term dependency problem inherent in general recurrent neural networks (RNNs). In some embodiments of this invention, determining the fully trained phoneme sequence extraction model includes:

[0058] Initialize the LSTM model, use the lip feature sequence corresponding to the lip region image sample as the training sample, input the training sample into the LSTM model, and obtain the prediction result of the phoneme sequence.

[0059] Based on the training samples and the prediction results, the value of the LSTM model loss function is obtained;

[0060] Based on the value of the loss function of the LSTM model, the fully trained phoneme sequence extraction model is obtained.

[0061] In some embodiments of the present invention, the sentence reasoning model with a subspace sparse self-attention mechanism includes a sentence reasoning network module, a language model determination module, and a reasoning sentence sequence module; the step of inputting the phoneme sequence into the sentence reasoning model with the subspace sparse self-attention mechanism to obtain the target sentence sequence includes:

[0062] The phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transition sentence subsequences.

[0063] All the transition sentence subsequences are input into the language model determination module, and the perplexity value of all the transition sentence subsequences is calculated based on the perplexity.

[0064] The transition sentence subsequence with the smallest perplexity value is selected based on the inference sentence sequence module to obtain the predicted target sentence sequence.

[0065] It should be noted that, in long sentence reasoning, to achieve better reasoning results, in some embodiments of the present invention, the sentence reasoning network module includes multiple sentence subsequence reasoning submodules. Each sentence subsequence reasoning submodule includes a masked multi-head self-attention mechanism module and a feedforward network module. The phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transitional sentence subsequences, including:

[0066] The masked multi-head self-attention mechanism module and the feedforward network module in each sentence subsequence inference submodule convert the vectors corresponding to the phoneme sequences into the corresponding sentence subsequences.

[0067] It should be noted that the masked multi-head self-attention mechanism module includes multiple head self-attention blocks. The masked multi-head self-attention mechanism utilizes a mask layer to hide future information, and the feedforward neural network is implemented by two one-dimensional convolutions and layer normalization. In some embodiments of the present invention, the masked multi-head self-attention mechanism module and the feedforward network module in each sentence subsequence inference submodule convert the vector corresponding to the phoneme sequence into the corresponding sentence subsequence, including:

[0068] The phoneme sequence is processed by a masked multi-head self-attention mechanism module to obtain the first vector;

[0069] The second vector is obtained by multiplying the first vector and the vector corresponding to the pronunciation factor sequence;

[0070] The second vector layer is normalized and input into the feedforward neural network module for dimensionality reduction to obtain the third vector;

[0071] Multiplying the third vector and the second vector and then performing layer normalization yields the complete sequence of transition sentences.

[0072] In some embodiments of the present invention, the module execution process of the masked multi-head self-attention mechanism module includes:

[0073] The input vector is transformed linearly to obtain three matrices: query Q, key K, and value V.

[0074] The three matrices are convolved to obtain the corresponding word vectors;

[0075] Perform a dimensionality reduction operation on the word vector of the key K and the word vector of the value V;

[0076] The word vectors of the query Q, the key K, and the value V are calculated to obtain the output vector.

[0077] In a specific embodiment of the present invention Figure 2 This is a module structure diagram of an embodiment of the masked multi-head self-attention mechanism module provided by the present invention. The calculation process in each head self-attention block is as follows:

[0078] Step 1: Input vector X∈R C×H×W Where C represents the number of channels, and H and W represent the height and width in space, respectively, let Q, K, and V be the one-dimensional convolution of the three matrices. W θ W γ Different embedding representations are directly generated through convolution. in For the dimension of the embedded representation, the generation process formula is as follows:

[0079]

[0080] Step 2: The three embedded representations obtained θ and γ fold into Size, where N = H·W;

[0081] Step 3: Perform dimensionality reduction on the obtained embedding representations θ and γ, reducing their dimension from N to S. The specific steps for dimensionality reduction are as follows:

[0082] To reduce the dimension from N to S, representative points need to be sampled from θ and γ, instead of all input points. This reduces the output size to half its original size, as shown in the following formula:

[0083] X S =Maxpool(ELU(Conv1d[X N )),

[0084] Conv1d(·) is a 1D convolution, ELU(·) is an activation function, Maxpool(·) is a max pooling layer with a stride of 2, and several key features are selected to further reduce the dimensionality and extract the priority features that play a major role.

[0085] Step 4: Embedded representation The steps include calculating the dimensionality-reduced embedding representations θ and γ to obtain the final output representation vector O.

[0086] First, it is represented by embedding. And θ generate the attention matrix V∈R N×N Its calculation formula is After normalization, we get Next, the attention matrix V is multiplied by the embedding representation γ to obtain the final output representation vector O, i.e.

[0087] The calculation process reveals that when the dimension S of the dimensionality reduction of the embedding representations θ and γ is set to be much smaller than... When the dimension is N, i.e., S << N, the output dimension remains unchanged in the sparse attention matrix. The internal process can be explained by the following formula:

[0088]

[0089] It should be noted that perplexity is used to evaluate the quality of a language model; perplexity is the value obtained by performing an exponential operation on the cross-entropy loss function. In some embodiments of the present invention, all the transition sentence subsequences are input into the language model determination module, including:

[0090] The word corresponding to the sentence subsequence is determined based on whether the phoneme sequence corresponding to the sentence subsequence includes a subset:

[0091] If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches a word, then output the word;

[0092] If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches multiple words, then output the word with the largest expected value.

[0093] If the phoneme sequence corresponding to the sentence subsequence contains multiple subsets, then the perplexity value of the sentence subsequence is calculated based on the perplexity.

[0094] In a specific embodiment of the present invention, if the phoneme sequence corresponding to the sentence subsequence contains multiple subsets, in the first iteration, the words matching the first two sets will be combined in possible ways based on the following rules;

[0095] 1) The 50 combinations with the lowest confusion values ​​are retained;

[0096] 2) These combinations can be matched with word combinations of the next phoneme subset;

[0097] 3) Keep the 50 combinations with the lowest confusion values ​​and continue iterating over the rest of the sequence until the end of the sequence.

[0098] In some embodiments of the present invention, calculating the perplexity value of the sentence subsequence based on perplexity includes:

[0099] The words corresponding to the first subset and the words corresponding to the second subset of the sentence subsequence are matched to obtain a new subset, and a first perplexity value is calculated between them based on the perplexity, to obtain the word combination with the lowest preset number of words with the lowest first perplexity value;

[0100] The words corresponding to the new subset are matched with the words corresponding to the next subset to obtain a new subset, and a second perplexity value is calculated between them. The word combination with the lowest preset number of words with the second perplexity value is obtained. This process is repeated until all subsets are matched to obtain all target sentence sequences.

[0101] In a specific embodiment of the present invention, the steps for calculating the perplexity value are as follows:

[0102] First, the probabilistic relationship between the phoneme sequence and its corresponding sentence is obtained. Then, the combination of words with the highest probability is selected to derive the inference sentence. The probabilistic relationship between the phoneme sequence and its corresponding sentence is as follows:

[0103]

[0104]

[0105] In the formula: where P is the phoneme sequence, p i For the i-th subset, W C Represents any given combination of words, w i This corresponds to each matched word in the word string.

[0106] The process of selecting the word combination with the highest probability to obtain the reasoning sentence is as follows:

[0107]

[0108]

[0109] It is the combination with the greatest probability, which takes into account a subset of phonemes of each combination C, where C belongs to the combination set C. * .

[0110] The information entropy of each word and P(w1,w2,…,w) N Related to (w1, w2, ..., w) N The possible word sequences, i.e., transition sentence subsequences, belong to the word set W. Specifically as follows:

[0111]

[0112]

[0113] In the formula, N represents the number of words, and PP represents the perplexity value.

[0114] Accordingly, this application also provides a lip-reading recognition system based on a subspace sparse attention mechanism, including a processor and a memory interconnected, wherein the processor is programmed or configured to perform the steps or functions in the lip-reading recognition method based on a subspace sparse attention mechanism provided in the above-described method embodiments.

[0115] In summary, the lip-reading method, system, and computer-readable storage medium based on a subspace sparse attention mechanism provided by this invention first acquires a sequence of lip region images, extracts a lip feature sequence from the lip region image sequence, then inputs the lip feature sequence into a well-trained phoneme sequence extraction model to obtain a sequence of pronounced phonemes, and finally inputs the pronounced phoneme sequence into a sentence inference model equipped with a subspace sparse self-attention mechanism to obtain the target sentence sequence. This invention utilizes a special attention mechanism to enhance contextual information, enabling the prediction of long sentence sequences in a single forward operation, thereby significantly improving inference speed and accuracy, and providing technical support for related applications.

[0116] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the lip reading recognition method based on subspace sparse attention mechanism provided in the above-described method embodiments.

[0117] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0118] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A lip-reading recognition method based on a subspace sparse attention mechanism, characterized in that, include: Obtain a sequence of lip region images, and extract a sequence of lip features based on the sequence of lip region images; The lip feature sequence is input into a pre-set, well-trained phoneme sequence extraction model to obtain a phoneme sequence corresponding to the lip feature sequence. The phoneme sequence is input into a sentence reasoning model with a subspace sparse self-attention mechanism to obtain the target sentence sequence; The sentence reasoning model with a subspace sparse self-attention mechanism includes a sentence reasoning network module, a language model determination module, and a reasoning sentence sequence module. The step of inputting the phoneme sequence into a sentence reasoning model equipped with a subspace sparse self-attention mechanism to obtain the target sentence sequence includes: The phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transition sentence subsequences. All the transition sentence subsequences are input into the language model determination module, and the perplexity value of all the transition sentence subsequences is calculated based on the perplexity. Based on the reasoning sentence sequence module, the transition sentence subsequence with the smallest perplexity value is selected to obtain the predicted target sentence sequence; The sentence reasoning network module includes multiple sentence subsequence reasoning submodules. Each sentence subsequence reasoning submodule includes a masked multi-head self-attention mechanism module and a feedforward network module. The phoneme sequence is input into the sentence reasoning network module for sentence reasoning to obtain all transitional sentence subsequences, including: The masked multi-head self-attention mechanism module and the feedforward network module in each sentence subsequence inference submodule convert the vectors corresponding to the phoneme sequences into the corresponding sentence subsequences.

2. The lip-reading recognition method based on subspace sparse attention mechanism according to claim 1, characterized in that, Determining the fully trained phoneme sequence extraction model includes: Initialize the LSTM model, use the lip feature sequence corresponding to the lip region image sample as the training sample, input the training sample into the LSTM model, and obtain the prediction result of the phoneme sequence. Based on the training samples and the prediction results, the value of the LSTM model loss function is obtained; Based on the value of the loss function of the LSTM model, the fully trained phoneme sequence extraction model is obtained.

3. The lip-reading recognition method based on subspace sparse attention mechanism according to claim 1, characterized in that, The masked multi-head self-attention mechanism module and feedforward network module in each sentence subsequence inference submodule convert the vectors corresponding to the phoneme sequences into corresponding sentence subsequences, including: The phoneme sequence is processed by a masked multi-head self-attention mechanism module to obtain the first vector; The second vector is obtained by multiplying the first vector and the vector corresponding to the pronunciation factor sequence; The second vector layer is normalized and input into the feedforward neural network module for dimensionality reduction to obtain the third vector; Multiplying the third vector and the second vector and then performing layer normalization yields the complete sequence of transition sentences.

4. The lip-reading recognition method based on subspace sparse attention mechanism according to claim 3, characterized in that, The module execution process of the masked multi-head self-attention mechanism module includes: The input vector is transformed linearly to obtain three matrices: query Q, key K, and value V. The three matrices are convolved to obtain the corresponding word vectors; Perform a dimensionality reduction operation on the word vector of the key K and the word vector of the value V; The word vectors of the query Q, the key K, and the value V are calculated to obtain the output vector.

5. The lip-reading recognition method based on subspace sparse attention mechanism according to claim 1, characterized in that, The entire sequence of transition sentences is input into the language model decision module, including: The word corresponding to the sentence subsequence is determined based on whether the phoneme sequence corresponding to the sentence subsequence includes a subset: If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches a word, then output the word; If the phoneme sequence corresponding to the sentence subsequence contains only one subset, and the subset matches multiple words, then output the word with the largest expected value. If the phoneme sequence corresponding to the sentence subsequence contains multiple subsets, then the perplexity value of the sentence subsequence is calculated based on the perplexity.

6. The lip-reading recognition method based on subspace sparse attention mechanism according to claim 1, characterized in that, Calculating the perplexity value of the sentence subsequence based on perplexity includes: The words corresponding to the first subset and the words corresponding to the second subset of the sentence subsequence are matched to obtain a new subset, and a first perplexity value is calculated between them based on the perplexity, to obtain the word combination with the lowest preset number of words with the lowest first perplexity value; The words corresponding to the new subset are matched with the words corresponding to the next subset to obtain a new subset, and a second perplexity value is calculated between them. The word combination with the lowest preset number of words with the second perplexity value is obtained. This process is repeated until all subsets are matched to obtain all target sentence sequences.

7. A lip-reading recognition system based on a subspace sparse attention mechanism, comprising interconnected processors and memory, characterized in that, The processor is programmed or configured to perform the steps of the lip reading method based on the subspace sparse attention mechanism as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the lip-reading method based on subspace sparse attention mechanism as described in any one of claims 1 to 6.

Images