Mongolian lip language recognition method based on multi-modal training

Abstract

The invention provides a Mongolian lip language recognition method based on multi-modal training. The Mongolian lip language recognition method comprises the steps of S1, a preprocessing stage, S2, a training stage and S3, a reasoning stage. The method comprises the following steps: S1, a preprocessing stage: segmenting words in a language model training text into forms of stems, lattice suffixes and verbs except verbs; s2, a training stage: firstly, performing feature extraction on audio and video by using ResNet to obtain feature representation of each mode; the feature representation is then input into a Conformer encoder, and a multi-modal shared representation is extracted through deep fusion; s3, a reasoning stage: firstly, performing fine feature extraction on a video image through ResNet, and then inputting the extracted video features into a Conformer encoder for deep processing; and finally, accurately predicting the text content by using a forward S4D decoder. The audio information is fused in the training process, so that the system can capture more voice features from a multi-mode angle, and the fuzzy and uncertainty possibly caused by only depending on the video information are made up.

Description

Technical Field

[0001] This application relates to the field of lip-reading recognition technology, and more particularly, to a Mongolian lip-reading recognition method based on multi-modal training. Background Art

[0002] Lip-reading recognition (Visual Speech Recognition, VSR) is a technology that recognizes speech content through the lip movement trajectories in videos. In recent years, with the rapid development of deep learning and computer vision technologies, VSR technology has been significantly improved and has received extensive attention especially in application scenarios such as speech recognition in noisy environments and hearing-impaired assistive devices.

[0003] Currently, deep learning-based lip-reading recognition models are the mainstream research methods. ResNet (Residual Network) is widely used to extract lip features in video frames. Compared with traditional convolutional neural networks, ResNet solves the problem of gradient disappearance during the training of deep networks by introducing residual connections, enabling the network to capture subtle visual information more deeply, especially performing well in complex lip movements. Through residual blocks, ResNet can retain more global information while extracting local lip features, providing high-quality feature representations for subsequent time series modeling. At the same time, Long Short-Term Memory (LSTM) or Recurrent Neural Network (RNN) is used to capture the time series dynamic information in videos and can analyze the continuous changes in lip movements. In recent years, the introduction of Transformer and Conformer models has further enhanced the performance of VSR in long-sentence-level tasks. Transformer captures long-range dependencies through self-attention mechanisms, while Conformer combines convolutional and self-attention mechanisms, being able to handle local dynamics and capture global dependencies, making the model perform even better in long-sentence lip-reading recognition. The framework diagram of existing lip-reading recognition systems is as Figure 1 shown.

[0004] However, most existing research mainly focuses on high-resource languages such as English and Chinese, while there is less research on lip-reading recognition of low-resource languages such as Mongolian, which also brings challenges and opportunities to this field. There are still the following problems in the implementation of Mongolian lip-reading recognition:

[0005] Problem 1: Long-sequence texts contain a large amount of context information and complex semantic structures. Traditional encoders and decoders often struggle to fully capture the full picture of this information during processing. Due to the relatively distant information dependencies in long sequences, the model is prone to overlooking key features or misinterpreting the actual meaning of the context, resulting in inaccurate semantic expressions and significant deviations in recognition results. Especially when dealing with a hierarchical language like Mongolian, the subtle differences in long texts are crucial for semantic understanding. However, existing models show obvious deficiencies in the semantic encoding and decoding of long-sequence texts and are unable to fully explore their potential semantic relationships, thus affecting the overall recognition effect.

[0006] Problem 2: When lip reading recognition solely relies on video information, the recognition accuracy is likely to decline due to the subtle changes in the speaker's lip movements or environmental factors. Mongolian has demonstrated excellent results in speech recognition. Therefore, during the lip reading recognition process, the introduction of audio information can effectively assist the model in learning the speech features in the video and improving the overall recognition performance. However, the process of fusing audio and video information poses great challenges. Audio and video modalities have different time scales and feature distributions. How to maintain the synchronization and information complementarity between the two during the fusion process is an issue to be solved. If not handled properly, the fusion process may introduce noise or inter-modal inconsistencies, instead reducing the performance of the model. Therefore, designing an effective multi-modal fusion mechanism that enables audio information to support the video modality without overly interfering with the original features is the key to improving the recognition accuracy.

[0007] In summary, the recognition accuracy of existing Mongolian lip reading recognition methods is relatively low. Summary of the Invention

[0008] In view of this, the present application provides a Mongolian lip reading recognition method based on multi-modal training to overcome the deficiencies of the prior art in the lip reading recognition of Mongolian long sentences.

[0009] To achieve the above object, the technical solution adopted in the present application is as follows: The Mongolian lip reading recognition method based on multi-modal training includes S1: preprocessing stage, S2: training stage, and S3: inference stage; S1: Preprocessing stage. Cut the words other than verbs in the language model training text into the form of stems and case suffixes, and at the same time establish a pronunciation dictionary based on stems, case suffixes, and verbs; accurately extract the lip region in the video image through a face detection algorithm and a lip localization algorithm, and crop out the lip images in each frame of the video; S2: In the training phase, first, ResNet is used to extract features from audio and video to obtain the feature representations of each modality; the feature representations are then input into the Conformer encoder, and through deep fusion, a multi-modal shared representation is refined; then, the multi-modal shared representation is respectively input into the forward and backward S4D decoders, where the forward and backward S4D decoders are respectively used to predict the sequential sequence and the reverse sequence of the text. S3: In the inference phase, first, ResNet is used to perform fine-grained feature extraction on video images, and the extracted video features are then input into the Conformer encoder for deep processing; finally, the forward S4D (Structured StateSpace Decoder) is used to accurately predict the text content, thereby effectively capturing the semantic information in the video and generating the corresponding text output.

[0010] Furthermore, the S4D decoder in the training phase and the inference phase includes three modules, namely the S4D module, the multi-head attention layer (Multi-Head Attention Block), and the feed-forward layer (Feed Forward Block), and the S4D decoder performs the following steps: Step 1, the decoder first receives the output features of the encoder and sends them into the S4D module, where the S4D module performs sequence modeling, effectively capturing the temporal information and the structured dependencies between features, and improving the temporal performance of decoding. Step 2, the features from the S4D module are then input into the multi-head attention layer, where the multi-head attention layer processes the input features in parallel through multiple attention heads to capture the global context information and ensure that the model can focus on the dependencies between different time steps. Step 3, the features from the multi-head attention layer are then input into the feed-forward layer, where at this stage, the features undergo non-linear transformation to enhance the complexity and expressive ability of the feature representation.

[0011] Furthermore, the recurrence formula of the S4D module is: where is the hidden state, is the input feature, and the matrices and represent the state transition matrix and the input weight matrix respectively; The core formula of the multi-head attention layer is: where Q, K, and V represent the query, key, and value respectively, is the dimension of the key, denotes the transpose of a matrix; The core formula of the feed-forward layer is: where and are learnable weight matrices, and are bias terms.

[0012] Furthermore, the ResNet is ResNet-18, which is a residual network, and the specific formula is: where is the input feature, represents the residual function composed of convolutional layers, { } are the convolutional kernel parameters, is the non-linear activation function.

[0013] Furthermore, after using the ResNet to extract features from the audio and video in S2, it also includes: Project the extracted visual and audio features through a linear projection layer into a unified feature space, denoted as and respectively, that is: where and are linear projection matrices, and are bias terms, and represent the video feature and the audio feature respectively.

[0014] Furthermore, each layer of the Conformer encoder includes four modules, namely: the first Feed-Forward module, the Self-Attention module, the Convolution module, and the second Feed-Forward module. Assuming the input feature is H, the processing formulas after passing through each module are as follows: Among them, FFN represents the feed-forward network, MHSA represents the multi-head self-attention mechanism, and Conv represents the convolutional module; after being processed by multiple layers of Conformer, the fused features are output , where is the feature representation of the -th data, and

[0015] is the set of feature representations of all data. Further, in the S2: training stage, the shared representations of the multi-modal are respectively input into the forward and backward S4D decoders, and the forward and backward S4D decoders are respectively used to predict the sequential sequence and the reverse sequence of the text, specifically including: Among them, represents the conditional probability of selecting at the current time step given the previous decoding output and the encoder input ; (2) Use the beam search algorithm to increase the possibility of finding the global optimal solution; during the beam search process, the accumulated log probability is used to evaluate the score of the candidate sequence: Among them, is the generated candidate sequence.

[0016] Compared with the prior art, the beneficial effects of this application are: 1. By integrating audio information into the training process, the system can capture more speech features from a multi-modal perspective, making up for the possible ambiguity and uncertainty relying solely on video information; 2. The temporal information in lip reading is very important, while the unidirectional decoder can only capture forward dependencies and may ignore backward semantic information. By using a bidirectional decoder, the model can simultaneously utilize forward and backward temporal features, fully capture context information, and improve the recognition accuracy;

[0017] 3. The S4D block uses the state space model and can better capture temporal structure information and long-term dependencies. Compared with the traditional self-attention mechanism, the S4D block has more advantages in temporal tasks because it can more effectively propagate and integrate information across the entire sequence. Therefore, the effect of long text lip reading is improved. Description of the Drawings

[0018] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0019] Figure 1 is a framework diagram of a lip-reading recognition system in the prior art; Figure 2 is a flowchart of a Mongolian lip-reading recognition method based on multi-modal training of the present application; Figure 3 is a training flowchart of the lip-reading recognition system of the present application; Figure 4 is an inference flowchart of the lip-reading recognition system of the present application; Figure 5 is a decoder framework diagram of a lip-reading recognition system in the prior art; Figure 6 is a decoder framework diagram of the S4D of the lip-reading recognition system of the present application; Figure 7 is a schematic diagram of the Mongolian word splicing and word formation method of the present application; Figure 8 is an example diagram of a whole-word pronunciation dictionary of the present application. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application.

[0021] Current methods mainly focus on the recognition of short sentences or words in English or Chinese, and certain results have been achieved. However, when dealing with Mongolian and complex long sentences, the recognition rate is relatively low and the error rate is relatively high. Therefore, the present application proposes a model that combines a multi-modal Conformer network and a structured state space decoder (S4D), which significantly improves the accuracy and efficiency of Mongolian lip-reading recognition, especially in the recognition of long sentences.

[0022] As Figure 2 shown, the present application provides a Mongolian lip-reading recognition method based on multi-modal training, including S1: a preprocessing stage, S2: a training stage, and S3: an inference stage; S1: In the preprocessing stage, words other than verbs in the language model training text are segmented into stems and case suffixes, and a pronunciation dictionary based on stems, case suffixes, and verbs is established; the lip region in the video image is accurately extracted through a face detection algorithm and a lip localization algorithm, and the lip images in each frame of the video are cropped out. Data preprocessing: 1. Text content processing: Mongolian, as a typical agglutinative language, usually forms words through the combination of roots and affixes. In the text processing stage, first, the traditional writing system of Mongolian is converted into Latin alphabet representation to standardize and simplify data input. This Latinization process can provide a consistent coding basis for subsequent processing. On this basis, a pronunciation dictionary is constructed to accurately map the pronunciation features of each word, including the combination rules of vowels and consonants, stress positions, and liaison characteristics. By constructing the dictionary, not only can the high consistency between text content and speech be ensured, but also a foundation for the subsequent coding process can be laid. Subsequently, the text content is encoded word by word or symbol by symbol according to the pronunciation dictionary to ensure that the pronunciation corresponding to each symbol can be accurately captured by the model. The encoding process standardizes the Mongolian speech data by converting the phonemes in the dictionary into numerical vectors.

[0023] 2. Video content processing: When processing video information related to speech, lip features are crucial. First, a face detection algorithm is used to detect the face position in the video, and then a lip localization algorithm or key point detection algorithm is further used to accurately extract the lip region, and the extracted lip region is preprocessed. The lip video is converted into a grayscale image to remove the interference of color information and retain the accurate information of the lip contour and dynamic changes, which helps the model focus on the shape changes related to pronunciation. In addition, to enhance the robustness and generalization ability of the model for different parts in the time series, a time masking technique is randomly introduced, that is, randomly masking some frame sequences of the lip video. Through the masking process of a specific time period, the model can learn the ability to effectively decode speech features even under incomplete time information, thereby improving its adaptability and robustness in the case of time loss.

[0024] 0. Audio content processing: In the processing of audio data, the time masking technique is also applied. This method breaks the continuity of the audio sequence by randomly masking certain segments of the audio, enabling the model to learn the tolerance for locally lost information during training, thereby improving the decoding robustness for audio content. Through time masking, the model's recovery ability in the face of incomplete speech input can be enhanced. Especially in multimodal fusion tasks, the processing of audio masking can effectively simulate the noise or missing situations in the actual scenario, further improving the system's adaptability to the real scenario.

[0025] 4. Construction of Whole-Word Pronunciation Dictionary: As Figure 8 shown, the construction of the whole-word pronunciation dictionary in this application includes words, word-formation forms, and pronunciations.

[0026] Mongolian is a typical agglutinative language. Words are formed by splicing roots with various affixes, as Figure 7 shown. Affixes are divided into three categories: word-formation affixes, inflectional suffixes, and ending suffixes. Among them, word-formation affixes and inflectional suffixes are responsible for changing the meaning of words, while ending suffixes mainly undertake grammatical functions and are always located at the end of words. Word-formation affixes and inflectional suffixes are attached to the root respectively to form a stem. Once the stem is combined with the ending suffix, a complete Mongolian word can be generated. Ending suffixes include case suffixes, possessive suffixes, gerund suffixes (marking time or person), and adverbial participle suffixes. For the participle suffix, if it serves as the predicate of the main clause, it can be regarded as an ending suffix; if it acts as a static word (especially when combined with the case suffix), it is regarded as a stem suffix. Generally, the order of word-formation affixes, inflectional suffixes, and ending suffixes is fixed: word-formation affixes come first, inflectional suffixes come second, and ending suffixes come last. Multiple word-formation affixes and inflectional suffixes can appear, while there is usually only one ending suffix, unless it encounters the reflexive possessive structure in Mongolian script, where double ending suffixes can appear. This way of splicing roots, word-formation affixes, inflectional suffixes, and ending suffixes becomes the basis for the formation of Mongolian words, and most words are composed of different stems combined with ending suffixes. Thus, in the lip-reading recognition system, the training and recognition of words can be converted into the training and recognition of stems and ending suffixes.

[0027] S2: In the training stage, first use ResNet to extract features from audio and video to obtain the feature representations of each modality; the feature representations are then input into the Conformer encoder, and through deep fusion, a multi-modal shared representation is refined; then the multi-modal shared representation is respectively input into the forward and backward S4D decoders, and the forward and backward S4D decoders are respectively used to predict the sequential sequence and reverse sequence of the text; Specifically, as Figure 3As shown, the processing flow in the training phase of this application is divided into three main stages: feature extraction, encoding, and decoding. Each stage combines visual and audio features to improve the recognition ability at the long-sentence level. The ResNet can be ResNet-18, which is a residual network containing multiple convolutional and residual blocks. First, ResNet-18 is used to extract features from audio and video to obtain the feature representations of each modality. These features are then input into the Conformer encoder, and through deep fusion, a multi-modal shared representation is refined. To make full use of the context information of the text, the model then introduces forward and backward S4D decoders, which are used to predict the sequential and reverse sequences of the text respectively. The forward S4D decoder and the backward S4D decoder have the same structure, and the only difference is whether the input sequence is forward or backward. Through this bidirectional prediction mechanism, the model can not only capture the forward and backward dependencies of the text, but also maximize the synergy between multi-modal features, improving the understanding ability and generalization performance of the target task.

[0028] S3: In the inference phase, first, the ResNet is used to perform fine-grained feature extraction on the video image. The extracted video features are then input into the Conformer encoder for deep processing. Finally, the forward S4D decoder is used to accurately predict the text content, so as to effectively capture the semantic information in the video and generate the corresponding text output.

[0029] As Figure 4 shown, in the inference phase, the system first performs fine-grained feature extraction on the video through ResNet-18. These video features are then input into the Conformer encoder for deep processing. Finally, the model uses the forward S4D decoder to accurately predict the text content, so as to effectively capture the semantic information in the video and generate the corresponding text output.

[0030] During the inference process, the model only uses Mongolian video content and combines the Conformer module and the forward decoder for lip-reading recognition. First, the input video content extracts frame-level features through 3D ResNet-18 to generate a serialized video feature representation. Subsequently, these features are input into the Conformer module, where the convolutional network is used to capture local temporal information, and at the same time, the self-attention mechanism is used to model the global context relationship to better extract the semantic features in the video. Then, the forward decoder gradually predicts each character of the Mongolian sequence based on the high-dimensional feature sequence output by the Conformer, ensuring the temporal consistency of the decoding process.

[0031] Furthermore, as Figure 6As shown, the S4D decoder in the training phase and the inference phase includes three modules, namely the S4D module, the Multi-Head Attention Block, and the Feed Forward Block. The S4D decoder performs the following steps (both the forward and reverse S4D decoders execute according to the following steps): Step 1, the decoder first receives the output features of the encoder and feeds them into the S4D module. The S4D module performs sequence modeling, effectively capturing the temporal information and the structured dependencies between features, and enhancing the temporal performance of decoding. The S4D decoding structure is as Figure 6 shown, including three main modules: the feed forward layer, the multi-head attention layer, and the S4D module. These modules together form a basic unit, and this basic unit is repeated N times to achieve the layer-by-layer processing and decoding of the input encoded features.

[0032] As an important part of the S4D decoder, the S4D module captures the dynamic sequence information in the features by diagonalizing the structured state space model. The recurrence formula of the S4D module is:

[0033] where, is the hidden state, is the input feature, and the matrices and represent the state transition matrix and the input weight matrix. Through diagonalization, the S4D module enhances the dynamic modeling ability of the model while maintaining computational efficiency.

[0034] Step 2, the features output by the S4D module are input into the multi-head attention layer. The multi-head attention layer processes the input features in parallel through multiple attention heads, capturing the global context information to ensure that the model can focus on the dependencies between different time steps; The multi-head attention layer is based on the self-attention mechanism and can effectively capture the long-range dependencies in the input features. Its core formula is as follows:

[0035] where Q, K, and V represent the query, key, and value respectively, is the dimension of the key. Through the multi-head attention mechanism, the decoder can extract information from multiple representation spaces simultaneously, enriching the feature representation. The representation space is the high-dimensional feature space generated by the model for the input data. In this space, the input data is represented as a certain numerical vector, and these vectors capture the important features or semantic information of the input data.

[0036] Step 3, the features passing through the multi-head attention layer are input into the feed-forward layer. At this stage, the features undergo a non-linear transformation to enhance the complexity and expressiveness of the feature representation.

[0037] The feed-forward layer is a fully connected layer, usually consisting of two linear transformations and a non-linear activation function ReLU. The formula is:

[0038] where and are learnable weight matrices, and are bias terms. This module is used to increase the non-linear expressiveness of the network.

[0039] Figure 5 is the decoder framework diagram of the lip-reading recognition system in the prior art, Figure 6 is the decoder framework diagram of the S4D decoder of the lip-reading recognition system in this application; by comparing Figure 5 and Figure 6 it can be seen that the key improvement in this application lies in the S4D module. The S4D block uses a state space model and can better capture temporal structure information and long-term dependencies. Compared with the traditional self-attention mechanism, the S4D block has more advantages in temporal tasks because it can more effectively spread and integrate information across the entire sequence. Therefore, the effect of long-text lip-reading recognition is improved.

[0040] Furthermore, the ResNet is ResNet-18. The ResNet-18 is a residual network, and the specific formula is: where is the input feature, represents the residual function composed of convolutional layers, { } are the convolutional kernel parameters, is the non-linear activation function.

[0041] Furthermore, after using ResNet to extract features from audio and video in S2, it further includes: Project the extracted visual and audio features through a linear projection layer into a unified feature space, denoted as and respectively, that is: where and are linear projection matrices, and are bias terms, and respectively represent video features and audio features.

[0042] Furthermore, each layer of the Conformer encoder includes four modules, namely: the first Feed-Forward module, the Self-Attention module, the Convolution module, and the second Feed-Forward module. Assuming the input feature is H, the processing formulas after passing through each module are as follows: where FFN represents the feed-forward network, MHSA represents the multi-head self-attention mechanism, and Conv represents the convolution module; after being processed by multiple layers of Conformer, the fused feature is output , where is the feature representation of the th data, is the set of feature representations of all data.

[0043] The projected visual and audio features are input into the Conformer encoder to capture the temporal and context-dependent information between modalities. The Conformer encoder combines convolution and self-attention mechanisms and is suitable for sequence modeling tasks.

[0044] Furthermore, in the S2: training stage, the shared representations of the multi-modal are respectively input into the forward and backward S4D decoders. The forward and backward S4D decoders are respectively used to predict the sequential sequence and the reverse sequence of the text, specifically including: (1) Use the softmax function to convert the unnormalized logits into a probability distribution, so as to generate the selection probability for each word or character at each time step; specifically, given the logit vector z at the t-th time step, the softmax function is defined as: where represents the conditional probability of selecting at the current time step given the previous decoding output and the encoder input ; Logits in deep learning refer to the raw prediction values of the output layer of a model before applying the activation function. Specifically, Logits are the output results of a neural network model at the last layer (usually a fully connected layer or an output layer). These output results are the linear outputs of the model for each class and have not been processed by activation functions such as softmax or sigmoid. Logits have not undergone a transformation of the probability distribution, so they do not directly represent probabilities but can be regarded as the original measure of the model's confidence or probability for each class.

[0045] (2) Use the Beam Search algorithm to increase the likelihood of finding the global optimal solution; during the beam search process, the cumulative log probabilities are used to evaluate the scores of candidate sequences: where, is the generated candidate sequence.

[0046] To more effectively select the optimal solution in the decoding space, the Beam Search algorithm is usually used. This algorithm expands the search space by retaining the B candidate paths with the highest probabilities at each time step, thereby increasing the likelihood of finding the global optimal solution. Beam search calculates the log probabilities of all extended candidates at each time step and retains the B candidate paths with the largest log probabilities until all candidate sequences reach the preset maximum length or the end-of-generation marker. Finally, the sequence with the highest score is selected as the decoding output.

[0047] By combining softmax decoding with beam search, the model can effectively generate the sequence with the maximum conditional probability, avoiding the local optimality problem caused by greedy search, thereby improving the quality of sequence generation.

[0048] In this application, by incorporating audio information during the training process, the system can capture more speech features from a multi-modal perspective, making up for the possible ambiguity and uncertainty that may occur relying solely on video information. In addition, the temporal information in lip reading is very important, while a unidirectional decoder can only capture forward dependencies and may ignore backward semantic information. In this application, by using a bidirectional decoder, the model can utilize both forward and backward temporal features simultaneously, fully capturing context information and improving the accuracy of recognition. Moreover, the S4D module in this application uses a state space model and can better capture temporal structure information and long-term dependencies. Compared with the traditional self-attention mechanism, the S4D block has more advantages in temporal tasks because it can more effectively propagate and integrate information across the entire sequence. Therefore, the effect of long-text lip reading is improved.

[0049] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A Mongolian lip reading recognition method based on multimodal training, characterized in that: It includes S1: preprocessing stage, S2: training stage and S3: reasoning stage; S1: In the preprocessing stage, the words other than verbs in the language model training text are segmented into stems and case suffixes, and a pronunciation dictionary based on stems, case suffixes and verbs is established; the lip area in the video image is accurately extracted through the face detection algorithm and lip positioning algorithm, and the lip image in each frame of the video is cropped out; S2: In the training phase, ResNet is first used to extract features from audio and video to obtain feature representations of each modality. The feature representations are then input into the Conformer encoder, and multi-modal shared representations are extracted through deep fusion. The multi-modal shared representations are then input into the forward and reverse S4D decoders, respectively, which are used to predict the sequential and reverse sequences of the text, respectively. S3: In the inference stage, ResNet is first used to extract fine features of the video image, and the extracted video features are then input into the Conformer encoder for deep processing; finally, the forward S4D decoder is used to accurately predict the text content, thereby effectively capturing the semantic information in the video and generating the corresponding text output.

2. The Mongolian lip reading recognition method based on multimodal training as claimed in claim 1, characterized in that: The S4D decoder in the training phase and the inference phase includes three modules, namely, the S4D module (Structured State Space Block), the multi-head attention layer (Multi-Head Attention Block) and the feed forward layer (Feed Forward Block). The S4D decoder performs the following steps: Step 1: The decoder first receives the output features of the encoder and feeds them into the S4D module. The S4D module performs sequence modeling to effectively capture the structural dependency between timing information and features, thereby improving the timing performance of decoding. Step 2: The features from the S4D module are then input into a multi-head attention layer, which processes the input features in parallel through multiple attention heads to capture global context information and ensure that the model can focus on the dependencies between different time steps. In step 3, the features of the multi-head attention layer are then input into the feedforward layer. At this stage, the features undergo nonlinear transformation to enhance the complexity and expressiveness of the feature representation.

3. The Mongolian lip reading recognition method based on multimodal training as claimed in claim 2, characterized in that: The recursive formula of the S4D module is: in, is the hidden state, is the input feature, matrix and Represent the state transfer matrix and input weight matrix respectively; The core formula of the multi-head attention layer is: Where Q, K, and V represent query, key, and value, respectively. is the dimension of the key; The core formula of the feedforward layer is: in, and is a learnable weight matrix, and is the bias term.

4. The Mongolian lip reading recognition method based on multimodal training according to any one of claims 1 to 3, characterized in that: The ResNet is ResNet-18, which is a residual network, and the specific formula is: in, is the input feature, represents the residual function composed of the convolutional layer, { } is the convolution kernel parameter, is a non-linear activation function.

5. The Mongolian lip reading recognition method based on multimodal training as claimed in claim 4, characterized in that: After extracting features from audio and video using ResNet in S2, the following steps are also included: The extracted visual and audio features are projected into a unified feature space through a linear projection layer and are represented as and ,Right now: in, and is the linear projection matrix, and is the bias term, and Represent video features and audio features respectively.

6. The Mongolian lip reading recognition method based on multimodal training as claimed in claim 5, characterized in that: Each layer of the Conformer encoder includes four modules, namely: the first Feed-Forward module, the Self-Attention module, the Convolution module and the second Feed-Forward module. Assuming that the input feature is H, the processing formulas of each module are: Among them, FFN represents feedforward network, MHSA represents multi-head self-attention mechanism, and Conv represents convolution module; after multi-layer Conformer processing, the output fusion feature ,in For the The characteristic representation of data is is the feature representation set for all data.

7. The Mongolian lip reading recognition method based on multimodal training as claimed in claim 6, characterized in that: The S2: training phase, wherein the multimodal shared representation is input into the forward and reverse S4D decoders respectively, and used to predict the sequential sequence and reverse sequence of the text respectively, specifically includes: (1) Use the softmax function to convert the unnormalized logits into a probability distribution, thereby generating a selection probability for each word or character at each time step; specifically, given the logit vector z at the tth time step, the softmax function is defined as: in, Indicates the decoding output of a given preamble and encoder input When the current time step is selected The conditional probability of (2) Use the beam search algorithm to increase the possibility of finding the global optimal solution; during the beam search process, the accumulated logarithmic probability is used to evaluate the score of the candidate sequence: in, is the generated candidate sequence.

Images