A method for recognizing Chinese lip speech combined with tone information

Abstract

The application discloses a Chinese lip speech recognition method combined with tone information and belongs to the field of lip speech recognition. The method comprises the following steps: obtaining user input, wherein the user input comprises tone information and mouth movement video; pre-processing the user input to obtain pre-processed data; and constructing a multi-modal Chinese lip reading model, inputting the pre-processed data into the multi-modal Chinese lip reading model to obtain a lip speech recognition result. The application combines tone information with lip reading recognition technology, deeply analyzes visual information, combines with linguistic knowledge, constructs a lip reading recognition model containing tone information, and improves recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of lip reading recognition, and in particular relates to a Chinese lip reading recognition method combining tone information. Background Art

[0002] While silent speech has great potential for applications in virtual reality (VR), it presents ambiguity issues for Chinese speakers. Chinese and English belong to different language families, and there are many differences between the two. The most notable difference is that Mandarin Chinese is a tonal language. Tones are used to distinguish word meanings in Chinese. Even words with the same lip movements can have completely different meanings due to different tones. For example, the lip movements for "rock" and "cover" may appear to be the same, but their meanings are completely different. The lack of tonal information when using silent speech input limits the accuracy of Chinese speakers' expressions.

[0003] Therefore, this paper proposes a tone-assisted silent speech input method for Chinese speakers. In this method, users input the tones of Chinese characters simultaneously with the silent speech. This method significantly reduces ambiguity during input, thereby significantly improving input accuracy. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a Chinese lip reading recognition method that combines tone information to solve the problems existing in the above prior art.

[0005] To achieve the above object, the present invention provides a Chinese lip reading recognition method incorporating tone information, comprising:

[0006] Obtaining user input, wherein the user input includes tone information and mouth movement video;

[0007] Preprocessing the user input to obtain preprocessed data;

[0008] A multimodal Chinese lip reading model is constructed, and the preprocessed data is input into the multimodal Chinese lip reading model to obtain a lip reading recognition result.

[0009] Optionally, the process of obtaining user input includes:

[0010] The user wears an HTC VIVE head-mounted display and places their thumb on the touchpad of a handheld controller. The camera on the HTC VIVE head-mounted display captures a video of the user's mouth movements. Simultaneously, the touchpad records a trajectory of the user's tone information, thereby obtaining the tone information and the mouth movement video.

[0011] The picture frames in the mouth movement video correspond one-to-one to the tone trajectory coordinates in the tone information.

[0012] Optionally, the process of preprocessing the user input comprises preprocessing the tonal information, wherein the process of preprocessing the tonal information comprises:

[0013] converting the trajectory coordinates of the tonal information into a bitmap image;

[0014] inputting the bitmap image into a convolutional neural network for convolutional processing to obtain convolutional features;

[0015] mapping the convolutional features to tonal classification results to obtain preprocessed tonal information.

[0016] Optionally, the process of preprocessing the user input comprises preprocessing the mouth movement video, and the process of preprocessing the mouth movement video comprises:

[0017] retrieving mouth key points in the mouth movement video through Mediapipe;

[0018] calculating a mouth center based on the mouth key points;

[0019] cutting video frames into preprocessed mouth images of 96*96 based on the mouth center.

[0020] Optionally, the process of inputting the preprocessed data into the multi-modal Chinese lip reading model to obtain the lip speech recognition result comprises:

[0021] extracting a video feature vector of the mouth movement video based on the preprocessed mouth images;

[0022] concatenating the preprocessed tonal information and the video feature vector to obtain a tonal enhanced feature vector;

[0023] under the guidance of a tonal sequence guide, using the tonal enhanced feature vector to make the decoder obtain the lip speech recognition result step by step.

[0024] Optionally, the process of obtaining the video feature vector comprises:

[0025] extracting spatial features of the mouth movement video based on ResNet-18;

[0026] performing temporal modeling using a conformal encoder based on the spatial features, and replacing the first convolutional layer in the ResNet-18 with a 3D convolutional layer to obtain compressed video features;

[0027] obtaining a video feature vector based on the compressed video features.

[0028] Optionally, the process of concatenating the preprocessed tonal information and the video feature vector comprises:

[0029] The tone sequence is labeled to obtain a tone label, and the interval between Chinese characters is marked as 0;

[0030] The tone label is mapped to a spatial feature to obtain a continuous tone vector;

[0031] The continuous tone vector and the video feature vector are spliced frame by frame to obtain a tone-enhanced feature vector.

[0032] Optionally, the interaction module comprises: a UI submodule, a data acquisition submodule, a data preprocessing submodule, and a lip reading submodule.

[0033] The UI submodule is configured to prompt user operation steps and display recognition results.

[0034] The data acquisition submodule is configured to acquire mouth movement videos and tone information.

[0035] The data preprocessing submodule is configured to preprocess the acquired mouth movement videos and tone information.

[0036] The lip reading submodule is configured to perform lip reading to infer text content expected to be input by a user.

[0037] Compared with the prior art, the present application has the following advantages and technical effects:

[0038] The present application combines tone information and lip reading recognition technology, analyzes visual information in depth, and constructs a lip reading recognition model containing tone information by combining linguistic knowledge, thereby improving recognition accuracy.

[0039] The present application not only promotes technical innovation of lip reading technology in processing tone language and improves the accuracy of Chinese recognition, but also provides better communication barrier-free support for hearing-impaired and speech-impaired people. In addition, this technology enriches the application scenarios in a multi-language environment and promotes the development of this field. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated herein in their entirety. The illustrative embodiments of the present application, and their description, are used to explain the present application and are not intended to limit the present application unduly.

[0041] Figure 1 A ToneLipNet model diagram of an embodiment of the present application;

[0042] Figure 2 A running process of the "tone-guided sequence modeling" part of an embodiment of the present application. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Example 1

[0046] This paper first proposes a sequence-to-sequence Chinese lip-reading model that embeds tone information. Compared with the baseline system, it embeds visual information and tone information based on user input, and realizes the inclusion of tone modeling when performing sentence-level lip reading recognition.

[0047] like Figure 1 As shown in the figure, the user's input tone information and mouth movement data are first collected and processed separately to prepare for the subsequent recognition process. Then, the proposed multimodal Chinese lip reading model ToneLipNet is used for recognition, and the recognition results are finally fed back to the user.

[0048] When the user's mouth shapes input a Chinese character, the corresponding tone symbol of the Chinese character is drawn on the touchpad of the controller at the same time. The mouth movement of each Chinese character is carried out simultaneously with the hand drawing movement of the tone. Gestures are designed to represent the five tones of Chinese. The first tone is represented by a horizontal line; the second tone is represented by a slanted line from the lower left to the upper right; the third tone is represented by a U-shaped curve that first descends and then rises; the fourth tone is represented by a slanted line from the upper left to the lower right. In Chinese, there is usually no tone symbol for the light tone. The fifth tone (light tone) is designed to be represented by a static gesture on the touchpad. This design effectively reflects its characteristics in spoken language, that is, the pronunciation is relatively short and lacks obvious changes in rising and falling.

[0049] Information collection: The user wears an HTC VIVE head-mounted display and uses the touchpad on the handheld controller to input tones. When the user's thumb is placed on the touchpad, the camera integrated into the helmet is activated to start capturing a video of the mouth movements, while the touchpad starts recording the coordinates (x, y) on the trajectory drawn by the user. The system synchronously records the mouth video frames and the points on the tone gesture trajectory at a fixed frequency (60Hz). Specifically, the system records 60 frames of images per second and captures 60 corresponding tone trajectory coordinates. When the user's thumb is lifted from the touchpad, a complete tone gesture trajectory is recorded.

[0050] Not every frame has tone information in a sentence, because there is a time interval between the lifting action of the user's thumb after the last Chinese character input is completed and the putting down action when the next character starts to be input. These intervals contain the transition action between the mouth actions of two characters, which is helpful for recognition, so these video frames are retained.

[0051] Information processing: For tone information, this embodiment builds a convolutional neural network to recognize the tone track. The trackpad records the user's gesture track and converts it into a 30x30 pixel bitmap image. The CNN model processes the image through three convolutional layers (20x5x5, 50x5x5, 100x5x5), and after each convolution, a max pooling layer is added to reduce the feature map size and reduce the amount of calculation. The pooling operation reduces the height and width by half. After three layers of convolution and pooling, the features are flattened and further analyzed through fully connected layers. Finally, the model maps the features to the corresponding classification results (Tone1-Tone5) through the output layer containing Softmax, completing the recognition. For video frame information, Mediapipe is used to detect the mouth key points in the video frame, and then the mouth center is calculated according to the mouth key points. Then the video frame is cut to 96*96 size according to the mouth center.

[0052] Lip reading: The multimodal Chinese lip reading model ToneLipnet proposed in this embodiment is an end-to-end sentence-level model that maps variable-length video frame sequences and tone sequences to text. It consists of two parts: tone-enhanced feature extraction and tone-guided sequence modeling. This embodiment uses the baseline code of CNVSRC2023 (Chinese Continuous Visual Speech Recognition Challenge) as the baseline [https: / / cnceleb.org / competition].

[0053] Tone-enhanced feature extraction: First, we use ResNet-18 and conformer encoder to extract spatial features, and then use conformer encoder to model temporal dynamics. The first convolutional layer in ResNet-18 is replaced by a 3D convolutional layer. Finally, a global average pooling layer is added to compress visual features. The conformer encoder consists of an embedding layer, followed by a series of conformer blocks. Each frame in the original video is extracted as a 768-dimensional feature vector. The final feature is of the dimension (T x 768), where T is the number of frames in the original video. Then, the tone information is combined with the video feature vector frame by frame. The tone sequence (Tone1 to Tone5 are labeled as numbers 1-5, and the interval between Chinese characters is labeled as 0) is mapped to a 64-dimensional feature space through an embedding layer. This step converts discrete tone labels into continuous feature vectors, making it easier to integrate tone information with video features. Finally, the embedded tone vector and the video feature vector are concatenated frame by frame to generate an enhanced feature representation, i.e., tone-enhanced features. This process combines visual information with non-visual information, synchronously fusing the spatiotemporal dynamics of lip movements and the changes in tone, thereby significantly improving the accuracy and robustness of the model in lip-reading without audio signals.

[0054] As shown in Figure 2 Tone-guided sequence modeling: First, the known Chinese character sequence is embedded into a 768-dimensional vector. Absolute position encoding is also added to the embedding. Then the tone of the Chinese character is mapped to a 64-dimensional feature space through a tone embedding layer. The 64-dimensional vector is concatenated with the Chinese character vector word by word to obtain a tensor of shape (L, 832), where L is the length of the predicted Chinese character sequence (including the start token <sos>). Unlike the previous tone-enhanced feature extraction, the tone corresponding to the Chinese character is not spliced on it. The tone of the next word is always spliced to the current word to inform the model of the tone of the next word in advance, guiding the prediction process of the decoder. A six-layer transformer decoder is used for prediction, and each layer includes a self-attention layer and a cross-attention layer. Among them, the K=Q=V of the self-attention layer. The cross-attention layer takes the features output by the self-attention layer as Q, and the tone-enhanced features as K and V (K=V). A complete prediction process is shown in Figure 2 , first, the start token <sos>The tone "fourth tone" of the first character "去" is input into the Transformer decoder, and the model combines the tone-enhanced features to predict "去". Then, the newly predicted "去" is concatenated with the embedding vector of the next tone "first tone", and the model predicts the next character "吃" based on the known Chinese characters, the tone of the next character, and the tone-enhanced features. And so on, finally predicting "去吃饭吗".

[0055] Loss function: Let \(x=[x_1,\ldots,x T \) and \(y = [y_1,\ldots,y L \) represent the input feature sequence and the target sequence respectively, where \(T\) and \(L\) are their respective lengths. The CTC loss assumes conditional independence between predictions and is expressed as In contrast, the loss based on the attention mechanism eliminates this assumption by directly using the chain rule to estimate the posterior probability: In this embodiment, a hybrid loss of the two is used to enforce monotonic alignment while avoiding the assumption of conditional independence. The loss function is calculated as follows:

[0056] \(L=\alpha\log p CTC (y|x)+(1 - \alpha)\log p CE (y|x)

[0057] where \(\alpha\) is the weight that balances the CTC loss and the loss based on the attention mechanism.

[0058] As a specific implementation of this embodiment, the baseline model is compared with ToneLipNet on the publicly available CN-CVS dataset, aiming to preliminarily verify the improvement effect of tone information on the accuracy of silent speech input. In addition, the model trained in the experiment will be used as the pre-training basis for subsequent actual datasets. By training on a large-scale dataset, the model can more fully capture diverse corpus information, thereby enhancing its generalization ability in actual application scenarios.

[0059] The baseline model is an end-to-end sentence-level model that maps a variable-length sequence of video frames to text. The encoder consists of ResNet and Conformer, and the decoder is based on the Transformer decoder.

[0060] For each piece of video data in CN-CVS, there is a corresponding audio file, as well as a sequence of text labels and the tone corresponding to each character. In order to meet the requirements of ToneLipNet for input data, it is necessary to match the corresponding tone for each frame in the video, and the original data set does not contain this information. In order to obtain the corresponding tone for each frame in the video, it is necessary to obtain the start and end timestamps of each Chinese character in the video, and it is very difficult to separate Chinese characters only by video information. Therefore, the audio information synchronized with the video is used to obtain the timestamps in the audio track, and then it is mapped to the video frame. The synchronization of video frames and word-by-word text is achieved by forced alignment. For this purpose, a pinyin-to-phoneme mapping dictionary is constructed. A method based on Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM) is used to align the audio and word-by-word text through the MFA (Montreal ForcedAligner) tool to generate the start and end timestamps of each Chinese character. Thus, the tone of each character and the video frame are accurately matched.

[0061] The learning rate scheduling strategy combines the rising phase and the cosine annealing decay. In the rising phase, the learning rate is linearly increased from zero to a predefined predetermined initial learning rate. After the rising phase, the learning rate gradually decays according to the cosine annealing function, gradually decreasing as the training iteration progresses, until reaching a minimum value of zero. The initial learning rate is set to 0.001, the warm-up period is 5 cycles, and the weight decay is 0.03. The model is trained using two NVIDIA GeForce RTX 3090 GPUs, for a total of 75 cycles, with a total training time of 8 days and 10 hours.

[0062] Measurement indicators: Character Error Rate (CER) is used to measure the error degree of the prediction result. CER is defined as (S+D+I) / N, where S is the number of replacements required from the true sequence to the predicted sequence, D is the number of deletions, I is the number of insertions, and N is the number of Chinese characters in the reference sequence.

[0063] Accuracy measures the performance of the model from another perspective, by calculating the proportion of correct predictions. Accuracy is defined as C / N, where C represents the number of correctly predicted Chinese characters, and N represents the total number of Chinese characters in the reference sequence. The results are shown in Table 1.

[0064] Table 1

[0065]

[0066]

[0067] As a specific embodiment of the present embodiment includes a Chinese lip speech recognition method based on a Chinese lip speech recognition system combined with tone information is developed.

[0068] The Chinese lip language recognition system comprises a plurality of sub-modules, namely, a UI sub-module, a data acquisition sub-module, a data preprocessing sub-module, and a lip language recognition sub-module.

[0069] The Chinese lip language recognition system is developed based on a Unity development engine and using a C# programming language, and effectively acquires user input information and displays an interface. In addition, the lip language recognition sub-module and the data preprocessing sub-module are encapsulated by using a Flask framework to return corresponding lip language recognition results according to lip movement videos or tone information.

[0070] First, the system acquires tone information and mouth movement data input by a user, and a Unity front end communicates with a backend interface through a UnityWebRequest class. A data preprocessing sub-module encapsulated by a Flask backend processes the two types of data to prepare for a subsequent recognition process. Then, a lip language recognition sub-module uses a proposed multi-modal Chinese lip reading model ToneLipNet to perform recognition, and finally feeds back the recognition result to the user. A UI sub-module is responsible for displaying the recognition result.

[0071] The information acquisition sub-module: the user wears an HTC VIVE head-mounted display and uses a touchpad on a handheld controller to input tones. When the user's thumb is placed on the touchpad, a camera integrated in the headset is activated to start recording a video of mouth movements, and the touchpad starts recording the coordinates (x, y) on the trajectory drawn by the user. The system synchronously records the mouth video frames and the points on the tone gesture trajectory at a fixed frequency (60 Hz). Specifically, the system records 60 frames of images per second, and captures 60 tone trajectory coordinates corresponding to each frame. When the user's thumb is lifted from the touchpad, a complete tone gesture trajectory is recorded.

[0072] Not every frame in a sentence has tone information, because there is a time interval between the lifting action of the user's thumb after the input of the previous Chinese character is completed and the putting-down action at the start of the input of the next character. These intervals contain the transition actions between the mouth movements of two characters, which are helpful for recognition, and therefore these video frames are retained.

[0073] Information processing sub-module: For tone information, this embodiment builds a convolutional neural network to identify the tone track. The trackpad records the user's gesture track and converts it into a 30x30 pixel bitmap image. The CNN model processes the image through three convolutional layers (20x5x5, 50x5x5, 100x5x5), each followed by a max pooling layer to reduce the feature map size and reduce the amount of calculation. The pooling operation reduces the height and width by half. After three layers of convolution and pooling, the features are flattened and further analyzed through fully connected layers. Finally, the model maps the features to the corresponding classification results (Tone1-Tone5) through an output layer containing Softmax, completing the identification. For video frame information, the mouth key points in the video frame are detected by Mediapipe, and the mouth center is calculated according to the mouth key points. Then the video frame is cut into 96*96 size according to the mouth center.

[0074] As a specific embodiment of this embodiment: In order to further verify the performance on actual user data, this embodiment collects a dataset containing 5498 samples. First, this embodiment constructs a Chinese corpus, randomly selects 500 English phrases from the standard text input corpus. A native Chinese speaker translates the English phrases into Chinese, and another expert checks and corrects them to ensure the correctness of the translation results, in line with the Chinese order and style.

[0075] This embodiment recruited 12 participants. Each participant entered the 500 Chinese sentences in the corpus once. The participant rested for 5 minutes after entering 50 sentences each time. When collecting, the participant sat in a chair and wore a head-mounted display. The system interface displays the current sentence, and the participant inputs the tone using the touchpad on the handle while speaking the sentence with lip movements, and presses the trigger button on the handle to submit the sentence. At this time, the system saves the data and displays the next sentence to be entered. The system will save all video frames from the time the participant's finger touches the tablet to the time the trigger button is pressed, and identify all the tones input by the participant, record the start frame and end frame corresponding to each tone. Artificially screened each data, deleted samples with incorrect tone input and mouth video recording error, and finally the dataset contains 5498 samples.

[0076] This embodiment uses the collected dataset to verify the facilitating effect of tone in actual use and evaluate the robustness of the model proposed in this embodiment under different test conditions. In these three conditions, this embodiment compares the baseline and the Tonelipnet proposed in this embodiment, and the measurement indicators use the character error rate (CER) and the accuracy.

[0077] Three data set division methods are designed to evaluate the performance of the model, each method has its specific test target:

[0078] 1. The first division method is to randomly divide the data set, and reserve a part of the samples as the test set, to evaluate the performance of the model under the condition of conventional test set division.

[0079] 2. The second division method is to use part of the participants' samples as the test set, to simulate the model's recognition ability of new individuals in actual application, and verify the model's performance on unseen face data.

[0080] 3. The third division method is to select a part of the 500 sentences in the corpus as the test set, which will not appear during training, to test the model's recognition ability for new sentences.

[0081] For ToneLipNet, the model trained on the previous CNCVS is fine-tuned. This embodiment adopts a two-stage training strategy, the first stage only optimizes the CTC loss. The second stage jointly optimizes the CTC loss and the CE loss. The learning rate of the first stage is fixed at 5.0e-06. The second stage uses two optimizers with different learning rates, the first optimizer updates the gradient for the parameters of the encoder, and the learning rate is set to 1.0e-06, the second optimizer updates the gradient for the parameters of the decoder, and the learning rate is set to 1.0e-06. The learning rates of the two optimizers gradually decay according to the cosine function, gradually decreasing to zero as the training iteration progresses. The entire training process uses exponential moving average (Exponential Moving Average), and the decay factor is 0.998.

[0082] This embodiment randomly selects 548 samples from the data set as the test set, and the remaining 4950 samples as the training set, the ratio of training set to test set is 9:1. After statistics, it is found that the individuals and sentences appearing in the test set have all appeared in the training set. The experimental results are shown in Table 2, ToneLipNet outperforms the baseline method in performance, with a character error rate (CER) of 19%, and the CER of the baseline method is 46%. The accuracy of ToneLipNet is 79%, and the accuracy of the baseline is 43%. This leads to a CER of 0.41 times that of the baseline, and an accuracy of 1.8 times that of the baseline. This shows that the ToneLipNet of this embodiment still has obvious advantages on the data actually input by the user, and the improvement brought by the addition of tone still exists.

[0083] The training set was composed of samples from 12 participants. One 6-fold cross-validation was performed, each fold containing samples from two participants, and the participants in each fold were randomly assigned. Since each participant recorded 500 sentences in the corpus, ideally each fold should contain 500 samples, but since some samples failed to be successfully input, the number of samples in each validation fold was slightly different, 903, 917, 918, 923, 917, 920 respectively. The test results are shown in Table 2, ToneLipNet outperformed the baseline method in performance, with a Character Error Rate (CER) of 28%, while the CER of the baseline method was 64%. The accuracy of ToneLipNet was 70%, while the accuracy of the baseline was 21%. This shows that after adding tone information, the CER is reduced to 0.43 times that of the baseline, and the accuracy is increased to 3.3 times that of the baseline. Compared with the test results under the normal condition in 9.2.1, both ToneLipNet and the baseline have a decrease in performance when encountering unseen individuals. The CER of ToneLipNet increased by 9% compared with the normal condition, while the CER of the baseline increased by 18% compared with the normal condition. The accuracy of ToneLipNet decreased by 9%, which is 0.88 times that of the normal condition. However, the accuracy of the baseline decreased by 22% (about half), which is 0.48 times that of the normal condition. This shows that the performance of ToneLipNet of the present embodiment is more stable when encountering unseen individuals.

[0084] The embodiment evaluates the processing ability of the ToneLipNet model for new sentences by 10-fold cross-validation. Each fold contains 50 sentences in the corpus. Since there are a total of 500 sentences in the corpus, and each sentence is spoken by 12 users, ideally, each fold should contain 600 samples (12 users x 50 sentences). However, in practice, due to the absence of some samples, the number of samples in each validation fold is slightly different, which is 547, 541, 551, 555, 563, 563, 554, 544, 549, and 531, respectively. The test results are shown in Table 2. The ToneLipNet outperforms the baseline method in performance, with a character error rate (CER) of 37%, and the CER of the baseline method is 69%. The accuracy of the ToneLipNet is 62%, and the accuracy of the baseline is 21%. This shows that after adding the tone information, the CER is reduced to 0.53 times of the baseline, and the accuracy is increased by 2.9 times. Compared with the results under the conventional test conditions in 9.2.1, the performance of both models decreases when encountering new sentences, but the performance of the ToneLipNet is more stable. The CER of the ToneLipNet increases by 18% compared with the conventional conditions, while the CER of the baseline increases by 23% compared with the conventional conditions. The accuracy of the ToneLipNet decreases by 17% and is 0.78 times of the conventional conditions. The accuracy of the baseline decreases by 22% (about half), and is 0.48 times of the conventional conditions. This shows that the performance of the ToneLipNet of the embodiment is more stable when encountering new sentences. The introduction of the tone makes the performance of the recognition more stable under various test conditions.

[0085] Table 2

[0086]

[0087] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application can be easily thought by those skilled in the art, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / sos> < / sos>

Claims

1. A Chinese lip reading method combined with tone information, characterized in that, The method comprises the following steps: obtaining user input, wherein the user input comprises tone information and mouth movement video; preprocessing the user input to obtain preprocessed data; constructing a multi-modal Chinese lip reading model, and inputting the preprocessed data into the multi-modal Chinese lip reading model to obtain lip speech recognition results; The process of obtaining user input comprises: The user wears an HTC VIVE head-mounted display, and the user's thumb is placed on the touchpad of the handheld controller. A camera on the HTC VIVE head-mounted display captures the mouth movement video, and at the same time, the touchpad records the trajectory drawn by the user about the tone information to obtain the tone information and the mouth movement video. The picture frames in the mouth movement video correspond one-to-one to the trajectory coordinates in the tone information. The process of preprocessing the user input comprises preprocessing the tone information, wherein the process of preprocessing the tone information comprises: Converting the trajectory coordinates of the tone information into a bitmap image; inputting the bitmap image into a convolutional neural network for convolutional processing to obtain convolutional features; mapping the convolutional features to tone classification results to obtain preprocessed tone information; The process of preprocessing the user input comprises preprocessing the mouth movement video, and the process of preprocessing the mouth movement video comprises: Retrieving mouth key points in the mouth movement video through Mediapipe; calculating the mouth center based on the mouth key points; cutting video frames into 96*96 preprocessed mouth images based on the mouth center; The process of inputting the preprocessed data into the multi-modal Chinese lip reading model to obtain lip speech recognition results comprises: extracting a video feature vector of the mouth movement video based on the preprocessed mouth image; concatenating the preprocessed tone information and the video feature vector to obtain a tone-enhanced feature vector; under the guidance of a tone sequence guide, using the tone-enhanced feature vector to make the decoder obtain lip speech recognition results step by step.

2. The Chinese lipreading method incorporating tonal information according to claim 1, wherein, The process of obtaining the video feature vector comprises: extracting spatial features of the mouth movement video based on ResNet-18; based on the spatial features, using a conformal encoder for time series modeling, and replacing the first convolutional layer in the ResNet-18 with a 3D convolutional layer to obtain compressed video features; obtaining a video feature vector based on the compressed video features.

3. The method according to claim 1, wherein the method further comprises: determining the tone information of the Chinese words in the Chinese lip speech according to the phonetic information of the Chinese words in the Chinese lip speech. The process of concatenating the preprocessed tone information and the video feature vector comprises: labeling the tone sequence to obtain tone labels, and the interval between Chinese characters is marked as 0; mapping the tone labels to spatial features to obtain continuous tone vectors; concatenating the continuous tone vectors and the video feature vectors frame by frame to obtain tone-enhanced feature vectors.

4. A Chinese lip reading system combined with tone information, characterized in that, The system for implementing the Chinese lip speech recognition method combined with tone information according to any one of claims 1-3 comprises a UI submodule, a data acquisition submodule, a data preprocessing submodule, and a lip speech recognition submodule; The UI submodule is used to prompt the user to operate the steps and display the recognition results. The data acquisition submodule is used for acquiring the mouth action video and the intonation information; The data preprocessing submodule is used for preprocessing the acquired mouth action video and the intonation information; The lip language recognition submodule is used for lip language recognition, and infers the text content expected to be input by the user.

Images