Chinese lip language recognition method in combination with tone information

Abstract

The invention discloses a Chinese lip language recognition method in combination with tone information, and belongs to the field of lip language recognition, the method comprises the following steps: obtaining user input which comprises tone information and a mouth action video; preprocessing the user input to obtain preprocessed data; and constructing a multi-modal Chinese lip reading model, and inputting the preprocessed data into the multi-modal Chinese lip reading model to obtain a lip language recognition result. According to the method, tone information and a lip reading recognition technology are combined, visual information is deeply analyzed, linguistic knowledge is combined, a lip reading recognition model containing the tone information is constructed, and the recognition accuracy is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of lip reading recognition, and in particular relates to a Chinese lip reading recognition method combined with tone information. Background Art

[0002] Although silent speech has great potential for application in virtual reality (VR), silent speech input has an ambiguity problem for Chinese speakers. Chinese and English belong to different language families, and there are many differences between the two. The most notable one is that Mandarin Chinese is a tonal language. Tones are used to distinguish word meanings in Chinese, and even words with the same lip movements can have completely different meanings due to different tones. For example, the lip movements of "rock" and "cover" may seem the same, but their meanings are completely different. The lack of tonal information limits the accuracy of Chinese speakers' expressions when using silent speech input.

[0003] Therefore, this paper proposes a tone-assisted silent speech input method for Chinese users. In this method, users input the tone of Chinese characters while inputting silent speech. This method will greatly reduce the ambiguity problem in the input process, thereby significantly improving the accuracy of input. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a Chinese lip reading recognition method combining tone information to solve the above problems existing in the prior art.

[0005] To achieve the above object, the present invention provides a Chinese lip reading recognition method combined with tone information, comprising:

[0006] Obtaining user input, wherein the user input includes tone information and mouth movement video;

[0007] Preprocessing the user input to obtain preprocessed data;

[0008] A multimodal Chinese lip-reading model is constructed, and the preprocessed data is input into the multimodal Chinese lip-reading model to obtain a lip reading recognition result.

[0009] Optionally, the process of obtaining user input includes:

[0010] The user wears an HTC VIVE head mounted display, and the user's thumb is placed on the touch pad of the handheld controller. The camera on the HTC VIVE head mounted display captures a mouth movement video. At the same time, the touch pad records the trajectory of the tone information drawn by the user, thereby obtaining the tone information and the mouth movement video.

[0011] The picture frames in the mouth movement video correspond one to one with the tone trajectory coordinates in the tone information.

[0012] Optionally, the process of preprocessing the user input includes preprocessing the tone information, wherein the process of preprocessing the tone information includes:

[0013] converting the trajectory coordinates of the tone information into a bitmap image;

[0014] Inputting the bitmap image into a convolutional neural network for convolution processing to obtain convolution features;

[0015] The convolution feature is mapped to the tone classification result to obtain preprocessed tone information.

[0016] Optionally, the process of preprocessing the user input includes preprocessing the mouth movement video, and the process of preprocessing the mouth movement video includes:

[0017] Retrieving mouth key points in the mouth action video through Mediapipe;

[0018] Calculating a mouth center based on the mouth key points;

[0019] The video frame is cut into 96*96 pre-processed mouth images based on the mouth center.

[0020] Optionally, the process of inputting the preprocessed data into the multimodal Chinese lip reading model to obtain a lip reading recognition result includes:

[0021] Extracting a video feature vector of a mouth action video based on the preprocessed mouth image;

[0022] splicing the preprocessed tone information with the video feature vector to obtain a tone enhancement feature vector;

[0023] Under the guidance of the tone sequence, the tone enhancement feature vector is used to enable the decoder to gradually obtain the lip reading recognition result.

[0024] Optionally, the process of obtaining the video feature vector includes:

[0025] Extracting spatial features of the mouth movement video based on ResNet-18;

[0026] Based on the spatial features, a conformal encoder is used to perform temporal modeling, and the first convolution layer in the ResNet-18 is replaced with a 3D convolution layer to obtain compressed video features;

[0027] A video feature vector is obtained based on the compressed video feature.

[0028] Optionally, the process of splicing the preprocessed tone information with the video feature vector includes:

[0029] Mark the tone sequence to get the tone tag, and the interval between Chinese characters is marked as 0;

[0030] Mapping the tone tags to spatial features to obtain continuous tone vectors;

[0031] The continuous tone vectors are concatenated with the video feature vectors frame by frame to obtain a tone enhancement feature vector.

[0032] Optionally, the interaction module includes: a UI submodule, a data collection submodule, a data preprocessing submodule, and a lip reading recognition submodule;

[0033] The UI submodule is used to prompt the user operation steps and display the recognition results;

[0034] The data acquisition submodule is used to obtain mouth movement video and tone information;

[0035] The data preprocessing submodule preprocesses the acquired mouth movement video and tone information;

[0036] The lip reading recognition submodule is used for lip reading recognition to infer the text content that the user expects to input.

[0037] Compared with the prior art, the present invention has the following advantages and technical effects:

[0038] The present invention combines tone information with lip reading recognition technology, constructs a lip reading recognition model containing tone information by deeply analyzing visual information and combining linguistic knowledge, and improves recognition accuracy.

[0039] The invention not only promotes the technical innovation of lip reading technology in processing tonal languages ​​and improves the accuracy of Chinese recognition, but also provides better communication barrier-free support for people with hearing impairment and speech disorders. In addition, this technology enriches the application scenarios in a multilingual environment and promotes the development of this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 This is a ToneLipNet model diagram of an embodiment of the present invention;

[0042] Figure 2 This is the running process of the "tone-guided sequence modeling" part of the embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] The present invention firstly proposes a sequence-to-sequence Chinese lip-reading model that embeds tone information. Compared with the baseline system, it embeds visual information and tone information based on user input, and realizes the inclusion of tone in modeling when performing sentence-level lip reading recognition.

[0047] like Figure 1 As shown in the figure, first, the tone information and mouth movement data input by the user are collected, and the two types of data are processed separately to prepare for the subsequent recognition process. Then, the proposed multimodal Chinese lip reading model ToneLipNet is used for recognition, and finally the recognition results are fed back to the user.

[0048] When the user's mouth shapes input a Chinese character, the tone symbol corresponding to the Chinese character is drawn on the touchpad of the controller at the same time. The mouth movement of each Chinese character is performed simultaneously with the hand tone drawing movement. Gestures are designed to represent the five tones of Chinese. The first tone is represented by a horizontal line; the second tone is represented by a slanted line from the lower left to the upper right; the third tone is represented by a U-shaped curve that first descends and then rises; the fourth tone is represented by a slanted line from the upper left to the lower right. In Chinese, there is usually no tone symbol for the light tone, and the fifth tone (light tone) is designed to be represented by a static gesture on the touchpad. This design effectively reflects its characteristics in spoken language, that is, the pronunciation is relatively short and lacks obvious changes in rising and falling.

[0049] Information collection: The user wears the HTC VIVE head-mounted display and uses the touchpad on the handheld controller to input tones. When the user's thumb is placed on the touchpad, the camera integrated in the helmet is activated to start shooting a video of the mouth movement, and the touchpad starts recording the coordinates (x, y) on the trajectory drawn by the user. The system synchronously records the mouth video frames and the points on the tone gesture trajectory at a fixed frequency (60Hz). Specifically, the system records 60 frames of images per second and captures 60 tone trajectory coordinates that correspond to each one. When the user's thumb is lifted from the touchpad, a complete tone gesture trajectory is recorded.

[0050] In a sentence, not every frame has tone information, because there is a time interval between the user's thumb lifting action after the previous Chinese character is input and the thumb lowering action when the next character is input. These intervals contain the transition between the mouth movements of two characters, which is helpful for recognition, so these video frames are retained.

[0051] Information processing: For tone information, this embodiment builds a convolutional neural network to identify the tone trajectory. The touchpad records the user's gesture trajectory and converts it into a 30×30 pixel bitmap image. The CNN model processes the image through three convolutional layers (20×5×5, 50×5×5, 100×5×5), and each layer of convolution is followed by a maximum pooling layer to reduce the size of the feature map and reduce the amount of calculation. The pooling operation reduces the height and width by half. After three layers of convolution and pooling, the features are flattened and further analyzed through the fully connected layer. Finally, the model maps the features to the corresponding classification results (Tone1-Tone5) through an output layer containing Softmax to complete the recognition. For video frame information, the mouth key points in the video frame are detected through Mediapipe, and the mouth center is calculated based on the mouth key points, and then the video frame is cut into a size of 96*96 based on the mouth center.

[0052] Lip reading recognition: The multimodal Chinese lip reading model ToneLipnet proposed in this example is an end-to-end sentence-level model that maps variable-length video frame sequences and tone sequences to text. It consists of two parts: tone-enhanced feature extraction and tone-guided sequence modeling. This example uses the baseline code of CNVSRC2023 (Chinese Continuous Visual Speech Recognition Challenge) as the baseline [https: / / cnceleb.org / competition].

[0053] Feature extraction for tone enhancement: First, spatial feature extraction is performed using ResNet-18 and conformal encoder, and then temporal modeling is performed using conformal encoder. The first convolutional layer in ResNet-18 is replaced by a 3D convolutional layer. Finally, a global average pooling layer is added to compress visual features. The Conformer encoder consists of an embedding layer and a series of conformer blocks connected to it. Each frame in the original video is extracted as a 768-dimensional feature vector. Finally, the feature of (T×768) dimension is obtained, where T is the number of frames in the original video. Subsequently, the tone information is combined with the video feature vector frame by frame. The tone sequence (Tone1 to Tone5 are marked as numbers 1-5, and the intervals between Chinese characters are marked as 0) is passed through an embedding layer and mapped to a 64-dimensional feature space. This step converts discrete tone labels into continuous feature vectors, so that the tone information can be more naturally integrated with the video features. Finally, the embedded tone vector is spliced ​​with the video feature vector frame by frame to generate an enhanced feature representation, namely the tone enhanced feature. This process combines visual information with non-visual information, synchronously integrating the spatiotemporal dynamics of lip reading and changes in tone, thereby significantly improving the accuracy and robustness of the model in lip reading recognition in the absence of speech signals.

[0054] like Figure 2 As shown in Figure 1, tone-guided sequence modeling: First, the known Chinese character sequence is embedded into a 768-dimensional vector. The absolute position code is also added to the embedding. The tone of the Chinese character is then passed through a tone embedding layer to map it to a 64-dimensional feature space. The 64-dimensional vector is concatenated with the Chinese character vector word by word to obtain a tensor of shape (L, 832), where L is the length of the predicted Chinese character sequence (including the start marker <sos>). Unlike the previous feature extraction of tone enhancement, the corresponding tone is not spliced ​​on the Chinese character. The tone of the next word is always spliced ​​to the current word to inform the model of the tone of the next word in advance and guide the prediction process of the decoder. A six-layer transformer decoder is used for prediction, each layer includes a self-attention layer and a cross-attention layer. Among them, K=Q=V of the self-attention layer. The cross-attention layer uses the features output by the self-attention layer as Q, and the tone-enhanced features as K and V (K=V). A complete prediction process is as follows Figure 2 As shown, first, start tagging <sos>The tone "fourth tone" of the first character "去 (qù)" is input into the Transformer decoder, and the model combines the tone-enhanced features to predict "去 (qù)". Then, the newly predicted "去 (qù)" is concatenated with the embedding vector of the next tone "first tone", and the model predicts the next character according to the known Chinese characters, the tone of the next character, and the tone-enhanced features: "吃 (chī)". And so on, finally predicting "去吃饭吗 (qù chī fàn ma)".

[0055] Loss function: Let \(x = [x_1,\cdots,x\) T and \(y = [y_1,\cdots,y\) L represent the input feature sequence and the target sequence respectively, where \(T\) and \(L\) are their respective lengths. The CTC loss assumes conditional independence between predictions and is expressed as In contrast, the loss based on the attention mechanism eliminates this assumption by directly estimating the posterior probability using the chain rule: In this embodiment, a hybrid loss of the two is used to enforce monotonic alignment while avoiding the assumption of conditional independence. The loss function is calculated as follows:

[0056] \(L=\alpha\log p\) CTC (y|x)+(1 - \alpha)\log p\) CE (y|x)

[0057] where \(\alpha\) is the weight that balances the CTC loss and the loss based on the attention mechanism.

[0058] As a specific implementation of this embodiment, the baseline model is compared with ToneLipNet on the publicly available CN - CVS dataset, aiming to preliminarily verify the improvement effect of tone information on the accuracy of silent speech input. In addition, the model trained in the experiment will be used as the pre - training basis for subsequent actual datasets. By training on a large - scale dataset, the model can more fully capture diverse corpus information, thereby enhancing its generalization ability in actual application scenarios.

[0059] The baseline model is an end - to - end sentence - level model that maps a variable - length sequence of video frames to text. The encoder consists of ResNet and Conformer, and the decoder is based on the Transformer decoder.

[0060] Each video data in CN-CVS has a corresponding audio file, as well as a sequence of text labels and the corresponding tone of each character. In order to meet the input data requirements of ToneLipNet, it is necessary to match the corresponding tone for each frame in the video, but the original dataset does not contain this information. In order to match the corresponding tone for each frame in the video, it is necessary to obtain the start and end timestamps of each Chinese character in the video. It is difficult to separate Chinese characters only through video information. Therefore, the audio information synchronized with the video is used to obtain the timestamp in the audio track and then correspond it to the video frame. The synchronization of video frames and verbatim text is achieved through forced alignment. To this end, a mapping dictionary from pinyin to phonemes is constructed. A method based on hidden Markov model (HMM) and Gaussian mixture model (GMM) is used to align the audio with the verbatim text through the MFA (Montreal ForcedAligner) tool to generate the start and end timestamps of each Chinese character. This provides an accurate match between the tone of each character and the video frame.

[0061] The learning rate scheduling strategy combines a ramp-up phase and cosine annealing decay. In the ramp-up phase, the learning rate increases linearly from zero to a predefined predetermined initial learning rate. After the ramp-up, the learning rate gradually decays according to the cosine annealing function, gradually decreasing as the training iterations proceed until it reaches a minimum value of zero. The initial learning rate is set to 0.001, the warm-up period is 5 cycles, and the weight decay is 0.03. The model was trained using two NVIDIA GeForce RTX 3090 GPUs for a total of 75 cycles, with a total training time of 8 days and 10 hours.

[0062] Measurement indicators: Character error rate (CER) is used to measure the error degree of the prediction results. CER is defined as (S+D+I) / N, where S is the number of substitutions required to get from the true sequence to the predicted sequence, D is the number of deletions, I is the number of insertions, and N is the number of Chinese characters in the reference sequence.

[0063] The accuracy rate measures the performance of the model from another perspective and is determined by calculating the proportion of correct predictions. The accuracy rate is defined as C / N, where the C numerator represents the number of correctly predicted Chinese characters and N represents the total number of Chinese characters in the reference sequence. The results are shown in Table 1.

[0064] Table 1

[0065]

[0066]

[0067] As a specific implementation of this embodiment, a Chinese lip reading recognition system combined with tone information is developed based on the Chinese lip reading recognition method.

[0068] The Chinese lip reading recognition system consists of multiple submodules, namely, UI submodule, data acquisition submodule, data preprocessing submodule, and lip reading recognition submodule;

[0069] The Chinese lip reading recognition system is developed based on the Unity development engine and the C# programming language, which enables effective acquisition of user input information and interface display. In addition, the Flask framework is used to encapsulate the lip reading recognition submodule and the data preprocessing submodule to realize the function of returning the corresponding lip reading recognition results based on the lip movement video or tone information.

[0070] First, the system collects the tone information and mouth movement data input by the user, and the Unity front-end communicates with the back-end interface through the UnityWebRequest class. The data preprocessing submodule encapsulated in the Flask back-end processes these two types of data respectively to prepare for the subsequent recognition process. Then, the lip reading recognition submodule uses the proposed multimodal Chinese lip reading model ToneLipNet for recognition, and finally feeds back the recognition results to the user. The UI submodule is responsible for displaying the recognition results.

[0071] Information collection submodule: The user wears the HTC VIVE head-mounted display and uses the touchpad on the handheld controller to input tones. When the user's thumb is placed on the touchpad, the camera integrated in the helmet is activated to start shooting a video of the mouth movement, and the touchpad starts recording the coordinates (x, y) on the trajectory drawn by the user. The system synchronously records the mouth video frames and the points on the tone gesture trajectory at a fixed frequency (60Hz). Specifically, the system records 60 frames of images per second and captures 60 tone trajectory coordinates that correspond to each one. When the user's thumb is lifted from the touchpad, a complete tone gesture trajectory is recorded.

[0072] In a sentence, not every frame has tone information, because there is a time interval between the user's thumb lifting action after the previous Chinese character is input and the thumb lowering action when the next character is input. These intervals contain the transition between the mouth movements of two characters, which is helpful for recognition, so these video frames are retained.

[0073] Information processing submodule: For tone information, this embodiment builds a convolutional neural network to identify the tone trajectory. The touchpad records the user's gesture trajectory and converts it into a 30×30 pixel bitmap image. The CNN model processes the image through three convolutional layers (20×5×5, 50×5×5, 100×5×5), and each layer of convolution is followed by a maximum pooling layer to reduce the size of the feature map and reduce the amount of calculation. The pooling operation halves the height and width. After three layers of convolution and pooling, the features are flattened and further analyzed through the fully connected layer. Finally, the model maps the features to the corresponding classification results (Tone1-Tone5) through an output layer containing Softmax to complete the recognition. For video frame information, the mouth key points in the video frame are detected through Mediapipe, and the mouth center is calculated based on the mouth key points, and then the video frame is cut into a size of 96*96 based on the mouth center.

[0074] As a specific implementation of this embodiment: In order to further verify the performance on actual user data, this embodiment collected a data set containing 5498 samples. First, this embodiment constructed a Chinese corpus and randomly selected 500 English phrases from the standard text input corpus. An English expert whose native language is Chinese was asked to translate the English phrases into Chinese, and another expert checked and corrected them to ensure the correctness of the translation results and conform to the word order and style of Chinese.

[0075] This embodiment recruited 12 participants. Each participant entered 500 Chinese character sentences in the corpus. The participant rested for 5 minutes after entering 50 sentences each time. During the collection, the participant sat on a chair and wore a head-mounted display. The system interface will display the current sentence. The participant uses the touchpad on the handle to input the tone while speaking the sentence using lip reading, and presses the trigger button of the handle to submit the sentence. At this time, the system saves the data and displays the next sentence to be entered. The system will save all video frames from the time the participant's finger touches the tablet to the time the trigger button is pressed, and identify all the tones entered by the participant, and record the start frame and end frame corresponding to each tone. Each piece of data was manually screened, and samples with incorrect tone input and mouth video recording errors were deleted. The final data set contains 5498 samples.

[0076] This embodiment uses the collected data set to verify the promotion effect of tone in actual use and evaluate the robustness of the model proposed by the present invention under different test conditions. In these three conditions, this embodiment compares the baseline and the Tonelipnet proposed in this embodiment, and the measurement indicators use character error rate (CER) and accuracy.

[0077] This example designs three data set partitioning methods to evaluate model performance, each with its own specific test objectives:

[0078] 1. The first division method is to randomly divide the data set and retain a portion of samples as a test set to evaluate the performance of the model under conventional test set division.

[0079] 2. The second division method is to use samples of some participants as test sets to simulate the model's recognition ability when facing completely new individuals in actual applications and to verify the model's performance on unseen face data.

[0080] 3. The third division method is to select a part of the 500 sentences in the corpus as the test set. These sentences will not appear during training, in order to test the model's ability to recognize new sentences.

[0081] For ToneLipNet, this embodiment is fine-tuned based on the model previously trained on CNCVS. This embodiment adopts a two-stage training strategy. In the first stage, only CTC loss is optimized. In the second stage, both CTC loss and CEloss are jointly optimized. The learning rate in the first stage is fixed to 5.0e-06. In the second stage, two optimizers with different learning rates are used. The first optimizer performs gradient updates on the parameters of the encoder, and the learning rate is set to 1.0e-06. The second optimizer performs gradient updates on the parameters of the decoder, and the learning rate is set to 1.0e-06. The learning rates of the two optimizers gradually decay according to the cosine function, and gradually decrease as the training iterations proceed until they reach zero. The entire training process uses an exponential moving average (Exponential Moving Average) with a decay factor of 0.998.

[0082] In this embodiment, 548 samples are randomly selected from the data set as the test set, and the remaining 4950 samples are used as the training set, with the ratio of the training set to the test set being 9:1. After statistics, it was found that the individuals and sentences that appeared in the test set had appeared in the training set. The experimental results are shown in Table 2. ToneLipNet surpassed the baseline method in performance, with a character error rate (CER) of 19% and a CER of 46% for the baseline method. The accuracy of ToneLipNet was 79%, and the accuracy of the baseline was 43%. This resulted in a CER drop of 0.41 times that of the baseline and an accuracy increase of 1.8 times that of the baseline. This shows that ToneLipNet in this embodiment still has a clear advantage in the data actually input by the user, and the improvement brought about by the addition of tones still exists.

[0083] The training set consists of samples from 12 participants. This embodiment performs a 6-fold cross validation, each fold contains samples from two participants, and the participants in each fold are randomly assigned. Since each participant recorded 500 sentences in the corpus, ideally, each fold should contain 500 samples, but since some samples failed to be successfully input, the number of samples in each validation fold is slightly different, namely: 903, 917, 918, 923, 917, 920. The test results are shown in Table 2. ToneLipNet outperforms the baseline method in performance, with a character error rate (CER) of 28%, while the baseline method has a CER of 64%. The accuracy of ToneLipNet is 70%, and the accuracy of the baseline is 21%. This shows that after adding tone information, the CER drops to 0.43 times that of the baseline, while the accuracy increases to 3.3 times that of the baseline. Compared with the test results under normal conditions in 9.2.1, the performance of ToneLipNet and baseline both decreased when encountering unseen individuals. The CER of ToneLipNet increased by 9% compared with normal conditions, while the CER of baseline increased by 18% compared with normal conditions. The accuracy of ToneLipNet decreased by 9%, which was 0.88 times that under normal conditions. However, the accuracy of baseline decreased by 22% (about half), which was 0.48 times that under normal conditions. This shows that the performance of ToneLipNet of this embodiment is more stable when encountering unseen individuals.

[0084] This embodiment evaluates the processing ability of the ToneLipNet model for new sentences through 10-fold cross validation. Each fold contains 50 sentences from the corpus. Since there are 500 sentences in the corpus and each sentence is said by 12 users, ideally each fold should contain 600 samples (12 users × 50 sentences). But in fact, due to the missing of some samples, the number of samples in each validation fold is slightly different, namely 547, 541, 551, 555, 563, 563, 554, 544, 549, and 531. The test results are shown in Table 2. ToneLipNet outperforms the baseline method in performance, with a character error rate (CER) of 37% and a CER of 69% for the baseline method. The accuracy of ToneLipNet is 62%, and the accuracy of the baseline is 21%. This shows that after adding tone information, the CER drops to 0.53 times that of the baseline, and the accuracy increases by 2.9 times. Compared with the results under normal test conditions in 9.2.1, the performance of both models declined when encountering unseen sentences, but ToneLipNet performed more stably. The CER of ToneLipNet increased by 18% compared with normal conditions, while the CER of baseline increased by 23% compared with normal conditions. The accuracy of ToneLipNet decreased by 17%, which was 0.78 times that under normal conditions. The accuracy of baseline decreased by 22% (about half), which was 0.48 times that under normal conditions. This shows that the performance of ToneLipNet in this embodiment is more stable when encountering unseen sentences. The introduction of tones makes the recognition performance more robust under various test conditions.

[0085] Table 2

[0086]

[0087] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / sos> < / sos>

Claims

1. A Chinese lip reading recognition method combining tone information, characterized in that: include: Obtaining user input, wherein the user input includes tone information and mouth movement video; Preprocessing the user input to obtain preprocessed data; A multimodal Chinese lip-reading model is constructed, and the preprocessed data is input into the multimodal Chinese lip-reading model to obtain a lip reading recognition result.

2. The Chinese lip reading recognition method combined with tone information according to claim 1, characterized in that: The process of obtaining user input includes: The user wears an HTC VIVE head mounted display, and the user's thumb is placed on the touch pad of the handheld controller. The camera on the HTC VIVE head mounted display captures a mouth movement video. At the same time, the touch pad records the trajectory of the tone information drawn by the user, thereby obtaining the tone information and the mouth movement video. The picture frames in the mouth movement video correspond one to one with the trajectory coordinates in the tone information.

3. The Chinese lip reading recognition method combined with tone information according to claim 2, characterized in that: The process of preprocessing the user input includes preprocessing the tone information, wherein the process of preprocessing the tone information includes: converting the trajectory coordinates of the tone information into a bitmap image; Inputting the bitmap image into a convolutional neural network for convolution processing to obtain convolution features; The convolution feature is mapped to the tone classification result to obtain preprocessed tone information.

4. The Chinese lip reading recognition method combined with tone information according to claim 3, characterized in that: The process of preprocessing the user input includes preprocessing the mouth movement video, and the process of preprocessing the mouth movement video includes: Retrieving mouth key points in the mouth action video through Mediapipe; Calculating a mouth center based on the mouth key points; The video frame is cut into 96*96 pre-processed mouth images based on the mouth center.

5. The Chinese lip reading recognition method combined with tone information according to claim 4, characterized in that: The process of inputting the pre-processed data into the multimodal Chinese lip reading model to obtain the lip reading recognition result comprises: Extracting a video feature vector of a mouth action video based on the preprocessed mouth image; splicing the preprocessed tone information with the video feature vector to obtain a tone enhancement feature vector; Under the guidance of the tone sequence, the tone enhancement feature vector is used to enable the decoder to gradually obtain the lip reading recognition result.

6. The Chinese lip reading recognition method combined with tone information according to claim 5, characterized in that: The process of obtaining the video feature vector includes: Extracting spatial features of the mouth movement video based on ResNet-18; Based on the spatial features, a conformal encoder is used to perform temporal modeling, and the first convolution layer in the ResNet-18 is replaced with a 3D convolution layer to obtain compressed video features; A video feature vector is obtained based on the compressed video feature.

7. The Chinese lip reading recognition method in combination with tone information according to claim 5, characterized in that: The process of splicing the pre-processed tone information with the video feature vector includes: Mark the tone sequence to get the tone tag, and the interval between Chinese characters is marked as 0; Mapping the tone tags to spatial features to obtain continuous tone vectors; The continuous tone vectors are concatenated with the video feature vectors frame by frame to obtain a tone enhancement feature vector.

8. The Chinese lip reading recognition method in combination with tone information according to claim 1, characterized in that: Based on the Chinese lip reading recognition method, a lip reading recognition system combining tone information is developed; the lip reading recognition system includes a UI submodule, a data acquisition submodule, a data preprocessing submodule, and a lip reading recognition submodule; The UI submodule is used to prompt the user operation steps and display the recognition results; The data acquisition submodule is used to obtain mouth movement video and tone information; The data preprocessing submodule preprocesses the acquired mouth movement video and tone information; The lip reading recognition submodule is used for lip reading recognition to infer the text content that the user expects to input.

Images