Lip reading recognition method, device, equipment, medium and product

Abstract

The present application discloses a lip reading recognition method, apparatus, equipment, medium and product, which relates to the intersection of machine vision and natural language processing. The method includes obtaining the original video to be recognized, initializing the text sequence, cutting the original video to obtain multiple frame images, and extracting the intra-frame features and inter-frame features of each frame image. Video features are obtained based on the inter-frame features of all frame images. Based on the video features and the text sequence, a lip reading recognition model is used to obtain lip reading recognition results. The lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set. The present application obtains video features by obtaining the intra-frame features and inter-frame features of each frame image, and further improves the feature expression ability and discrimination of the model by performing multimodal feature fusion of the text sequence and the video features, and further improves the accuracy of lip reading recognition.

Description

Technical Field

[0001] The present application relates to the intersection of machine vision and natural language processing, and in particular to a lip reading recognition method, apparatus, device, medium, and product. Background Art

[0002] Lip reading recognition translates a sequence of video frames of a speaker's lips into text. Lip reading recognition technology has broad application prospects. It can be used in barrier-free communication systems for people who have lost their voices due to vocal cord or larynx damage. It can also assist in collecting user language information, such as interpreting information from silent videos. Furthermore, lip reading recognition can be combined with other biometric technologies to enhance identification security.

[0003] In the lip reading recognition process, there is not a unique correspondence between mouth shape and pronunciation, or pronunciation and text. There are often multiple possible alternative results, and the most likely result needs to be calculated in real time. Due to the stress of Chinese and the lack of feature points in the lip area, the feature representation and discrimination of existing lip reading recognition systems are relatively poor. Although existing lip reading recognition methods have achieved remarkable results in the recognition of some word-level datasets, they still have common defects: sequence-level lip reading recognition is lacking, and most lip reading recognition models still remain at the word-level classification task, without considering homophones from the perspective of sequence and context; the degree of modeling of video and text sequences is insufficient, and there is insufficient attention to the inter-frame and intra-frame of video sequences, and there is no combination of text context and multimodal data of video frames for learning. Most of the work is two-stage and not end-to-end. This results in the poor feature representation and discrimination of existing technologies, and the low accuracy of lip reading recognition. Summary of the Invention

[0004] The purpose of this application is to provide a lip reading recognition method, device, equipment, medium and product, which can obtain video features by acquiring intra-frame features and inter-frame features of each frame image, and further improve the feature expression ability and discrimination of the model by multimodal feature fusion of text sequence and video features, thereby improving the accuracy of lip reading recognition.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a lip reading recognition method, comprising:

[0007] Get the original video to be recognized and initialize the text sequence;

[0008] Cutting the original video to obtain a plurality of frame images; each frame image includes a lip region of the speaker;

[0009] Extracting intra-frame features of each frame image, and extracting inter-frame features of each frame image based on the intra-frame features of all frame images;

[0010] Obtain video features based on inter-frame features of all frame images;

[0011] Based on the video features and the text sequence, a lip reading recognition model is used to obtain a lip reading recognition result; the lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set; each training sample in the training sample set includes a sample video feature, a sample text sequence and a target lip reading recognition result.

[0012] Optionally, extracting intra-frame features of each frame image, and extracting inter-frame features of each frame image based on the intra-frame features of all frame images, specifically includes:

[0013] For any frame image, the frame image is segmented to obtain multiple embedding vectors;

[0014] Performing spatial position encoding on each embedded vector to obtain multiple spatial encoding vectors of the frame image;

[0015] Extracting intra-frame features of the frame image using a multi-head self-attention mechanism based on all spatial encoding vectors of the frame image;

[0016] Performing temporal position coding on intra-frame features of the frame image to obtain a temporal coding vector of the frame image;

[0017] According to the time coding vector of all frame images, a multi-head self-attention mechanism is used to extract the inter-frame features of each frame image.

[0018] Optionally, the method for building the lip reading recognition model includes:

[0019] Obtain multiple sample videos and the target lip reading recognition results corresponding to each sample video, and initialize the sample text sequence corresponding to each sample video;

[0020] For any sample video, the sample video is cut to obtain a plurality of sample frame images; each sample frame image includes a lip area of a speaker;

[0021] Extracting intra-frame features of each sample frame image of the sample video, and extracting inter-frame features of each sample frame image of the sample video based on the intra-frame features of all sample frame images of the sample video;

[0022] Obtaining sample video features of the sample video according to inter-frame features of all sample frame images of the sample video;

[0023] A training sample set is obtained based on the sample video features, sample text sequence, and target lip reading recognition results corresponding to each sample video;

[0024] The training sample set is used to train the neural network multimodal fusion model to obtain the lip reading recognition model; the lip reading recognition model is the trained neural network multimodal fusion model.

[0025] Optionally, the text sequence includes a start character and an end character, and the neural network multimodal fusion model includes a deep learning algorithm and a multimodal feature fusion method;

[0026] According to the video features and the text sequence, a lip reading recognition model is used to obtain a lip reading recognition result, which specifically includes:

[0027] For the i-th iteration, based on the first i labels of the text sequence, a deep learning algorithm is used to obtain the text features corresponding to the first i labels; i>0; the first label of the text sequence is the start character;

[0028] Based on the text features corresponding to the first i labels and the video features, a multimodal feature fusion method is used to obtain the i-th predicted word, and the i-th predicted word is used as the i+1-th label of the text sequence; if the i-th predicted word is the terminal character, then the 2nd to i-th labels of the text sequence are lip reading recognition results, otherwise, the i+1-th iteration is performed.

[0029] Optionally, for the i-th iteration, based on the first i labels of the text sequence, a deep learning algorithm is used to obtain text features corresponding to the first i labels, specifically including:

[0030] Determine the word vector corresponding to each label in the first i labels respectively;

[0031] The word vector corresponding to each of the first i labels is positionally encoded, and a masked multi-head self-attention mechanism is used to obtain the text features corresponding to the first i labels.

[0032] Optionally, a multimodal feature fusion method is used to obtain an i-th predicted word element based on the text features corresponding to the first i labels and the video features, and the i-th predicted word element is used as the i+1-th label of the text sequence, specifically including:

[0033] According to the text features corresponding to the first i tags and the video features, a cross-attention mechanism is used to obtain fusion features;

[0034] According to the fusion features, a feedforward neural network is used to obtain enhanced features;

[0035] According to the enhanced features, an i-th predicted word is obtained, and the i-th predicted word is used as the i+1-th label of the text sequence.

[0036] In a second aspect, the present application provides a lip reading recognition device, which is applied to the above-mentioned lip reading recognition method, comprising:

[0037] The data acquisition module is used to obtain the original video to be recognized and initialize the text sequence;

[0038] A video cutting module is used to cut the original video to obtain multiple frame images; each frame image contains the speaker's lip area;

[0039] An image feature extraction module is used to extract intra-frame features of each frame image, and extract inter-frame features of each frame image based on the intra-frame features of all frame images;

[0040] The video feature extraction module is used to obtain video features based on the inter-frame features of all frame images;

[0041] The lip reading recognition module is used to obtain lip reading recognition results based on the video features and the text sequence using a lip reading recognition model; the lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set; each training sample in the training sample set includes a sample video feature, a sample text sequence, and a target lip reading recognition result.

[0042] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned lip reading recognition method.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned lip reading recognition method when executed by a processor.

[0044] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned lip reading recognition method when executed by a processor.

[0045] According to the specific embodiments provided in this application, this application has the following technical effects:

[0046] The present application provides a lip reading recognition method, apparatus, device, medium and product. By acquiring intra-frame features and inter-frame features of each frame image, video features are obtained, thereby improving the feature expression capability and discrimination of the lip reading recognition method; a multimodal feature fusion method is adopted to perform multimodal feature fusion on text sequences and video features for lip reading recognition, thereby improving the accuracy of lip reading recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 This is a diagram of an application environment of a lip reading recognition method in one embodiment of the present application.

[0049] Figure 2 A flowchart of a lip reading recognition method provided in one embodiment of the present application is provided.

[0050] Figure 3 A schematic diagram of the overall process of implementing lip reading recognition with the help of a Transformer encoder according to one embodiment of the present application.

[0051] Figure 4 for Figure 2 Detailed flowchart of step 206 in FIG.

[0052] Figure 5 A schematic diagram of the functional modules of a lip reading recognition device provided in one embodiment of the present application.

[0053] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] This application proposes a lip reading recognition method, device, equipment, medium and product. By acquiring the intra-frame features and inter-frame features of each frame image, video features are obtained, thereby improving the feature expression ability and discrimination of the lip reading recognition method; a multimodal feature fusion method is adopted to perform multimodal feature fusion on text sequences and video features for lip reading recognition, thereby improving the accuracy of lip reading recognition.

[0056] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] The lip reading recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. , the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the original video to be recognized to the server 104. After receiving the original video, the server 104 cuts the original video to obtain multiple frame images. For any frame image, the intra-frame features and inter-frame features are extracted. Based on the inter-frame features of all frame images, the video features are obtained. Finally, based on the text sequence and video features, a multimodal feature fusion method is used to obtain the lip reading recognition results. The server 104 can feed back the obtained lip reading recognition results to the terminal 102. In addition, in some embodiments, the lip reading recognition method can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform lip reading recognition on the original video, or the server 104 can obtain the original video from the data storage system and perform lip reading recognition on the original video.

[0058] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0059] In an exemplary embodiment, Figure 2 As shown, a lip reading recognition method is provided. The method is executed by a computer device. Specifically, it can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, including the following steps 201 to 205.

[0060] In step 201, the original video to be recognized is obtained and a text sequence is initialized.

[0061] Step 202: Cut the original video to obtain multiple frame images, each of which contains the speaker's lip area.

[0062] Step 203: extract intra-frame features of each frame image, and extract inter-frame features of each frame image based on the intra-frame features of all frame images.

[0063] Step 204: Obtain video features based on inter-frame features of all frame images.

[0064] In step 205, a lip reading recognition model is used based on the video features and text sequence to obtain a lip reading recognition result. The lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set. Each training sample in the training sample set includes sample video features, a sample text sequence, and the target lip reading recognition result.

[0065] In an exemplary embodiment, the method for building a lip reading recognition model includes the following steps.

[0066] (1) Obtain multiple sample videos and the target lip reading recognition results corresponding to each sample video, and initialize the sample text sequence corresponding to each sample video.

[0067] (2) For any sample video, the sample video is cut to obtain multiple sample frame images. Each sample frame image contains the speaker's lip area.

[0068] (3) Extracting intra-frame features of each sample frame image of the sample video, and extracting inter-frame features of each sample frame image of the sample video based on the intra-frame features of all sample frame images of the sample video.

[0069] (4) According to the inter-frame features of all sample frame images of the sample video, the sample video features of the sample video are obtained.

[0070] (5) A training sample set is obtained based on the sample video features, sample text sequence and target lip reading recognition results corresponding to each sample video.

[0071] (6) Using the training sample set, the neural network multimodal fusion model is trained to obtain a lip reading recognition model. The lip reading recognition model is the trained neural network multimodal fusion model.

[0072] In this embodiment, the training effect of the neural network multimodal fusion model is evaluated and optimized by calculating the cross entropy loss and edit distance loss between the predicted sample text sequence and the target lip reading recognition result.

[0073] In an exemplary embodiment, Figure 2 Before step 203, each frame image is cropped based on facial landmarks to obtain a lip region image to reduce interference from invalid pixels. The lip region image is filtered using a Gaussian-Adaptive Bilateral Filter (GABF) and resized for subsequent image segmentation. In this embodiment, all lip region images are resized to 64×64 pixels.

[0074] In an exemplary embodiment, Figure 2 Step 203 in the process can be replaced by the following steps 2031 to 2035.

[0075] Step 2031: For any frame image, the frame image is segmented to obtain multiple embedding vectors. In this embodiment, the Frame Embedding module is used to segment the original video to be recognized into multiple frame images. The lip region is extracted from the frame images, and the lip region image is filtered and size normalized. Finally, the processed 64×64 pixel lip region image is evenly segmented into four parts to obtain four embedding vectors. The four embedding vectors are flattened into a one-dimensional sequence and mapped to a low-dimensional feature space through a linear layer.

[0076] Step 2032: Perform spatial position encoding on each embedded vector to obtain multiple spatial encoding vectors of the frame image.

[0077] Step 2033: Based on all spatial coding vectors of the frame image, a multi-head self-attention mechanism is used to extract the intra-frame features of the frame image.

[0078] Step 2034: perform temporal position coding on the intra-frame features of the frame image to obtain a temporal coding vector of the frame image.

[0079] Step 2035: Based on the time coding vectors of all frame images, a multi-head self-attention mechanism is used to extract the inter-frame features of each frame image.

[0080] In this embodiment, the Transformer encoder is used to extract video features and text features, and feature fusion is performed to achieve lip reading recognition. The process of using the Transformer encoder to implement the above operation is as follows Figure 3 As shown. When extracting video features, all embedding vectors of all frame images are input into the Transformer encoder. Taking the frame image as the unit, a learnable identifier "cls" is added to the zeroth position of each frame image. The spatial position of each embedding vector of each frame image is encoded. By adding each embedding vector of each frame image with the learnable spatial position information, a sequence with spatial position information is formed, which helps the Transformer encoder understand the spatial relationship and sequence order within each frame image. A multi-head self-attention mechanism is used to perform spatial attention calculations, calculate the relationship between each position within each frame image, and thus obtain the spatial feature representation of each frame image, and extract the intra-frame features. The intra-frame features are lip shape features.

[0081] In this embodiment, if Figure 3As shown in the figure, temporal attention is calculated based on the results of spatial attention. A learnable identifier "cls" is added to the intra-frame features of each frame image. The temporal position of each intra-frame feature is encoded. The temporal attention module is used to calculate the temporal relationship between different frames, thereby obtaining the inter-frame features of the original video. The inter-frame features represent the temporal sequence of lip movements.

[0082] Based on the results of temporal attention calculations, namely inter-frame features, a feedforward neural network is used to perform feature fusion to generate video features. The feedforward neural network fuses intra-frame and inter-frame features after each layer, optimizes the feature representation through layer normalization and activation functions, and ultimately generates video features for lip reading recognition.

[0083] Text feature extraction is performed in parallel with video feature extraction. Based on the trained neural network multimodal fusion model, lip reading recognition results can be obtained. In an exemplary embodiment, the text sequence includes a start character and an end character. The neural network multimodal fusion model includes a deep learning algorithm and a multimodal feature fusion method. Figure 4 As shown, Figure 2 Step 205 in the process may be replaced by the following steps 401 and 402.

[0084] Step 401: For the i-th iteration, based on the first i labels of the text sequence, a deep learning algorithm is used to obtain the text features corresponding to the first i labels. Where i>0, the first label of the text sequence is the start character.

[0085] In step 402, a multimodal feature fusion method is used to obtain the i-th predicted word-unit based on the text features and video features corresponding to the first i labels. This predicted word-unit is used as the i+1-th label of the text sequence. If the i-th predicted word-unit is a terminal character, the second through i-th labels of the text sequence are the lip reading recognition results. Otherwise, the i+1-th iteration is performed.

[0086] In an exemplary embodiment, Figure 4 Step 401 in the process may be replaced by the following steps 4011 and 4012.

[0087] Step 4011, determine the word vector corresponding to each tag in the first i tags respectively.

[0088] In step 4012, the word vector corresponding to each of the first i tags is positionally encoded, and a masked multi-head self-attention mechanism is used to obtain the text features corresponding to the first i tags.

[0089] In an exemplary embodiment, Figure 4 Step 402 in the process may be replaced by the following steps 4021 to 4023.

[0090] In step 4021, a cross-attention mechanism is used to obtain fusion features based on the text features and video features corresponding to the first i tags.

[0091] In step 4022, a feedforward neural network is used to obtain enhanced features based on the fused features.

[0092] In step 4023, the i-th predicted word is obtained based on the enhanced features, and the i-th predicted word is used as the i+1-th label of the text sequence.

[0093] In this embodiment, if Figure 3 As shown in the figure, word vector embedding is performed based on the text sequence. Xavier is used for weight initialization, and the embedding weights are automatically updated through gradient and backpropagation. The word vector corresponding to each label in the text sequence is positionally encoded. Adding the word vector to the learnable positional encoding enables the Transformer decoder to consider the order of words in the text data. The positionally encoded word vectors are trained using a masked multi-head attention mechanism to obtain text features. Specifically, the Transformer decoder's self-attention mechanism captures contextual information by calculating the relationship between each position in the text sequence. Text features are the sequential and semantic relationships of the text.

[0094] Based on the text features and video features obtained through parallel processing of text and video, a cross-attention calculation is performed to generate fused features. This cross-attention calculation simultaneously focuses on the spatiotemporal characteristics of the video and the contextual relationships of the text, fusing features from both the video and text modalities and enhancing the multimodal understanding capabilities of the lip reading recognition method. The fused features are then input into a feedforward neural network for feature enhancement, resulting in enhanced features. The enhanced features are then decoded in the Transformer decoder to obtain the final embedding vector. This embedding vector is further processed and converted into predicted word units, resulting in the final lip reading recognition result.

[0095] In an exemplary embodiment, based on the trained neural network multimodal fusion model, the autoregressive mechanism is used to sequentially obtain predicted word units, and the obtained word units are sequentially filled into the initialized text sequence. Figure 3 As shown, the text sequence after initialization includes a start character and a stop character. The start character is located at the first label position of the text sequence, and the stop character is located at the last label position of the text sequence.

[0096] The lip reading recognition method of the present application obtains the video features at one time based on the original video to be recognized. For the text part, based on the trained neural network multimodal fusion model, the text features corresponding to each predicted word are obtained in sequence through iteration. In the first iteration, the text features corresponding to the first label are extracted, that is, the text features corresponding to the start character. According to the video features and the text features corresponding to the start character, the lip reading recognition model is used to predict the first predicted word and use it as the second label of the text sequence. In the second iteration, according to the first two labels of the text sequence, that is, the start character and the first predicted word, the text features corresponding to the two are obtained. Further combined with the video features, the lip reading recognition model is used to predict the second predicted word and use it as the third label of the text sequence, and so on. When iterating to the last label of the text sequence, that is, the end character, it is detected that the predicted word is the end character, the loop ends, and the label sequence of the text sequence excluding the start character and the end character is used as the final lip reading recognition result.

[0097] Based on the same inventive concept, embodiments of the present application also provide a lip reading recognition device for implementing the aforementioned lip reading recognition method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more lip reading recognition device embodiments provided below can be found in the above-described limitations of the lip reading recognition method and will not be further elaborated here.

[0098] In an exemplary embodiment, Figure 5 As shown, a lip reading recognition device is provided, comprising: a data acquisition module 501, a video segmentation module 502, an image feature extraction module 503, a video feature extraction module 504, and a lip reading recognition module 505. The data acquisition module 501 is used to acquire the original video to be recognized and initialize the text sequence. The video segmentation module 502 is used to segment the original video to obtain multiple frame images. Each frame image contains the speaker's lip area. The image feature extraction module 503 is used to extract intra-frame features of each frame image and, based on the intra-frame features of all frame images, extract inter-frame features of each frame image. The video feature extraction module 504 is used to obtain video features based on the inter-frame features of all frame images. The lip reading recognition module 505 is used to obtain lip reading recognition results using a lip reading recognition model based on the video features and the text sequence. The lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set. Each training sample in the training sample set includes sample video features, a sample text sequence, and the target lip reading recognition result.

[0099] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store original video. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a lip reading recognition method is implemented.

[0100] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0101] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0102] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0103] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0104] The beneficial effects of this application are as follows:

[0105] This application successfully captures the spatiotemporal relationships within and outside the video frames and the interconnections between text contexts by performing feature extraction on the original video and text sequences in parallel, achieving effective fusion of multimodal features. Even in the absence of audio modality input, it can still maintain a high recognition capability and effectively reduce the character error rate.

[0106] The lip reading recognition method provided in this application addresses the problems of Chinese accent and insufficient feature points in the lip area in existing lip reading recognition systems, proposing a practical and effective solution and significantly improving the model's feature representation and discrimination. By cropping the lip area using facial key points, the interference of invalid pixels is reduced. The spatiotemporal attention mechanism is used to extract inter-frame and intra-frame features, and cross-attention is calculated by combining video features with text features, demonstrating superior performance that exceeds traditional convolution.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0109] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0110] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A lip reading recognition method, characterized in that: The lip reading recognition method comprises: Get the original video to be recognized and initialize the text sequence; Cutting the original video to obtain a plurality of frame images; each frame image includes a lip region of the speaker; Extracting intra-frame features of each frame image, and extracting inter-frame features of each frame image based on the intra-frame features of all frame images; Obtain video features based on inter-frame features of all frame images; A lip reading recognition model is used to obtain a lip reading recognition result based on the video features and the text sequence; the lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set; each training sample in the training sample set includes a sample video feature, a sample text sequence, and a target lip reading recognition result; The text sequence includes a start character and an end character, the end character is located at the last label position of the text sequence, and the neural network multimodal fusion model includes a deep learning algorithm and a multimodal feature fusion method; According to the video features and the text sequence, a lip reading recognition model is used to obtain a lip reading recognition result, which specifically includes: For the i-th iteration, based on the first i labels of the text sequence, a deep learning algorithm is used to obtain the text features corresponding to the first i labels; i>0; the first label of the text sequence is the start character; Based on the text features corresponding to the first i labels and the video features, a multimodal feature fusion method is used to obtain the i-th predicted word, and the i-th predicted word is used as the i+1-th label of the text sequence; if the i-th predicted word is the terminal character, then the 2nd to i-th labels of the text sequence are lip reading recognition results, otherwise, the i+1-th iteration is performed.

2. The lip reading recognition method according to claim 1, wherein: Extracting intra-frame features of each frame image, and extracting inter-frame features of each frame image based on the intra-frame features of all frame images, specifically including: For any frame image, the frame image is segmented to obtain multiple embedding vectors; Performing spatial position encoding on each embedded vector to obtain multiple spatial encoding vectors of the frame image; Extracting intra-frame features of the frame image using a multi-head self-attention mechanism based on all spatial encoding vectors of the frame image; Performing temporal position coding on intra-frame features of the frame image to obtain a temporal coding vector of the frame image; According to the time coding vector of all frame images, a multi-head self-attention mechanism is used to extract the inter-frame features of each frame image.

3. The lip reading recognition method according to claim 1, wherein: The method for building the lip reading recognition model includes: Obtain multiple sample videos and the target lip reading recognition results corresponding to each sample video, and initialize the sample text sequence corresponding to each sample video; For any sample video, the sample video is cut to obtain a plurality of sample frame images; each sample frame image includes a lip area of a speaker; Extracting intra-frame features of each sample frame image of the sample video, and extracting inter-frame features of each sample frame image of the sample video based on the intra-frame features of all sample frame images of the sample video; Obtaining sample video features of the sample video according to inter-frame features of all sample frame images of the sample video; A training sample set is obtained based on the sample video features, sample text sequence, and target lip reading recognition results corresponding to each sample video; The training sample set is used to train the neural network multimodal fusion model to obtain the lip reading recognition model; the lip reading recognition model is the trained neural network multimodal fusion model.

4. The lip reading recognition method according to claim 1, wherein: For the i-th iteration, based on the first i labels of the text sequence, a deep learning algorithm is used to obtain the text features corresponding to the first i labels, specifically including: Determine the word vector corresponding to each label in the first i labels respectively; The word vector corresponding to each of the first i labels is positionally encoded, and a masked multi-head self-attention mechanism is used to obtain the text features corresponding to the first i labels.

5. The lip reading recognition method according to claim 1, wherein: According to the text features corresponding to the first i labels and the video features, a multimodal feature fusion method is used to obtain the i-th predicted word, and the i-th predicted word is used as the i+1-th label of the text sequence, specifically including: According to the text features corresponding to the first i tags and the video features, a cross-attention mechanism is used to obtain fusion features; According to the fusion features, a feedforward neural network is used to obtain enhanced features; According to the enhanced features, an i-th predicted word is obtained, and the i-th predicted word is used as the i+1-th label of the text sequence.

6. A lip reading recognition device, applied to the lip reading recognition method according to any one of claims 1 to 5, characterized in that: The lip reading recognition device comprises: The data acquisition module is used to obtain the original video to be recognized and initialize the text sequence; A video cutting module is used to cut the original video to obtain multiple frame images; each frame image contains the speaker's lip area; An image feature extraction module is used to extract intra-frame features of each frame image, and extract inter-frame features of each frame image based on the intra-frame features of all frame images; The video feature extraction module is used to obtain video features based on the inter-frame features of all frame images; The lip reading recognition module is used to obtain lip reading recognition results based on the video features and the text sequence using a lip reading recognition model; the lip reading recognition model is a neural network multimodal fusion model pre-built based on a training sample set; each training sample in the training sample set includes a sample video feature, a sample text sequence, and a target lip reading recognition result.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lip reading recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lip reading recognition method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the lip reading recognition method according to any one of claims 1 to 5 is implemented.

Images