Sentence-level lip reading recognition method, device and medium based on video word boundaries

Abstract

The present invention discloses a method, system, device and medium for sentence-level lip reading recognition based on video word boundaries, which belongs to the field of machine vision and natural language processing. The method includes the following steps: obtaining video data, obtaining a visual feature sequence based on the video data; wherein the length of the visual feature sequence is consistent with the length of the video data; inputting the visual feature sequence into a sentence-level lip reading recognition model using video word boundaries, outputting the predicted probability of each character corresponding to each frame in the video, and realizing sentence-level lip reading recognition. The present invention introduces video word boundary information into the sentence-level lip reading recognition model to improve the model's grasp of the video duration corresponding to each word in the video, thereby making the model more accurate in predicting the number of words in the sentence, thereby improving the accuracy of the sentence length recognized by the model.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision and natural language processing, and in particular to a method, system, device and medium for sentence-level lip reading recognition based on video word boundaries. Background Art

[0002] Currently, speech recognition technology is mainly used to identify the corresponding text from the speaker's voice. However, the effect of speech recognition is poor when the background environment of the speaker is noisy or the speaker is far away. Therefore, lip reading recognition technology based on the speaker's video is used to help identify the content of the speech. In recent years, the effect of lip reading recognition technology using only silent video has improved to a certain extent, but some problems still exist. In sentence-level lip reading recognition, a common phenomenon is that the length of the predicted sentence is inconsistent with the actual length of the sentence, such as Figure 1 This phenomenon will greatly reduce the accuracy of lip reading recognition results, because missing or redundant words will be considered as misrecognized words, and during the lip reading recognition process, some misrecognized words will affect the recognition results of other words in the context. Summary of the Invention

[0003] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, system, device and medium for sentence-level lip reading recognition based on video word boundaries.

[0004] The technical solution adopted in the present invention is:

[0005] A sentence-level lip reading recognition method based on video word boundaries includes the following steps:

[0006] Obtaining video data, and obtaining a visual feature sequence based on the video data; wherein the length of the visual feature sequence is consistent with the length of the video data;

[0007] The visual feature sequence is input into the sentence-level lip reading recognition model that uses video word boundaries, and the predicted probability of each character corresponding to each frame in the video is output to achieve sentence-level lip reading recognition.

[0008] Furthermore, the sentence-level lip reading recognition model includes a position encoding, a visual feature encoder, a video word boundary information embedding module, a visual feature decoder, a one-dimensional convolutional layer, and a softmax layer; wherein the video word boundary information embedding module utilizes the video word boundaries to enhance the encoding results to assist the decoding process;

[0009] Among them, the visual feature sequence is used as the input of the position encoding, and the output of the position encoding is used as the input of the visual feature encoder;

[0010] The intermediate representation output by the visual feature encoder is combined with the video word boundary information as the input of the video word boundary information embedding module. The video word boundary information embedding module outputs the intermediate representation fused with the video word boundary information as the input of the visual feature decoder;

[0011] The output of the visual feature decoder is used as the input of the one-dimensional convolutional layer, and the output of the one-dimensional convolutional layer is used as the input of the Softmax layer, which outputs the predicted probability of the character.

[0012] Furthermore, the video word boundary information is recorded in the form of word boundary time;

[0013] The video word boundary information embedding module performs the following processing on the video word boundary information:

[0014] Use sequence S et =(t1,…,t n ) represents the end time of each word in the video, that is, the word boundary time, where n is the number of words; the sequence S et Multiply the time in by the video frame rate to get the word boundary frame index sequence S bfi ,Right now:

[0015] S bfi =(i1,…,i n ),i k =t k ·f,t k ∈S et

[0016] Where f is the video frame rate;

[0017] Use word boundary frame indicator vector V b =(b1,…,b T ) to indicate whether each frame in the video is a word boundary frame, where T represents the number of frames in the video; the word boundary frame indicator vector V b Same length as the video frame sequence;

[0018] Let word boundary frame indicator vector V b The element value at the word boundary frame index is 1, and 0 elsewhere, that is:

[0019]

[0020] Word boundary frame indicator vector V b After the embedding layer, the word boundary frame indicator matrix M is formed b ,Right now:

[0021] M b =embedding(V b )

[0022] In this process, the embedding layer frames the word boundary indication vector V b Each element in is expanded to a d-dimensional vector, where d is the dimension of the intermediate representation vector corresponding to each frame;

[0023] Therefore, the word boundary frame indicator matrix M b The size of the visual feature encoder corresponds to the output of the intermediate representation matrix M of the entire video inter The size of the word boundary frame indicator matrix M b and the intermediate representation matrix M inter The values ​​of the corresponding positions are added to form the intermediate representation M of the fusion video word boundary information imb ,Right now:

[0024] M imb =M inter +M b .

[0025] Furthermore, the training process of the sentence-level lip reading recognition model using video word boundaries includes two stages:

[0026] In the first stage, the sentence-level lip reading recognition model using video word boundaries is pre-trained;

[0027] In the second stage, the pre-trained sentence-level lip reading recognition model using video word boundaries is used as the teacher model, and the similar model without the video word boundary information embedding module is used as the student model for training in the knowledge distillation stage; after the knowledge distillation stage training is completed, the student model that does not receive video word boundary information as input is used to perform the sentence-level lip reading recognition task.

[0028] Furthermore, in the first stage, the teacher model is pre-trained using the Connectionist Temporal Classification (CTC) loss;

[0029] In the second stage, the knowledge distillation stage, the pre-trained parameters of the teacher model remain unchanged, and the parameters of the student model are adjusted during the training process; during the knowledge distillation process, the teacher model and the student model receive the same visual feature sequence as input, and only the teacher model receives video word boundary information.

[0030] Furthermore, the training of the knowledge distillation stage includes:

[0031] For the input visual feature sequence, the teacher model calculates and outputs an intermediate representation that integrates the video word boundary information For the same visual feature sequence, the student model outputs the intermediate representation computed by the visual feature encoder

[0032] Compute intermediate representation and The L1 loss between them is used as the knowledge distillation loss L KD ; The student model also outputs the final character probability and calculates the connectionist temporal classification loss L CTC ; Add these two losses weightedly to get the total target loss L of the student model stu ,Right now:

[0033] L stu =L CTC +λL KD

[0034]

[0035] Among them, ‖·‖1 is the L1 norm; Representation matrix The i-th vector in corresponds to the i-th frame in the video, Representation matrix The i-th vector corresponding to the i-th frame in the video; d ′ is a vector and Dimensions;

[0036] During the training process, the total target loss L is minimized stu To optimize the student model.

[0037] Furthermore, the position encoding is implemented using a sine-cosine function method;

[0038] The visual feature encoder and the visual feature decoder are both composed of a Transformer encoder, which is composed of multiple feedforward Transformer blocks connected in series. Each feedforward Transformer block includes a multi-head self-attention layer and a feedforward neural network layer.

[0039] Furthermore, obtaining a visual feature sequence according to the video data includes:

[0040] Extracting features from the video data using a pre-trained visual front-end model to obtain a visual feature sequence consistent with the length of the video data;

[0041] The visual front-end model consists of a 3D convolutional neural network and a 2D ResNet neural network.

[0042] Another technical solution adopted in the present invention is:

[0043] A sentence-level lip reading recognition device based on video word boundaries, comprising:

[0044] at least one processor;

[0045] at least one memory for storing at least one program;

[0046] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0047] Another technical solution adopted in the present invention is:

[0048] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0049] The beneficial effect of the present invention is that the present invention introduces video word boundary information into the sentence-level lip reading recognition model, thereby improving the model's grasp of the video duration corresponding to each word in the video, thereby making the model more accurate in predicting the number of words in the sentence, thereby improving the accuracy of the sentence length recognized by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 is a schematic diagram of length differences in sentence-level lip reading recognition;

[0052] Figure 2 is a schematic diagram of the video word boundaries;

[0053] Figure 3 2 is a schematic diagram of the structure of a sentence-level lip reading recognition model using video word boundaries in an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a video word boundary information embedding module according to an embodiment of the present invention;

[0055] Figure 5 This is a flowchart of using a feature-based knowledge distillation strategy to train a sentence-level lip reading recognition model in an embodiment of the present invention;

[0056] Figure 6 This is a flowchart of the steps of a sentence-level lip reading recognition method based on video word boundaries in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0059] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0060] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0061] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0062] Explanation of terms:

[0063] Lip reading: Lip reading, also known as lip recognition, refers to the process of identifying spoken words from a video of a person's lip movements, where the video used for recognition does not contain sound. In daily life, lip reading is difficult for most people, and few people can master this skill, such as some people with hearing impairments who have undergone specialized training. Currently, lip reading recognition mainly utilizes artificial intelligence technology and deep learning models, involving multiple fields such as machine vision and natural language processing. In production and daily life, lip reading recognition technology can be used to assist hearing-impaired people in communication, restore sound from silent videos, assist public security investigations, and even military reconnaissance. The application scenarios of lip reading recognition technology are vast.

[0064] Sentence-level lip reading: Sentence-level lip reading is a type of lip reading recognition task that identifies an entire sentence from a silent video of lip movements corresponding to that sentence. The corresponding word-level lip reading task involves identifying a single word from a silent video of lip movements corresponding to only that word. Word-level lip reading is a multi-classification task, while sentence-level lip reading is a more complex sequence-to-sequence recognition task. Because sentence-level lip reading uses each character as the classification unit, the recognition of each character is also dependent on other characters in the context. Sentence-level lip reading recognition generally begins by cropping the speaker's facial video to the position of the lips. This is then fed into a visual front-end model to extract a visual feature sequence that matches the length of the video. This extracted visual feature sequence is then fed into a sequence-to-sequence lip reading recognition model to obtain the predicted probabilities for each character in each frame of the video. Based on these probabilities, the sentence with the highest overall probability is determined as the recognized sentence.

[0065] Video Word Boundary: Video word boundary refers to the video frame position of the lip movement video that corresponds to the word boundary in the sentence, such as Figure 2 In lip reading datasets, word boundary information in videos is usually represented by the start and end times of each word. Since the end time of one word in a sentence is the start time of the next word, this time is called the boundary time between words. Using the boundary time and the video frame rate, the position of the corresponding word boundary frame in the video can be calculated.

[0066] Knowledge Distillation: Knowledge distillation originally refers to the process of refining the knowledge in a collection of models into a single model to reduce the computational cost of the model and make it easier to deploy. The knowledge here refers to the process output or result output of the model calculation. The model used to transfer knowledge is called the teacher model, and the model that receives knowledge is called the student model. The result of knowledge distillation is a well-trained student model for performing tasks. Later studies also used larger models with more parameters and better performance as teacher models, and smaller models with worse performance as student models. Through the training method of knowledge distillation, the output of the student model is made closer to the teacher model, thereby improving the performance of the student model. Traditionally, knowledge distillation is the transfer of knowledge output from the last layer of the model from the teacher model to the student model. In recent years, knowledge distillation using the output of the middle layer of the model as knowledge has also emerged, which is called feature-based knowledge distillation.

[0067] Transformer model: The Transformer model is a neural network model for sequence-to-sequence tasks. Its key feature is the multi-head attention mechanism. This mechanism can capture the dependencies between components in a sequence, especially those separated by long distances. Furthermore, this model can parallelize sequence-to-sequence tasks, speeding up task processing.

[0068] Existing lip reading recognition models primarily focus on using more complex deep neural networks. However, these models suffer from the following shortcomings: 1) Existing sentence-level lip reading recognition models fail to consider the impact of text-based features on recognition results, neglecting factors such as sentence length and word segmentation that can reduce recognition accuracy. 2) Existing sentence-level lip reading recognition models employ a sequence-to-sequence processing approach, but pay insufficient attention to local information within words and lack a good grasp of the video duration corresponding to each word in the video. 3) The length of sentences predicted by current sentence-level lip reading recognition models differs significantly from the actual length of sentences, creating a bottleneck for improving sentence-level lip reading recognition accuracy. 4) Current lip reading recognition tasks utilize video word boundaries only at the word level, meaning they can only recognize one word at a time. This approach fails to account for situations where video word boundaries are unavailable in real-world environments, resulting in limited practical application. 5) Current knowledge distillation strategies for lip reading recognition focus on improving lip reading recognition accuracy through audio, but are not practical for a single video modality.

[0069] In order to alleviate the existing technical problems, the present invention introduces video word boundary information into the sentence-level lip reading recognition model. The present invention adds the embedded features of the video word boundary information to the neural network model that converts the visual feature sequence into the predicted probability of each character, so that the neural network model can grasp the video length corresponding to each word in the sentence more accurately, so that the model can more accurately predict the number of words, and thus more accurately predict the sentence length. On the other hand, in the process of using the model for lip reading recognition in a real environment, the video word boundary information is generally not available, so the present invention proposes a sentence-level lip reading recognition model training method that uses a knowledge distillation strategy to implicitly utilize video word boundaries. The lip reading recognition model trained using this method can predict sentences with more accurate lengths, thereby improving the overall content accuracy of lip reading recognition.

[0070] like Figure 6 As shown, this embodiment provides a sentence-level lip reading recognition method based on video word boundaries, including the following steps:

[0071] S1. Obtain video data and obtain a visual feature sequence based on the video data; wherein the length of the visual feature sequence is consistent with the length of the video data;

[0072] S2. Input the visual feature sequence into the sentence-level lip reading recognition model using the video word boundary, output the predicted probability of each character corresponding to each frame in the video, and realize sentence-level lip reading recognition.

[0073] In this embodiment, by introducing video word boundary information into the sentence-level lip reading recognition model, the model's understanding of the duration of each word in the video is improved, thereby making the model's prediction of the number of words in the sentence more accurate, thereby improving the accuracy of the sentence length recognized by the model. In addition, video word boundary information is used to enhance the model's attention to local information in the video, making the predicted sentence length more accurate while improving the recognition accuracy of local words. By improving the accuracy of sentence length prediction and local word recognition by the sentence-level lip reading recognition model, the overall accuracy of sentence-level lip reading recognition is improved.

[0074] The above method is explained in detail below with reference to the accompanying drawings.

[0075] (1) Sentence-level lip reading recognition model using video word boundaries

[0076] The input of the sentence-level lip reading recognition model using video word boundaries (hereinafter referred to as this model) proposed in this embodiment is a visual feature sequence consistent with the length of the video extracted by the visual front-end model, and the output is the predicted probability of each character corresponding to each frame of the video. The visual front-end model consists of a 3D convolution and a 2D ResNet neural network and has been pre-trained. The structure of this model includes position encoding, visual feature encoder, video word boundary information embedding module, visual feature decoder, one-dimensional convolution layer and Softmax layer. The overall structure of this model is as follows Figure 3 shown.

[0077] As an optional implementation, the position encoding in this model is implemented using the same method as that in the Transformer model using sine and cosine functions. Both the visual feature encoder and the visual feature decoder are composed of a Transformer encoder, which is composed of multiple feedforward Transformer blocks connected in series. Each feedforward Transformer block includes a multi-head self-attention layer and a feedforward neural network layer. Optionally, in the implementation of this model, the number of feedforward Transformer blocks is 6. The one-dimensional convolutional layer converts the 512-dimensional feature vector representation corresponding to each video frame output by the Transfomer encoder into a vector with the same dimension as the dictionary length, and then passes through the Softmax layer to obtain the probability of each character in each video frame.

[0078] This model is the first to incorporate video word boundary information into sentence-level lip reading. To incorporate video word boundaries when processing visual features, we place a video word boundary information embedding module between the traditional encoder and decoder architecture. This module leverages video word boundaries to enhance the encoding results and assist in the decoding process.

[0079] The video word boundary information embedding module combines the video word boundary information with the intermediate representation output by the visual feature encoder to form an intermediate representation that integrates the video word boundary information. Video word boundary information is usually recorded in the form of word boundary time. Therefore, we perform a series of transformations so that the video word boundary information can be added to the intermediate representation matrix output by the visual feature encoder. First, we use the sequence S et =(t1,…,t n ) represents the end time of each word in the video, i.e., the word boundary time, where n is the number of words. Then the sequence S et Multiply the time in by the video frame rate to get the word boundary frame index sequence S bfi ,Right now:

[0080] S bfi =(i1,…,i n ),i k=t k ·f,t k ∈S et

[0081] Where f is the video frame rate. We use the word boundary frame indicator vector V b =(b1,…,b T ) to indicate whether each frame in the video is a word boundary frame, where T represents the number of frames in the video. The word boundary frame indicator vector V b The same as the length of the video frame sequence. We assume that V b The element value at the word boundary frame index is 1, and 0 elsewhere, that is:

[0082]

[0083] Word boundary frame indicator vector V b After the embedding layer, the word boundary frame indicator matrix M is formed b ,Right now:

[0084] M b =embedding(V b )

[0085] During this process, the embedding layer converts V b Each element in M ​​is expanded into a d-dimensional vector, where d is the dimension of the intermediate representation vector corresponding to each frame. b The size of the visual feature encoder corresponds to the output of the intermediate representation matrix M of the entire video inter The size of M b and M inter The values ​​of the corresponding positions are added to form the intermediate representation M of the fusion video word boundary information imb ,Right now:

[0086] M imb =M inter +M b

[0087] The schematic diagram of the video word boundary information embedding module is as follows Figure 4 shown.

[0088] This model is trained using the training set data from the lip reading recognition dataset. The Connectionist Temporal Classification (CTC) loss is used as the loss function to guide the optimization of model parameters. The trained model can be used to perform sentence-level lip reading recognition tasks. The character probabilities output by the model can be used to generate the final predicted text through greedy search or beam search with a language model.

[0089] (2) Knowledge Distillation-Based Lip Reading Recognition Model Training Method that Implicitly Utilizes Video Word Boundary Information

[0090] In the process of sentence-level lip reading recognition in real situations, we generally cannot directly obtain the video word boundaries. In order to improve the accuracy of lip reading recognition in this case, the present invention proposes a two-stage lip reading recognition model training method based on knowledge distillation that implicitly utilizes video word boundary information. In the first stage, we first pre-train the sentence-level lip reading recognition model that utilizes video word boundaries mentioned above. In the second stage, we use the pre-trained sentence-level lip reading recognition model that utilizes video word boundaries as the teacher model, and a similar model without the video word boundary information embedding module as the student model for training in the knowledge distillation stage. After the training in the knowledge distillation stage is completed, we use the student model that does not receive video word boundary information as input to perform the sentence-level lip reading recognition task.

[0091] In the first stage of the method, we pre-train the teacher model using CTC loss. In the second stage, the knowledge distillation stage, the pre-trained parameters of the teacher model remain unchanged, and the parameters of the student model are adjusted during the training process. During the knowledge distillation process, the teacher model and the student model receive the same visual feature sequence as input, and only the teacher model receives the video word boundary information. We designed a knowledge distillation strategy that is more closely related to the video word boundary information. The schematic diagram of model training using this knowledge distillation strategy is shown in the figure below. Figure 5 shown.

[0092] In the process of model training using the knowledge distillation strategy, we use the intermediate representation of the fusion video word boundary information as knowledge. For the input visual feature sequence, the teacher model calculates and outputs the intermediate representation of the fusion video word boundary information For the same visual feature sequence, the student model outputs the intermediate representation computed by the visual feature encoder We calculate and The L1 loss between them is used as the knowledge distillation loss L KD The student model also outputs the final character probability and calculates the CTC loss L CTC The weighted sum of these two losses gives the total target loss L of the student model. stu ,Right now:

[0093] L stu =L CTC +λL KD

[0094]

[0095] where ‖·‖1 is the L1 norm. Representation matrix The i-th vector in corresponds to the i-th frame in the video, Representation matrix The i-th vector in d corresponds to the i-th frame in the video. ′ is a vector and During training, the total target loss L is minimized. stu To optimize the student model, this knowledge distillation strategy aims to make the output of the visual feature encoder in the student model close to the output of the video word boundary information embedding module in the teacher model. This allows the student model's visual feature encoder to implicitly learn the video word boundary information, and its output can help the visual feature decoder perform the decoding process.

[0096] After the knowledge distillation-based lip reading recognition model is trained, we use the student model to perform sentence-level lip reading recognition. For the input video, we first use the pre-trained visual front-end model to extract the visual feature sequence. This visual feature sequence is then fed into the trained student model to calculate character probabilities. The character probabilities are then used to generate the final predicted text using greedy search or beam search with a language model.

[0097] In summary, the method of the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0098] 1) The present invention introduces video word boundary information into the sentence-level lip reading recognition model to improve the model's understanding of the video duration corresponding to each word in the video, thereby making the model more accurate in predicting the number of words in a sentence, thereby improving the accuracy of the sentence length recognized by the model.

[0099] 2) The present invention utilizes the boundary information of video words to enhance the model's attention to local information of the video, thereby making the predicted sentence length more accurate while improving the recognition accuracy of local vocabulary.

[0100] 3) The present invention improves the overall accuracy of sentence-level lip reading recognition by improving the accuracy of sentence length predicted by the sentence-level lip reading recognition model and the recognition accuracy of local vocabulary.

[0101] 4) In order to improve the accuracy of lip reading recognition in a real environment where video word boundaries are unavailable, the present invention proposes a method for training a sentence-level lip reading recognition model using a knowledge distillation strategy. The trained model can implicitly use video word boundaries to improve the length accuracy and overall content accuracy of the sentences predicted by the model without video word boundaries as input.

[0102] This embodiment further provides a sentence-level lip reading recognition device based on video word boundaries, comprising:

[0103] at least one processor;

[0104] at least one memory for storing at least one program;

[0105] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 6 The method shown.

[0106] A sentence-level lip reading recognition device based on video word boundaries in this embodiment can execute a sentence-level lip reading recognition method based on video word boundaries provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0107] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 6 The method shown.

[0108] This embodiment also provides a storage medium storing instructions or programs that can execute a sentence-level lip reading recognition method based on video word boundaries provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0109] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0110] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0112] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0113] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0114] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0115] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0116] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0117] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A sentence-level lip reading recognition method based on video word boundaries, characterized by: The following steps are involved: Acquire video data, and obtain a visual feature sequence according to the video data; The visual feature sequence is input into a sentence-level lip reading recognition model that uses video word boundaries. The model outputs the predicted probability of each character in each frame of the video, thus realizing sentence-level lip reading recognition. The sentence-level lip reading recognition model includes a positional encoder, a visual feature encoder, a video word boundary information embedding module, a visual feature decoder, a one-dimensional convolutional layer, and a softmax layer; wherein the video word boundary information embedding module utilizes the video word boundaries to enhance the encoding results to assist the decoding process; The visual feature sequence is used as the input of the position encoding, and the output of the position encoding is used as the input of the visual feature encoder. The intermediate representation output by the visual feature encoder is combined with the video word boundary information as the input of the video word boundary information embedding module. The video word boundary information embedding module outputs the intermediate representation fused with the video word boundary information as the input of the visual feature decoder. The output of the visual feature decoder is used as the input of the one-dimensional convolution layer, the output of the one-dimensional convolution layer is used as the input of the Softmax layer, and the Softmax layer outputs the predicted probability of the character; The video word boundary information is recorded in the form of word boundary time; The video word boundary information embedding module performs the following processing on the video word boundary information: Use sequence S et =(t1,…,t n ) represents the end time of each word in the video, that is, the word boundary time, where n is the number of words; the sequence S et Multiply the time in by the video frame rate to get the word boundary frame index sequence S bfi ,Right now: S bfi =(i1,…,i n ),i k =t k ·f,1≤k≤n,t k ∈S et Where f is the video frame rate; Use word boundary frame indicator vector V b =(b1,…,b T ) to indicate whether each frame in the video is a word boundary frame, where T represents the number of frames in the video; the word boundary frame indicator vector V b Same length as the video frame sequence; Word boundary frame indicator vector V b The element value at the word boundary frame index is 1, and 0 elsewhere, that is: Word boundary frame indicator vector V b After the embedding layer, the word boundary frame indicator matrix M is formed b ,Right now: M b =embedding(V b ) In this process, the embedding layer frames the word boundary indication vector V b Each element in is expanded to a d-dimensional vector, where d is the dimension of the intermediate representation vector corresponding to each frame; Word boundary frame indicator matrix M b The size of the visual feature encoder corresponds to the output of the intermediate representation matrix M of the entire video inter The size of the word boundary frame indicator matrix M b and the intermediate representation matrix M inter The values ​​of the corresponding positions are added to form the intermediate representation M of the fusion video word boundary information imb ,Right now: M imb =M inter +M b 。 2. The sentence-level lip reading recognition method based on video word boundaries according to claim 1 is characterized in that: The training process of the sentence-level lip reading recognition model using video word boundaries includes two stages: In the first stage, the sentence-level lip reading recognition model using video word boundaries is pre-trained; In the second stage, the pre-trained sentence-level lip reading recognition model using video word boundaries is used as the teacher model, and the similar model without the video word boundary information embedding module is used as the student model for training in the knowledge distillation stage; after the knowledge distillation stage training is completed, the student model that does not receive video word boundary information as input is used to perform the sentence-level lip reading recognition task.

3. The sentence-level lip reading recognition method based on video word boundaries according to claim 2 is characterized in that: In the first stage, the teacher model is pre-trained using the connectionist temporal classification loss; In the second stage, the pre-trained parameters of the teacher model remain unchanged, and the parameters of the student model are adjusted during the training process; during the knowledge distillation process, the teacher model and the student model receive the same visual feature sequence as input, and only the teacher model receives video word boundary information.

4. The sentence-level lip reading recognition method based on video word boundaries according to claim 2, characterized in that: The training of the knowledge distillation stage includes: For the input visual feature sequence, the teacher model calculates and outputs an intermediate representation that integrates the video word boundary information For the same visual feature sequence, the student model outputs the intermediate representation computed by the visual feature encoder Compute intermediate representation and The L1 loss between them is used as the knowledge distillation loss L KD ; The student model also outputs the final character probability and calculates the connectionist temporal classification loss L CTC ; Add these two losses weightedly to get the total target loss L of the student model stu ,Right now: THE stu =L CTC +λL KD Among them, ‖·‖1 is the L1 norm; Representation matrix The i-th vector in corresponds to the i-th frame in the video, Representation matrix The i-th vector in the video corresponding to the i-th frame; d′ is the vector and Dimensions; During the training process, the total target loss L is minimized stu To optimize the student model.

5. The sentence-level lip reading recognition method based on video word boundaries according to claim 1 is characterized in that: The position coding is achieved by using a sine-cosine function method; The visual feature encoder and the visual feature decoder are both composed of a Transformer encoder, which is composed of multiple feedforward Transformer blocks connected in series. Each feedforward Transformer block includes a multi-head self-attention layer and a feedforward neural network layer.

6. The sentence-level lip reading recognition method based on video word boundaries according to claim 1 is characterized in that: The step of obtaining a visual feature sequence according to the video data includes: Extracting features from the video data using a pre-trained visual front-end model to obtain a visual feature sequence consistent with the length of the video data; The visual front-end model consists of a 3D convolutional neural network and a 2D ResNet neural network.

7. A sentence-level lip reading recognition device based on video word boundaries, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images