Lip reading method and related device
By dividing the input channels of the convolutional layer and performing shuffling in lip reading recognition, the problem of high computational complexity of lightweight convolutional neural networks is solved, thereby optimizing computing resources and improving recognition efficiency.
Patent Information
- Application Number
- CN202211191003.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing lightweight convolutional neural network models have high computational complexity in lip reading, resulting in high computational resource consumption and low recognition efficiency.
Using a pre-defined neural network model, the input channels of the convolutional layer are divided into first and second input channels. Convolution processing is performed only on the first input channel, and the convolution result is shuffled with the unprocessed second input channel. The shuffled result is then used as the input to the next network layer.
It effectively simplifies computational complexity, reduces computational resource consumption, and improves lip reading efficiency, with negligible impact on the accuracy of recognition results.
Smart Images
Figure CN115482572B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning technology, and in particular to a lip reading method, as well as a lip reading device, electronic device, and computer-readable storage medium. Background Technology
[0002] With the continuous development of social productivity and technology, the demand for VR (Virtual Reality) technology is increasing across all industries. VR technology is a multi-source information fusion, interactive three-dimensional dynamic scene and entity behavior that can immerse users in the environment through system simulation. Currently, VR can be used to achieve more interactive functions to improve the quality of life, such as lip reading, which can realize speech recognition in noisy environments and eliminate some communication barriers.
[0003] VR products are Android devices that include a built-in camera. They can perform lip reading using lightweight convolutional neural networks (CNNs). However, while traditional lightweight CNN models can guarantee high accuracy, their computational complexity is still high due to the large number of parameters. This not only requires significant computing resources but also reduces the efficiency of lip reading.
[0004] Therefore, how to effectively simplify computational complexity, reduce the consumption of computing resources, and improve the efficiency of lip reading recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide a lip reading method that can effectively simplify computational complexity, reduce the consumption of computing resources, and improve lip reading efficiency. Another purpose of this application is to provide a lip reading device, electronic device, and computer-readable storage medium, all of which have the above-mentioned beneficial effects.
[0006] Firstly, this application provides a lip-reading recognition method, including:
[0007] Acquire consecutive video frames;
[0008] Determine the lip region of each video frame in the continuous video frames to obtain a region image;
[0009] The images of each region are processed using a preset neural network model to obtain lip reading recognition results;
[0010] Each convolutional layer in the preset neural network model is used for:
[0011] The input information of the first input channel is convolved to obtain the convolution result;
[0012] The input information of the second input channel is mixed with the convolution result to obtain a shuffling result;
[0013] Output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer.
[0014] Optionally, determining the lip region of each video frame in the consecutive video frames to obtain a region image includes:
[0015] Perform face recognition on each of the consecutive video frames to determine the face region;
[0016] The lip region is identified within the face region using a Haar classifier, and an image of the region is obtained.
[0017] Optionally, before determining the lip region of each video frame in the consecutive video frames and obtaining the region image, the method further includes:
[0018] Each video frame is processed to obtain a grayscale video frame.
[0019] Optionally, acquiring consecutive video frames includes:
[0020] The continuous video frames were obtained using OpenCV.
[0021] Optionally, the lip-reading method further includes:
[0022] Acquire continuous video frame samples;
[0023] Generate a sample dataset based on the continuous video frame samples;
[0024] The preset neural network model is obtained by training the model using the sample dataset.
[0025] Optionally, before training the model using the sample dataset to obtain the preset neural network model, the method further includes:
[0026] Each video frame sample in the sample dataset is converted to a format to obtain a video frame sample that meets a preset format; wherein, the format conversion includes one or more of the following: compression processing, grayscale processing, dimension adjustment processing, vectorization processing, and normalization processing.
[0027] Optionally, after processing the images of each region using a preset neural network model to obtain the lip-reading recognition result, the method further includes:
[0028] The preset neural network model is optimized using the lip reading results to obtain an optimized preset neural network model.
[0029] Secondly, this application also discloses a lip-reading recognition device, comprising:
[0030] The acquisition module is used to acquire continuous video frames;
[0031] The determination module is used to determine the lip region of each video frame in the continuous video frames and obtain a region image;
[0032] The processing module is used to process the images of each region using a preset neural network model to obtain lip reading recognition results;
[0033] Each convolutional layer in the preset neural network model includes:
[0034] A convolutional unit is used to perform convolution processing on the input information of the first input channel to obtain the convolution result;
[0035] The shuffling unit is used to shuffle the input information of the second input channel with the convolution result to obtain the shuffling result;
[0036] An output unit is used to output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer.
[0037] Thirdly, this application also discloses an electronic device, comprising:
[0038] Memory, used to store computer programs;
[0039] A processor for executing the computer program to implement any of the lip reading methods described above.
[0040] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the lip-reading methods described above.
[0041] This application provides a lip-reading recognition method, comprising: acquiring consecutive video frames; determining the lip region of each video frame in the consecutive video frames to obtain a region image; processing each region image using a preset neural network model to obtain a lip-reading recognition result; wherein each convolutional layer in the preset neural network model is used to: perform convolution processing on the input information of a first input channel to obtain a convolution result; perform shuffling processing on the input information of a second input channel and the convolution result to obtain a shuffling result; output the shuffling result; the sum of the number of channels of the first input channel and the number of channels of the second input channel is the total number of input channels of the convolutional layer.
[0042] By applying the technical solution provided in this application, lip reading recognition is achieved using a pre-created neural network model. In the neural network model, the role of the convolutional layer is feature extraction. When processing data, each convolutional layer only performs convolution processing on the input information of a portion of the input channels, while leaving the input information of the other portion of the input channels unprocessed. Then, the processed convolutional result and the unprocessed input information are mixed, and the mixed result is output as the input of the next network layer. Thus, each convolutional layer in the pre-defined neural network model only needs to process a portion of the input information instead of all the input information. That is, each convolutional layer only performs feature extraction on a portion of the input information. Obviously, compared to performing convolution operations on the input information of all input channels, the number of parameters can be greatly reduced, effectively simplifying the computational complexity. Therefore, this technical solution can improve the efficiency of lip reading recognition while reducing the consumption of computing resources. Furthermore, by using the shuffling technique, the processed convolutional results and the unprocessed input information can be shuffled before being input into the next convolutional layer. Thus, in all convolutional layers except the first one, the input information of the first and second input channels simultaneously includes both the processed convolutional results and the unprocessed input information. Therefore, this implementation method can achieve the feature extraction required for lip reading without significantly affecting the accuracy of the recognition results. Moreover, experiments have shown that the impact on the accuracy of the recognition results is negligible.
[0043] The lip-reading device, electronic device, and computer-readable storage medium provided in this application also have the above-mentioned technical effects, and will not be described in detail here. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.
[0045] Figure 1 A flowchart illustrating a lip-reading recognition method provided in this application;
[0046] Figure 2 A schematic diagram of the structure of a preset neural network model provided in this application;
[0047] Figure 3 A schematic diagram of the structure of a lip-reading device provided in this application;
[0048] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation
[0049] The core of this application is to provide a lip-reading recognition method that can effectively simplify computational complexity, reduce the consumption of computing resources, and improve lip-reading recognition efficiency. Another core aspect of this application is to provide a lip-reading recognition device, electronic device, and computer-readable storage medium, all of which have the aforementioned beneficial effects.
[0050] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0051] This application provides a lip reading method.
[0052] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a lip-reading recognition method provided in this application, which may include the following steps S101 to S103.
[0053] S101: Acquire consecutive video frames;
[0054] This step aims to acquire continuous video frames, which are the video frame sequences obtained by dividing a continuous video segment into video frames. It is understood that to achieve lip-reading recognition, continuous video of a person speaking is required to identify their speech information. Therefore, this step requires acquiring continuous video frames including faces. Further, dividing these continuous video frames into video frames yields the complete sequence of video frames. The method of acquiring these continuous video frames does not affect the implementation of this technical solution; it can be a continuous video frame sequence or continuous video directly input by a technician, or a continuous video frame sequence or continuous video acquired through an image acquisition device. This application does not limit the method in this regard.
[0055] S102: Determine the lip region of each video frame in a series of video frames and obtain the region image;
[0056] This step aims to acquire the lip image for each frame in a series of video frames. Specifically, after acquiring the series of video frames, image recognition can be performed on each frame to determine the lip region of the face in that frame, thereby obtaining a region image containing only the lip region of the face.
[0057] S103: Use a preset neural network model to process the images of each region to obtain lip reading results;
[0058] Each convolutional layer in the preset neural network model is used for:
[0059] The input information of the first input channel is convolved to obtain the convolution result;
[0060] The input information of the second input channel is mixed with the convolution result to obtain the shuffled result;
[0061] Output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer.
[0062] This step aims to achieve lip reading recognition based on a preset neural network model. After obtaining the regional images of the lip area of the face in each video frame, these images are output to the preset neural network model for processing. The output of this preset neural network model is the lip reading recognition result for consecutive video frames.
[0063] The preset neural network model is a pre-created neural network model used to realize lip reading recognition of continuous facial lip region images. It can be pre-stored in the corresponding storage medium (such as memory, external storage device, etc.) and can be directly called when needed.
[0064] It is understandable that a neural network model consists of multiple network layers, each used to perform different processing functions. Convolutional layers, in particular, are used for feature extraction. To simplify computation and reduce computational cost, each convolutional layer in the pre-defined neural network model can be configured as follows:
[0065] First, all input channels in the convolutional layer are divided into first input channels and second input channels. The sum of the number of first input channels and the number of second input channels equals the total number of input channels in the current convolutional layer (i.e., the total number of input channels mentioned above). The number of first and second input channels is not unique; they can be divided according to a preset ratio. For example, half of the input channels can be used as first input channels, and the remaining half as second input channels. However, to ensure the accuracy of lip-reading results, the number of first input channels should ideally not be less than the number of second input channels.
[0066] Furthermore, during data processing, the convolutional layer can perform convolution on only the input information of the first input channel to obtain the corresponding convolution result, while leaving the input information of the second input channel unprocessed. Then, the processed convolution result is mixed with the unprocessed input information to obtain a shuffled result. Thus, each convolutional layer in the pre-defined neural network model only needs to process a portion of the input information, rather than all of it. That is, each convolutional layer only performs feature extraction on a portion of the input information. Compared to performing convolution operations on the input information of all input channels, this significantly reduces the number of parameters and effectively simplifies computational complexity. Therefore, it can improve lip-reading efficiency while reducing computational resource consumption.
[0067] Finally, the shuffling result is output. This means the shuffling result serves as both the output of the current convolutional layer and the input of the next network layer. It's understandable that the shuffling process aims to shuffle the order of the two types of information, achieving their fusion. Therefore, in all convolutional layers except the first one, it can be ensured that the input information of both the first and second input channels simultaneously includes the processed convolutional result and the unprocessed input information. This effectively reduces the impact on the accuracy of lip reading recognition, ensuring the accuracy of the lip reading results.
[0068] Based on this, please refer to the following example:
[0069] For an input feature with resolution H×W (pixel height * pixel width), a spread factor of t, a depthwise convolutional kernel spatial size of 3, a convolutional stride of s, and a convolutional layer with N input channels and M output channels:
[0070] (1) The computational cost of a first-layer 1×1 ordinary convolution is: 1×1×H×W×N×N×t, the computational cost of a second-layer 3×3 depthwise convolution is: 3×3×H / s×W / s×N×t, and the computational cost of a third-layer 1×1 ordinary convolution is: 1×1×H / s×W / s×N×t×M. Therefore:
[0071] The computational complexity is:
[0072] 1×1×H×W×N×N×t+3×3×H / s×W / s×N×t+1×1×H / s×W / s×N×t×M=HWNt(N+9 / s 2 +M / s 2 );
[0073] (2) The parameters of the first 1×1 ordinary convolution are: 1×1×N×N×t+N×t, the parameters of the second 3×3 depthwise convolution are: 3×3×N×t+N×t, and the parameters of the third 1×1 ordinary convolution are: 1×1×N×t×M+M. Therefore:
[0074] The total number of parameters is:
[0075] 1×1×N×N×t+N×t+3×3×N×t+N×t+1×1×N×t×M+M=Nt(N+9+M)+2Nt+M.
[0076] Furthermore, let's assume: for an input feature with a resolution of 224×224, the convolutional layer has N=16 input channels, M=24 output channels, a stride of s=2, and a spread factor of t=6.
[0077] (1) If traditional techniques are used for data processing, then:
[0078] The computational complexity is: 224×224×16×6×(16+9 / 4+24 / 4)≈116.9M;
[0079] The total number of parameters is: 16×6×(16+9+24)+2×16×6+24=4920.
[0080] (2) If the technical solution provided in this application is used for data processing, then:
[0081] The computational complexity is: 224×224×(16 / 2)×6×(16 / 2+9 / 4+24 / 4)≈39.2M;
[0082] The total number of parameters is: (16 / 2)×6×(16 / 2+9+24)+2×(16 / 2)×6+24=2088.
[0083] Obviously, the technical solution provided in this application has significantly reduced computational complexity and total number of parameters compared to traditional technologies.
[0084] As can be seen, the lip-reading recognition method provided in this application utilizes a pre-created neural network model to achieve lip-reading recognition. In the neural network model, the role of the convolutional layer is feature extraction. When processing data, each convolutional layer only performs convolution processing on the input information of a portion of the input channels, while leaving the input information of the other portion of the input channels unprocessed. Then, the processed convolutional result and the unprocessed input information are shuffled together, and the shuffled result is output as the input of the next network layer. Thus, each convolutional layer in the pre-defined neural network model only needs to process a portion of the input information instead of all the input information. That is, each convolutional layer only performs feature extraction on a portion of the input information. Obviously, compared to performing convolution operations on the input information of all input channels, the number of parameters can be greatly reduced, effectively simplifying the computational complexity. Therefore, this technical solution can improve the efficiency of lip-reading recognition while reducing the consumption of computing resources. Furthermore, by using the shuffling technique, the processed convolutional results and the unprocessed input information can be shuffled before being input into the next convolutional layer. Thus, in all convolutional layers except the first one, the input information of the first and second input channels simultaneously includes both the processed convolutional results and the unprocessed input information. Therefore, this implementation method can achieve the feature extraction required for lip reading without significantly affecting the accuracy of the recognition results. Moreover, experiments have shown that the impact on the accuracy of the recognition results is negligible.
[0085] Based on the above embodiments:
[0086] In one embodiment of this application, the above-mentioned acquisition of continuous video frames may include: acquiring continuous video frames using OpenCV (a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source)).
[0087] This application provides a method for acquiring continuous video frames, specifically based on OpenCV. In the implementation process, OpenCV can call a camera to acquire real-time video, thereby obtaining continuous video frames. Furthermore, OpenCV includes a classifier that can be used to recognize the lip region of the face in step S102. Additionally, OpenCV can also call the preset neural network model in step S103, allowing the obtained lip region images of the continuous video frames to be directly input into the preset neural network model for processing.
[0088] In one embodiment of this application, the above-mentioned determination of the lip region of each video frame in a series of video frames to obtain a region image may include the following steps:
[0089] Perform face recognition on each video frame in a series of video frames to determine the face region;
[0090] The Haar classifier is used to identify and determine the lip region within the face region to obtain a region image.
[0091] This application provides a method for obtaining a facial lip region image from a video frame. For each video frame in a series of video frames, face recognition can be performed first to determine the face region in that video frame. Then, the region image corresponding to the face region is input into the Haar classifier in OpenCV for classification and recognition to determine the lip region of the face region image, thereby obtaining the facial lip region image. This achieves the acquisition of the facial lip region image from the video frame.
[0092] In one embodiment of this application, before determining the lip region of each video frame in a series of video frames and obtaining the region image, the method may further include: performing grayscale processing on each video frame to obtain a grayscale video frame.
[0093] The lip-reading recognition method provided in this application can further implement image grayscale processing to further reduce the complexity of data computation. After obtaining consecutive video frames, before acquiring the facial lip region image of each video frame in the consecutive video frames, grayscale processing can be performed on each video frame in the consecutive video frames to obtain the corresponding grayscale video frame. Then, the subsequent facial lip region image acquisition operation is performed on each grayscale video frame. It can be understood that image grayscale processing can convert a three-channel image into a single-channel image, thereby effectively reducing the computational complexity of image data processing, further improving data processing efficiency, and improving lip-reading recognition efficiency.
[0094] In one embodiment of this application, the lip-reading recognition method may further include the following steps:
[0095] Acquire continuous video frame samples;
[0096] Generate a sample dataset based on consecutive video frame samples;
[0097] The model is trained using the sample dataset to obtain the preset neural network model.
[0098] This application provides a method for creating a preset neural network model. First, continuous video frame samples can be acquired, a process that can also be implemented using OpenCV. Then, facial lip region recognition is performed on each video frame in the continuous video frame samples to obtain facial lip region image samples. These facial lip region image samples are then labeled to obtain a sample dataset. Clearly, this sample dataset includes a large number of labeled, continuous facial lip region image samples. Finally, the model is trained using this sample dataset to obtain the preset neural network model. During model training, the ADAM (a stochastic optimization algorithm) optimizer can be selected for model optimization.
[0099] For further information, please refer to [link / reference]. Figure 2 , Figure 2 The diagram below illustrates the structure of a pre-defined neural network model provided in this application. The configuration of each network layer in the pre-defined neural network model is as follows:
[0100] First layer: Convolutional layer, used to extract image features and preserve boundary pixels;
[0101] The second layer is the pooling layer, which is used to further reduce the dimensionality of the feature information extracted by the convolutional layer, thereby reducing the amount of computation. It is also used to enhance the invariance of image features, making them more robust to image shifts, rotations, etc.
[0102] The third layer: the dropout layer, used to reduce overfitting;
[0103] Fourth layer: Convolutional layer, used to further extract image features and preserve boundary pixels;
[0104] Fifth layer: Dropout layer;
[0105] Sixth layer: Convolutional layer;
[0106] Seventh layer: Pooling layer:
[0107] Eighth layer: Dropout layer;
[0108] Ninth layer: flatten layer; used to transform data from two dimensions to one dimension;
[0109] Layer 10: Fully connected layer: Used to generate the classifier to achieve the final output;
[0110] Eleventh: Dropout layer;
[0111] Twelfth: Fully connected layer;
[0112] Thirteenth: Classification layer.
[0113] In one embodiment of this application, before training the model using the sample dataset to obtain the preset neural network model, the method may further include: converting the format of each video frame sample in the sample dataset to obtain video frame samples that meet the preset format; wherein, the format conversion includes one or more of compression processing, grayscale processing, dimension adjustment processing, vectorization processing, and normalization processing.
[0114] To facilitate model training, each video frame sample in the sample dataset can be format-converted before training the model using the sample dataset to obtain video frame samples that meet a preset format. Here, the preset format is determined by the training system and model parameters, and this application does not limit it. The format conversion processing of the video frame samples includes, but is not limited to, compression processing, grayscale processing, dimensionality adjustment processing, vectorization processing, and normalization processing as described above.
[0115] (1) Compression processing: Used to compress video frame samples to a specified size;
[0116] (2) Grayscale processing: Similar to the grayscale processing of video frames described above, it will not be repeated here;
[0117] (3) Dimension adjustment processing: This is used to adjust the dimensions of video frame samples to the image dimensions required by the training system. It can be achieved using functions in the Keras and NumPy libraries (such as the reshape function).
[0118] (4) Vectorization: This is used to convert the image features in video frame samples into vector features, which can be achieved using one-hot encoding;
[0119] (5) Normalization: The purpose is to improve the convergence speed of the network, reduce the training time of the model, and adapt to activation functions with values in the range of (0,1) to increase the discriminative power.
[0120] In one embodiment of this application, after processing the images of each region using a preset neural network model to obtain lip reading results, the process may further include: optimizing the preset neural network model using the lip reading results to obtain an optimized preset neural network model.
[0121] The lip-reading recognition device provided in this application embodiment can further implement a model optimization function. After obtaining the final lip-reading recognition result, the lip-reading recognition result can be used to optimize the preset neural network model. Of course, it is preferable to use a larger number of lip-reading recognition results to perform the model optimization operation. It is conceivable that by optimizing the preset neural network model, the model accuracy can be effectively improved, and the accuracy of the lip-reading recognition result can be further improved in subsequent lip-reading recognition.
[0122] This application provides a lip reading device.
[0123] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a lip-reading device provided in this application. The lip-reading device may include:
[0124] Acquisition module 1 is used to acquire continuous video frames;
[0125] Module 2 is used to determine the lip region of each video frame in a series of video frames and obtain the region image;
[0126] Processing module 3 is used to process the images of each region using a preset neural network model to obtain lip reading recognition results;
[0127] Each convolutional layer in the preset neural network model includes:
[0128] Convolution unit 31 is used to perform convolution processing on the input information of the first input channel to obtain the convolution result;
[0129] The shuffling unit 32 is used to shuffle the input information of the second input channel with the convolution result to obtain the shuffling result;
[0130] Output unit 33 is used to output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer.
[0131] As can be seen, the lip-reading recognition device provided in this application embodiment utilizes a pre-created neural network model to achieve lip-reading recognition. In the neural network model, the role of the convolutional layer is feature extraction. When processing data, each convolutional layer only performs convolution processing on the input information of a portion of the input channels, while leaving the input information of the other portion of the input channels unprocessed. Then, the processed convolutional result and the unprocessed input information are shuffled together, and the shuffled result is output as the input of the next network layer. Thus, each convolutional layer in the pre-defined neural network model only needs to process a portion of the input information rather than all of the input information. That is, each convolutional layer only performs feature extraction on a portion of the input information. Obviously, compared to performing convolution operations on the input information of all input channels, the number of parameters can be greatly reduced, effectively simplifying the computational complexity. Therefore, this technical solution can improve the efficiency of lip-reading recognition while reducing the consumption of computing resources. Furthermore, by using the shuffling technique, the processed convolutional results and the unprocessed input information can be shuffled before being input into the next convolutional layer. Thus, in all convolutional layers except the first one, the input information of the first and second input channels simultaneously includes both the processed convolutional results and the unprocessed input information. Therefore, this implementation method can achieve the feature extraction required for lip reading without significantly affecting the accuracy of the recognition results. Moreover, experiments have shown that the impact on the accuracy of the recognition results is negligible.
[0132] In one embodiment of this application, the determining module 2 may include:
[0133] The face recognition unit is used to perform face recognition on each video frame in a series of video frames to determine the face region;
[0134] The lip recognition unit is used to identify and determine the lip region within the face region using a Haar classifier, thereby obtaining a region image.
[0135] In one embodiment of this application, the lip reading device may further include a grayscale processing module, which performs grayscale processing on each video frame to obtain a grayscale video frame before determining the lip region of each video frame in the continuous video frames and obtaining the region image.
[0136] In one embodiment of this application, the acquisition module 1 described above can be specifically used to acquire continuous video frames using OpenCV.
[0137] In one embodiment of this application, the lip-reading recognition device may further include a model building module for acquiring continuous video frame samples; generating a sample dataset based on the continuous video frame samples; and training the model using the sample dataset to obtain a preset neural network model.
[0138] In one embodiment of this application, the model building module is further configured to perform format conversion on each video frame sample in the sample dataset before training the model using the sample dataset to obtain a preset neural network model, thereby obtaining video frame samples that meet the preset format; wherein, the format conversion includes one or more of compression processing, grayscale processing, dimension adjustment processing, vectorization processing, and normalization processing.
[0139] In one embodiment of this application, the lip reading device may further include a model optimization module, which is used to optimize the preset neural network model using the lip reading results after processing the images of each region using the preset neural network model to obtain the lip reading results, so as to obtain an optimized preset neural network model.
[0140] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.
[0141] This application provides an electronic device.
[0142] Please refer to Figure 4 , Figure 4 This application provides a schematic diagram of the structure of an electronic device, which may include:
[0143] Memory, used to store computer programs;
[0144] A processor, used to execute computer programs, can implement the steps of any of the lip reading methods described above.
[0145] like Figure 4 The diagram shows the structural composition of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.
[0146] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.
[0147] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the lip reading method.
[0148] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:
[0149] Acquire consecutive video frames;
[0150] Identify the lip region in each video frame of a series of video frames to obtain the region image;
[0151] The images of each region are processed using a pre-defined neural network model to obtain lip reading results;
[0152] Each convolutional layer in the preset neural network model is used for:
[0153] The input information of the first input channel is convolved to obtain the convolution result;
[0154] The input information of the second input channel is mixed with the convolution result to obtain the shuffled result;
[0155] Output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer.
[0156] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.
[0157] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.
[0158] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.
[0159] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 4 More or fewer components as shown, or combinations of certain components.
[0160] This application provides a computer-readable storage medium.
[0161] The computer-readable storage medium provided in this application embodiment stores a computer program, which, when executed by a processor, can implement the steps of any of the lip reading methods described above.
[0162] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0163] For a description of the computer-readable storage medium provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.
[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0165] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0166] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0167] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.
Claims
1. A lip-reading recognition method, characterized in that, include: Acquire consecutive video frames; Determine the lip region of each video frame in the continuous video frames to obtain a region image; The images of each region are processed using a preset neural network model to obtain lip reading recognition results; In the preset neural network model, all input channels in each convolutional layer are divided into a first input channel and a second input channel, and each convolutional layer is used for: The input information of the first input channel is convolved to obtain the convolution result; The input information of the second input channel is mixed with the convolution result to obtain a shuffling result; Output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer, and the number of channels in the first input channel is greater than the number of channels in the second input channel.
2. The lip-reading recognition method according to claim 1, characterized in that, The step of determining the lip region of each video frame in the consecutive video frames and obtaining the region image includes: Perform face recognition on each of the consecutive video frames to determine the face region; The lip region is identified within the face region using a Haar classifier, and an image of the region is obtained.
3. The lip-reading recognition method according to claim 1, characterized in that, Before determining the lip region of each video frame in the consecutive video frames and obtaining the region image, the method further includes: Each video frame is processed to obtain a grayscale video frame.
4. The lip-reading recognition method according to claim 1, characterized in that, The acquisition of continuous video frames includes: The continuous video frames were obtained using OpenCV.
5. The lip-reading recognition method according to claim 1, characterized in that, Also includes: Acquire continuous video frame samples; Generate a sample dataset based on the continuous video frame samples; The preset neural network model is obtained by training the model using the sample dataset.
6. The lip-reading recognition method according to claim 5, characterized in that, Before training the model using the sample dataset to obtain the preset neural network model, the method further includes: Each video frame sample in the sample dataset is converted to a format to obtain a video frame sample that meets a preset format; wherein, the format conversion includes one or more of the following: compression processing, grayscale processing, dimension adjustment processing, vectorization processing, and normalization processing.
7. The lip-reading recognition method according to claim 5, characterized in that, After processing the images of each region using a preset neural network model to obtain lip reading results, the method further includes: The preset neural network model is optimized using the lip reading results to obtain an optimized preset neural network model.
8. A lip-reading recognition device, characterized in that, include: The acquisition module is used to acquire continuous video frames; The determination module is used to determine the lip region of each video frame in the continuous video frames and obtain a region image; The processing module is used to process the images of each region using a preset neural network model to obtain lip reading recognition results; In the preset neural network model, all input channels in each convolutional layer are divided into a first input channel and a second input channel, and each convolutional layer includes: A convolutional unit is used to perform convolution processing on the input information of the first input channel to obtain the convolution result; The shuffling unit is used to shuffle the input information of the second input channel with the convolution result to obtain the shuffling result; An output unit is used to output the shuffling result; the sum of the number of channels in the first input channel and the number of channels in the second input channel is the total number of input channels in the convolutional layer, and the number of channels in the first input channel is greater than the number of channels in the second input channel.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the lip reading method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the lip-reading method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
An iris living body detection method based on a lightweight convolutional neural network
CN109409342A