A lightweight lip reading recognition method and related equipment

Abstract

The present invention provides a lightweight lip reading recognition method and related equipment, including: obtaining a lip picture sequence sample set, inputting the lip reading picture sequence sample set into a lip reading recognition teacher model for recognition, obtaining a first probability distribution value of a word corresponding to each lip reading picture sequence sample, and distilling the first probability distribution value to obtain a first prediction value; inputting the lip reading picture sequence sample set into a lip reading recognition student model, using the first prediction value as a prediction target to guide the lip reading recognition student model for training, and obtaining a lightweight lip reading recognition model; inputting a lip reading video to be recognized into a target detection module for detection, obtaining a lip picture sequence, and inputting the lip picture sequence into the lightweight lip reading recognition model for lip reading recognition, and obtaining words corresponding to the lip picture sequence; solving the problem of weak real-time response of the existing lip recognition model, improving recognition efficiency, and improving the accuracy of model recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language recognition, and in particular to a lightweight lip reading recognition method and related equipment. Background Art

[0002] At present, since audio-based speech recognition is easily affected by background noise and overlapping speech, visual speech recognition technology represented by lip reading recognition has been developed to improve the efficiency of speech recognition and capture additional visual information. Compared with other recognition methods, such as gesture recognition and fingerprint recognition, lip reading recognition has been widely used in speech recognition assistance, communication for people with sensory impairments and other fields due to its high real-time responsiveness.

[0003] A few years ago, traditional lip reading recognition was widely studied, but it suffered from low recognition efficiency and limited training data. With the increasing scale of lip reading datasets and the increasing complexity of recognition requirements, deep learning-based lip reading recognition technology has garnered widespread attention in recent years. Its advantages include extracting deep features of lip movements, high generalization, and high recognition accuracy, and it has gradually become a mainstream research area in lip reading recognition. Deep learning-based lip reading recognition methods employ the principle of inputting lip reading images into a network front-end to extract lip reading features, transferring the front-end processing results to the back-end for classification and prediction, and training in an end-to-end manner. Many researchers have innovated in various directions, increasing model complexity and deepening network layers to improve recognition accuracy. Stafylakis et al. proposed applying a deep residual network (ResNet) to the front-end of a lip reading recognition model; Chuang et al. proposed a lip reading recognition model based on a spatiotemporal convolutional neural network; and Martinez et al. replaced the recurrent neural network (RNN) back-end with a multi-scale temporal convolutional network (TCN). These methods have achieved high recognition accuracy. However, as the complexity of lip reading recognition models increases and the number of network layers increases, the computational load and parameters increase rapidly, and the requirements for device performance continue to increase. This creates a problem: most existing lip reading recognition models cannot resolve the contradiction between reducing the model's computational load and improving recognition accuracy. This limits the embedding of high-performance lip reading recognition models in mobile devices and edge devices, and cannot fully realize the display application value of deep learning lip reading recognition models.

[0004] In addition, in terms of the recognition ability of the model itself, although lip reading recognition technology based on deep learning has made great progress, it still faces many problems in practical applications, such as: the lack of lip position detection steps; different speech habits will cause the lips to show different movement trajectories; pronunciation habits and dialect habits will cause differences in lip movements; lip movements have limitations, and changes in the external environment will cause the lips to present different states; these problems increase the difficulty of recognition and reduce the recognition accuracy. Summary of the Invention

[0005] The present invention provides a lightweight lip reading recognition method and related equipment, the purpose of which is to reduce the recognition difficulty and improve the recognition accuracy.

[0006] In order to achieve the above object, the present invention provides a lightweight lip reading recognition method, comprising:

[0007] Step 1: Obtain a sample set of lip image sequences;

[0008] Step 2: Input the lip reading picture sequence sample set into the lip reading recognition teacher model for recognition, obtain a first probability distribution value of the word corresponding to each lip reading picture sequence sample, and distill the first probability distribution value to obtain a first prediction value;

[0009] Step 3: Input the lip reading picture sequence sample set into the lip reading recognition student model, use the first prediction value as the prediction target, and guide the lip reading recognition student model to perform training, thereby obtaining a lightweight lip reading recognition model;

[0010] Step 4: Input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and then input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain the words corresponding to the lip image sequence.

[0011] More specifically, step 2 includes:

[0012] Inputting the lip image sequence sample set into the lip reading recognition teacher model for recognition, and obtaining a first probability distribution value of a word corresponding to each lip reading image sequence sample;

[0013] Under the temperature coefficient T=t, a smoothing operation is performed on the first probability distribution value through a softmax function to obtain a first prediction value.

[0014] More specifically, step 3 includes:

[0015] The true labels of the lip image sequence samples in the lip image sequence sample set are used as hard labels;

[0016] The first predicted value is used as the soft label;

[0017] The hard labels are input into the lip reading recognition student model, and the soft labels are used as prediction targets to guide the lip reading recognition student to perform training, thereby obtaining the second probability distribution value of the word corresponding to each lip reading image sequence sample;

[0018] Under the temperature coefficient T=t, the second probability distribution value is smoothed by the softmax function to obtain a second predicted value;

[0019] Calculate the first prediction value and the second prediction value using the average error loss function to obtain a distillation loss value;

[0020] Under the temperature coefficient T=1, the second probability distribution value is smoothed by the softmax function to obtain the third prediction value;

[0021] The third predicted value and hard label are calculated through the cross entropy loss function to obtain the student loss;

[0022] The total loss value is calculated based on the distillation loss and the student loss, and the lip reading recognition student model is iteratively updated using the total loss value to obtain a lightweight lip reading recognition model.

[0023] Furthermore, the total loss is calculated based on the distillation loss and the student loss, where (1-α)MSE(y s2 ,y t ) is the distillation loss, αCE(y s3 , y) is the student loss part, and the function expression of the total loss value is:

[0024] L total =αCE(y s3 ,y)+(1-α)MSE(y s2 ,y t )

[0025] Among them, α is the weight, CE is the cross entropy loss function, y s3 is the third predicted value output by the lip reading recognition student model, y is the hard label, MSE is the mean square error loss function, y s2 Output the second prediction value for the lip reading recognition student model, y t The first prediction value output by the lip reading recognition teacher model.

[0026] Furthermore, the lightweight lip reading recognition model includes a front-end module and a back-end module;

[0027] The front-end module includes: a sequentially connected spatiotemporal 3D convolutional layer, a first batch normalization layer, a first rectified linear activation unit, a maximum pooling layer, a SE-ResNet-18 network, and a global average pooling layer;

[0028] The input of the spatiotemporal 3D convolutional layer is connected to the output of the target detection module, and the output of the global average pooling layer is connected to the output of the backend module;

[0029] The backend module includes: a first gated recurrent unit, a second gated recurrent unit, a third gated recurrent unit, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a fully connected layer, a global average pooling layer, a second batch normalization layer, a second rectified linear activation unit, and a softmax output layer, which are connected in sequence;

[0030] The input of the first gated recurrent unit is connected to the output of the global average pooling layer, and the softmax output layer outputs the recognition result.

[0031] Furthermore, the SE-ResNet-18 network adds a squeeze module for squeezing the lip image sequence and an excitation module for excitation processing the lip image sequence to each residual block of the ResNet network. The SE-ResNet-18 network is used to extract features of the lip image sequence, and the extracted features are input into the average pooling layer for compression to obtain the compression result.

[0032] Furthermore, the lip reading video to be recognized is input into the target detection module for detection, and a lip image sequence is obtained, including:

[0033] The video captured by the system camera is read at a speed of 30 frames as the lip reading video to be recognized;

[0034] The lip reading video to be recognized is cut into frames and input into the target detection module for processing to obtain the lip coordinates;

[0035] Cut the lip reading video after frame cutting according to the lip coordinates to obtain the lip picture;

[0036] Preprocess the lip images to obtain lip reading image sequences.

[0037] The present invention also provides a lightweight lip reading recognition device, comprising:

[0038] An acquisition module is used to obtain a sample set of lip image sequences;

[0039] a distillation module, configured to input the lip reading picture sequence sample set into the lip reading recognition teacher model for recognition, obtain a first probability distribution value of a word corresponding to each lip reading picture sequence sample, and distill the first probability distribution value to obtain a first prediction value;

[0040] A training module is used to input a lip reading image sequence sample set into a lip reading recognition student model, guide the lip reading recognition student model to perform training with the first prediction value as a prediction target, and obtain a lightweight lip reading recognition model;

[0041] The detection and recognition module is used to input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and then input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain the words corresponding to the lip image sequence.

[0042] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the lightweight lip reading recognition method is implemented.

[0043] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a lightweight lip reading recognition method is implemented.

[0044] The above solution of the present invention has the following beneficial effects:

[0045] The present invention obtains a lip picture sequence sample set, inputs the lip reading picture sequence sample set into a lip reading recognition teacher model for recognition, obtains a first probability distribution value of a word corresponding to each lip reading picture sequence sample, and distills the first probability distribution value to obtain a first prediction value; inputs the lip reading picture sequence sample set into a lip reading recognition student model, takes the first prediction value as a prediction target, guides the lip reading recognition student model for training, and obtains a lightweight lip reading recognition model; inputs the lip reading video to be recognized into a target detection module for detection to obtain a lip picture sequence, and inputs the lip picture sequence into the lightweight lip reading recognition model for lip reading recognition to obtain words corresponding to the lip picture sequence; guides the lip reading recognition student model for training through a complex lip reading recognition teacher model, and allows the lip reading recognition student model to learn the knowledge of the lip reading recognition teacher model through distillation, so that the lip reading recognition student model has the recognition accuracy performance of the lip reading recognition teacher model while being lightweight. The target detection module and the lightweight lip reading recognition model are used to solve the problem of weak real-time response of the existing lip recognition model, improve the recognition efficiency, and improve the accuracy of model recognition.

[0046] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a module according to an embodiment of the present invention;

[0049] Figure 3 This is a principle block diagram of knowledge distillation in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the technical problems, technical solutions, and advantages to be solved by the present invention more clear, the following is a detailed description with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0052] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to a locking connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0053] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] In view of the existing problems, the present invention provides a lightweight lip reading recognition method and related equipment.

[0055] like Figure 1 、 2 As shown, an embodiment of the present invention provides a lightweight lip reading recognition method, including:

[0056] Step 1: Obtain a sample set of lip image sequences;

[0057] Step 2: Input the lip reading picture sequence sample set into the lip reading recognition teacher model for recognition, obtain a first probability distribution value of the word corresponding to each lip reading picture sequence sample, and distill the first probability distribution value to obtain a first prediction value;

[0058] Step 3: Input the lip reading picture sequence sample set into the lip reading recognition student model, use the first prediction value as the prediction target, and guide the lip reading recognition student model to perform training, thereby obtaining a lightweight lip reading recognition model;

[0059] Step 4: Input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and then input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain the words corresponding to the lip image sequence.

[0060] Specifically, the lip reading recognition teacher model and the lip reading recognition student model used in the embodiments of the present invention are both network frameworks built based on the end-to-end deep learning lip reading recognition idea. The lip reading recognition teacher model is appropriately deepened in terms of the number of network layers to enhance the network's performance. The model can learn higher-dimensional feature information and more subdivide the data.

[0061] A certain number of words are selected as the recognition library based on the actual situation. Each word can be considered as a category. In the preprocessing stage, a word list is generated based on the label and file name. The subscripts of the word list correspond to each category. During training, the training lip images are mirror-flipped for data augmentation. The Adam optimizer is used during training for fast convergence. The initial learning rate is set to 3×10 -4 , weight decay is set to 10 -4 .

[0062] Specifically, step 2 includes:

[0063] Inputting the lip image sequence sample set into the lip reading recognition teacher model for recognition, and obtaining a first probability distribution value of a word corresponding to each lip reading image sequence sample;

[0064] Under the temperature coefficient T=t, a smoothing operation is performed on the first probability distribution value through a softmax function to obtain a first prediction value.

[0065] Specifically, step 3 includes:

[0066] The true labels of the lip image sequence samples in the lip image sequence sample set are used as hard labels;

[0067] The first predicted value is used as the soft label;

[0068] The hard labels are input into the lip reading recognition student model, and the soft labels are used as prediction targets to guide the lip reading recognition student to perform training, thereby obtaining the second probability distribution value of the word corresponding to each lip reading image sequence sample;

[0069] Under the temperature coefficient T=t, the second probability distribution value is smoothed by the softmax function to obtain a second predicted value;

[0070] Calculate the first prediction value and the second prediction value using the average error loss function to obtain a distillation loss value;

[0071] Under the temperature coefficient T=1, the second probability distribution value is smoothed by the softmax function to obtain the third prediction value;

[0072] The third predicted value and hard label are calculated through the cross entropy loss function to obtain the student loss;

[0073] The total loss value is calculated based on the distillation loss and the student loss, and the lip reading recognition student model is iteratively updated using the total loss value to obtain a lightweight lip reading recognition model.

[0074] The lip reading recognition teacher model used in the embodiment of the present invention is a complex lip reading recognition model with good prediction accuracy. During the training process, there is no need to consider the restrictions on the number, size, and integration of model parameters. For input X, it can output Y relatively accurately, where Y is mapped by softmax, and the output value corresponds to the probability distribution value of the lip reading; the lip reading recognition student model used is a model with a relatively simple structure and a relatively small number of parameters. After training, it can also achieve the goal of relatively accurately outputting Y (corresponding words) for input X (lip sequence). In order to realize the possibility of using the lip reading recognition model on mobile devices, it is necessary to achieve lightweightness. Therefore, the embodiment of the present invention uses the lip reading recognition student model as a lightweight lip reading recognition model. However, since the lip reading recognition teacher model has a strong learning ability, in order to enhance the generalization ability of the lip reading recognition student model, the knowledge learned by the lip reading recognition teacher model is transferred to the lip reading recognition student model with relatively weaker learning ability.

[0075] like Figure 3 As shown in Figure 2, the process of knowledge distillation is as follows:

[0076] (1) Smoothing the first probability distribution value output by the lip reading recognition teacher model to obtain a soft label, while the true label of the lip image sequence sample set is a hard label;

[0077] (2) When performing distillation, the temperature coefficient T is introduced. The temperature coefficient T controls the distillation degree of the lip reading recognition teacher model and is positively correlated with the learning degree of the lip reading recognition student model. The higher T is, the richer the knowledge learned by the lip reading recognition student model. This coefficient controls the first probability distribution value Z output by the lip reading recognition teacher model. i Perform a smoothing operation, i represents the i-th row of the softmax weight matrix, corresponding to the i-th label; the first probability distribution value after the smoothing operation is the soft label, recorded as q i , the calculation formula of soft label is:

[0078]

[0079] (3) When training the lip reading recognition student model, the soft label plays the role of a supervisory signal. By minimizing the KL divergence of the distillation loss, the probability distribution of the lip reading recognition student model for words is made close to that of the lip reading recognition teacher model, thereby improving the generalization ability of the lip reading recognition student model. The hard label is used by the lip reading recognition student model to avoid incorrect information. Even if the trained lip reading recognition student model encounters a lip reading image sequence without a category label, it can still determine the word corresponding to the lip reading based on the previously learned knowledge.

[0080] (4) L2 regularization, i.e. weight attenuation, can make overfitting parameters tend to zero and reduce test error. Knowledge distillation can convert weight terms into distillation loss based on L2 regularization, which can reduce the error of model testing.

[0081] (5) The total loss function Loss of the knowledge distillation operation of the lip reading model can be defined as the weighted sum of the cross entropy loss function of the lip reading recognition student model and the KL divergence of the distillation loss. Its expression is shown in the following formula:

[0082] Loss=αCE(y s ,y)+(1-α)T 2 KL(y s ,y t )

[0083] Among them, α is the weight, CE is the cross entropy loss function, y s is the predicted probability of the lip reading recognition student model, y is the true label (lip sequence), y t is the predicted probability of the lip reading recognition teacher model.

[0084] The knowledge distillation method adopted in the embodiment of the present invention includes bidirectional calculation. First, in the forward calculation, the lip reading recognition teacher model takes the lip picture sequence sample set as input and outputs the first probability distribution value as the prediction standard; the lip reading recognition student model takes the lip sequence as input and is trained with soft labels and hard labels. The hard label can be understood as the definite answer that is usually known in the data set and can be considered as the true label; the soft label can be understood as the probability distribution value of the lip sequence corresponding to a certain word, which is the output of the lip reading recognition teacher model. The distillation loss will be calculated using the mean error loss function. The lip reading recognition student model is trained using hard labels to reduce the possibility of obtaining erroneous information through distillation. The student loss is calculated using the cross entropy loss function. The lip reading recognition student model then reversely obtains the total loss and gradient, updates the model optimizer, updates the model parameters and parameter weight values, and thus reduces the error; then the gradient is cleared and the data is iterated to continue knowledge distillation; after each knowledge distillation, the total loss value is calculated based on the distillation loss value and the student loss value, where (1-α)MSE(y s2 , yt) is the distillation loss, αCE(y s3, y) is the student loss, and the function expression of the total loss value is:

[0085] L total =αCE(y s ,y)+(1-α)MSE(y s ,y t )

[0086] Among them, α is the weight, CE is the cross entropy loss function, y s3 is the third predicted value output by the lip reading recognition student model, y is the hard label, MSE is the mean square error loss function, y s2 Output the second prediction value for the lip reading recognition student model, y t The first prediction value output by the lip reading recognition teacher model.

[0087] Specifically, the lightweight lip reading recognition model includes a front-end module and a back-end module;

[0088] The front-end module includes: a sequentially connected spatiotemporal 3D convolutional layer, a first batch normalization layer, a first rectified linear activation unit, a maximum pooling layer, a SE-ResNet-18 network, and a global average pooling layer;

[0089] The input of the spatiotemporal 3D convolutional layer is connected to the output of the target detection module, and the output of the global average pooling layer is connected to the output of the backend module;

[0090] The backend module includes: a first gated recurrent unit, a second gated recurrent unit, a third gated recurrent unit, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a fully connected layer, a global average pooling layer, a second batch normalization layer, a second rectified linear activation unit, and a softmax output layer, which are connected in sequence;

[0091] The input of the first gated recurrent unit is connected to the output of the global average pooling layer, and the softmax output layer outputs the recognition result.

[0092] Based on the lip reading recognition model proposed by RUAN, the embodiment of the present invention adds a first batch of processing normalization layers and a first corrected linear activation unit between the spatiotemporal three-dimensional convolution layer and the maximum pooling layer of the front-end module. After the first batch of processing normalization and corrected linear activation units, the network passes through the maximum pooling layer, which can speed up the training and convergence of the network, control gradient explosion, prevent gradient vanishing, prevent overfitting, and reduce the spatial size of the three-dimensional feature map. The formula of the ReLU function is as follows:

[0093]

[0094] Among them, a can be a very small number, which can make the neurons with output less than 0 also undergo a small update.

[0095] Batch Normalization and Rectified Linear Activation Units Batch normalization consists of four steps:

[0096] Mean processing, shifting the input data center to 0;

[0097] Variance processing: the data shifted to 0 is stretched or reduced in the output direction to a multiple of 1 standard deviation;

[0098] Gamma processing converts the signal of 1 standard deviation into gamma times;

[0099] Beta processing is performed and then the data is shifted in the output direction.

[0100] Specifically, the spatiotemporal convolution layer includes a convolution layer with a three-dimensional kernel of size 5×7×7, followed by a first batch normalization and first rectified linear activation unit and a maximum pooling (max-pooling) layer. The first batch normalization and first rectified linear activation units can be combined into one layer or be separate layers. It is able to capture the short-term dynamics of the lip area and has been shown to have advantages even when recurrent networks are deployed in the back-end module. The max-pooling layer compresses the extracted features and reduces the spatial size of the three-dimensional feature map. When using max-pooling, the time dimension is kept unchanged to avoid the loss of motion information in the sequence due to the short duration of the lip sequence images when the max-pooling layer is used to compress the features.

[0101] The SE-ResNet-18 network in this embodiment is an improved version of the conventional ResNet, with squeeze and excitation modules added to each residual block. The SE-ResNet-18 network is used to extract recognition features, which are then fed into a global average pooling layer (GAP) for further compression into a T×D-dimensional output, where D represents the final channel of the front-end module.

[0102] It should be noted that ResNet is a network built based on a deep residual network. In the residual network of ResNet, a convolution process is first performed, and then batch normalization (BN) and ReLU function activation are performed. After that, a convolution operation and batch normalization (BN) are performed again. Finally, the input feature map is added, that is, the output processed by the max-pool layer in the lip reading model is used as the input of the SE-ResNet-18 network. After addition, it enters the ReLU function for activation. The second residual module of the network adds a 1*1 convolution in the skip layer connection for downsampling. In the embodiment of the present invention, the SE-ResNet-18 network adds an extrusion module for extrusion processing of the lip picture sequence and an excitation module for excitation processing of the lip picture sequence to each residual block of the ResNet network. The SE-ResNet-18 network is used to extract features of the lip picture sequence, and the extracted features are input into the average pooling layer for compression to obtain a compression result.

[0103] The embodiment of the present invention adds five convolutional layers to the back-end module of the lip reading recognition model proposed by RUAN, reducing the number of convolutional layers to six one-dimensional convolutional layers (1D-Convolution) to improve model performance and reduce learning data. The back-end module utilizes a network composed of 1D convolution and gated recurrent unit (GRU) structures. The network consists of a softmax output layer, a fully connected layer with a global average pooling layer (FC+GAP), six one-dimensional convolutional layers (1D-Convolution), and three gated recurrent unit (BI-GRU) layers to capture and classify the underlying patterns of the sequence. The GRU is a variant of the traditional RNN. Like the LSTM, it can effectively capture semantic associations between long sequences and mitigate the gradient explosion phenomenon, but its structure and computation are simpler than the LSTM, making it more lightweight. A single Bi-GRU consists of two independent unidirectional GRUs. The input sequence is fed into one GRU in the normal order and into the other GRU in the reverse order. The outputs of the two GRUs are then concatenated at each time step to represent the entire sequence. Each GRU layer contains 1024 parameters, which means each Bi-GRU contains 2048 parameters. The parameters can be changed according to specific requirements.

[0104] The output of the Bi-GRU is passed to a series of one-dimensional convolutional layers. The kernel of the one-dimensional convolutional layer, denoted as M, is set to 64 and 32, as shown in Table 1:

[0105] Table 1

[0106]

[0107] This method makes the one-dimensional convolution kernel longer, improving the performance of temporal feature extraction. The output of the one-dimensional convolution layer is then processed by a fully connected layer into a T×C shape. The data from the fully connected layer (FC) is averaged in the temporal dimension using a global average pooling layer (GAP) and passed to the softmax output layer for classification. The softmax output layer is used to solve the multi-feature classification problem of the lip region. In the softmax function, an approximate probability is used to represent the likelihood of the output result. For each lip movement sequence sample x, its conditional probability corresponding to the pronunciation category l can be expressed as follows:

[0108]

[0109] Specifically, the lip reading video to be recognized is input into the target detection module for detection, and a lip image sequence is obtained, including:

[0110] The video captured by the system camera is read at a speed of 30 frames as the lip reading video to be recognized;

[0111] The lip reading video to be recognized is cut into frames and input into the target detection module for processing to obtain the lip coordinates;

[0112] Cut the lip reading video after frame cutting according to the lip coordinates to obtain the lip picture;

[0113] Preprocess the lip images to obtain lip reading image sequences.

[0114] To enhance the applicability of lip reading recognition, the embodiment of the present invention adds an object detection module before using the lightweight lip reading recognition model to recognize images. The object detection module integrates the YOLOv4-tiny algorithm. The performance comparison of the YOLOv4-tiny algorithm with other algorithms is shown in Table 2 below:

[0115] Table 2

[0116] algorithm mAP@0.5 / % Parameter quantity / Mega FPS YOLOv4-tiny 65.2 6.1 163.9 YOLOv3-tiny 58.4 8.7 151.5 YOLOv5s 78.1 7.2 116.3 YOLOv4 82.3 87.6 52.1

[0117] As shown in Table 2, the YOLOv4-tiny algorithm is a lightweight algorithm with high performance. The benefit of introducing an object detection algorithm is that it improves the system's real-time responsiveness. The system can capture the subject's lips, reducing the input of artificial lip images and improving application efficiency.

[0118] Specifically, the video captured by the system camera is read at a speed of 30 frames and saved as the lip reading video to be recognized in the local relevant directory; the video frame is cut and input into the target detection module for processing, the target detection module obtains the lip coordinates, crops and saves them to a new list and returns them; the image input by data processing is scaled and padded to 112, and bilateral filtering denoising is introduced to make the lip edges more three-dimensional and reduce image shadows; normalization, conversion of data type, and unification of the upper and lower boundaries are performed. After data processing is completed, a lip image sequence is obtained, and the lip image sequence is input into the lightweight lip reading recognition model, so that the system can output the predicted lip reading word category in combination with the vocabulary.

[0119] The embodiment of the present invention obtains a lip picture sequence sample set, inputs the lip reading picture sequence sample set into a lip reading recognition teacher model for recognition, obtains a first probability distribution value of a word corresponding to each lip reading picture sequence sample, and distills the first probability distribution value to obtain a first prediction value; inputs the lip reading picture sequence sample set into a lip reading recognition student model, uses the first prediction value as a prediction target, guides the lip reading recognition student model for training, and obtains a lightweight lip reading recognition model; inputs the lip reading video to be recognized into a target detection module for detection to obtain a lip picture sequence, and inputs the lip picture sequence into a lightweight lip reading recognition model for lip reading recognition to obtain words corresponding to the lip picture sequence; guides the lip reading recognition student model for training through a complex lip reading recognition teacher model, and allows the lip reading recognition student model to learn the knowledge of the lip reading recognition teacher model through distillation, so that the lip reading recognition student model has the recognition accuracy performance of the lip reading recognition teacher model while being lightweight. The target detection module and the lightweight lip reading recognition model are used to solve the problem of weak real-time response of the existing lip recognition model, improve the recognition efficiency, and improve the accuracy of model recognition.

[0120] The embodiment of the present invention further provides a lightweight lip reading recognition device, comprising:

[0121] An acquisition module is used to obtain a sample set of lip image sequences;

[0122] A first training module is configured to input a lip image sequence sample set into a lip reading recognition teacher model, train the lip reading recognition teacher model to obtain a first probability distribution value, and perform knowledge distillation on the first probability distribution value to obtain a distilled first probability distribution value;

[0123] The second training module is used to input the lip image sequence sample set into the lip reading recognition student model, train the lip reading recognition student model, and obtain a second probability distribution value and a student loss value;

[0124] A calculation module, configured to calculate the first probability distribution value and the second probability distribution value after distillation to obtain a distillation loss value;

[0125] The update module is used to calculate the total loss value based on the distillation loss value and the student loss value, and iteratively update the lip reading recognition student model using the total loss value to obtain a lightweight lip reading recognition model;

[0126] The detection and recognition module is used to input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and then input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain the words corresponding to the lip reading image sequence.

[0127] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0129] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the lightweight lip reading recognition method is implemented.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to a construction device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0131] An embodiment of the present invention further provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a lightweight lip reading recognition method when executing the computer program.

[0132] It should be noted that the terminal device may be a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), and other terminal devices. For example, the terminal device may be a station (ST, STAION) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, etc. The embodiments of the present invention do not impose any restrictions on the specific type of the terminal device.

[0133] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0134] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the terminal device, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0135] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0136] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A lightweight lip reading recognition method, characterized in that: include: Step 1: Obtain a sample set of lip image sequences; Step 2: Input the lip reading picture sequence sample set into the lip reading recognition teacher model for recognition, obtain a first probability distribution value of the word corresponding to each lip reading picture sequence sample, and distill the first probability distribution value to obtain a first prediction value; Step 3: Input the lip reading picture sequence sample set into the lip reading recognition student model, use the first prediction value as the prediction target, and guide the lip reading recognition student model to perform training, thereby obtaining a lightweight lip reading recognition model; Step 4: Input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain words corresponding to the lip image sequence; The lightweight lip reading recognition model includes a front-end module and a back-end module; The front-end module includes: a spatiotemporal three-dimensional convolutional layer, a first batch normalization layer, a first rectified linear activation unit, a maximum pooling layer, a SE-ResNet-18 network and a global average pooling layer connected in sequence; The input end of the spatiotemporal three-dimensional convolutional layer is connected to the output end of the target detection module, and the output end of the global average pooling layer is connected to the output end of the backend module; The back-end module includes: a first gated recurrent unit, a second gated recurrent unit, a third gated recurrent unit, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a fully connected layer, a global average pooling layer, a second batch normalization layer, a second rectified linear activation unit and a softmax output layer connected in sequence; The input end of the first gated recurrent unit is connected to the output end of the global average pooling layer, and the softmax output layer outputs the recognition result; The SE-ResNet-18 network adds a squeeze module and an excitation module for squeezing and excitation processing of the lip image sequence to each residual block of the ResNet network, uses the SE-ResNet-18 network to extract features of the lip image sequence, and inputs the extracted features into the average pooling layer for compression to obtain a compression result.

2. The lightweight lip reading recognition method according to claim 1, characterized in that: The step 2 includes: Inputting the lip picture sequence sample set into the lip reading recognition teacher model for recognition, and obtaining a first probability distribution value of a word corresponding to each lip reading picture sequence sample; Under the temperature coefficient T=t, a smoothing operation is performed on the first probability distribution value using a softmax function to obtain a first prediction value.

3. The lightweight lip reading recognition method according to claim 2, characterized in that: The step 3 comprises: Using the true labels of the lip image sequence samples in the lip image sequence sample set as hard labels; Using the first predicted value as a soft label; Inputting the hard labels into the lip reading recognition student model, using the soft labels as prediction targets, guiding the lip reading recognition student to perform training, and obtaining a second probability distribution value of a word corresponding to each lip reading picture sequence sample; Under the temperature coefficient T=t, smoothing the second probability distribution value by a softmax function to obtain a second predicted value; Calculating the first prediction value and the second prediction value using an average error loss function to obtain a distillation loss value; Under the temperature coefficient T=1, smoothing the second probability distribution value by using a softmax function to obtain a third predicted value; Calculating the third predicted value and the hard label using a cross entropy loss function to obtain a student loss; A total loss value is calculated based on the distillation loss and the student loss, and the lip reading recognition student model is iteratively updated using the total loss value to obtain a lightweight lip reading recognition model.

4. The lightweight lip reading recognition method according to claim 3, characterized in that: The total loss value is calculated based on the distillation loss and the student loss, where (1-α)MSE(y s2 ,y t ) is the distillation loss, αCE(y s3 ,y) is the student loss, and the function expression of the total loss value is: L total =αCE(and s3 ,y)+(1-α)MSE(y s2 ,and t ) Among them, α is the weight, CE is the cross entropy loss function, y s3 is the third predicted value output by the lip reading recognition student model, y is the hard label, MSE is the mean square error loss function, y s2 Output the second prediction value for the lip reading recognition student model, y t The first prediction value output by the lip reading recognition teacher model.

5. The lightweight lip reading recognition method according to claim 1, characterized in that: The lip reading video to be recognized is input into the target detection module for detection, and a lip image sequence is obtained, including: The video captured by the system camera is read at a speed of 30 frames as the lip reading video to be recognized; The lip reading video to be recognized is cut into frames and then input into the target detection module for processing to obtain lip coordinates; The lip reading video after frame cutting is cut according to the lip coordinates to obtain a lip picture; The lip image is preprocessed to obtain a lip reading image sequence.

6. A lightweight lip reading recognition device, characterized in that: include: An acquisition module is used to obtain a sample set of lip image sequences; a distillation module, configured to input the lip reading picture sequence sample set into a lip reading recognition teacher model for recognition, obtain a first probability distribution value of a word corresponding to each lip reading picture sequence sample, and distill the first probability distribution value to obtain a first prediction value; A training module is configured to input the lip reading picture sequence sample set into a lip reading recognition student model, use the first prediction value as a prediction target, guide the lip reading recognition student model to perform training, and obtain a lightweight lip reading recognition model; The detection and recognition module is used to input the lip reading video to be recognized into the target detection module for detection to obtain a lip image sequence, and input the lip image sequence into the lightweight lip reading recognition model for lip reading recognition to obtain words corresponding to the lip image sequence; The lightweight lip reading recognition model includes a front-end module and a back-end module; The front-end module includes: a spatiotemporal three-dimensional convolutional layer, a first batch normalization layer, a first rectified linear activation unit, a maximum pooling layer, a SE-ResNet-18 network and a global average pooling layer connected in sequence; The input end of the spatiotemporal three-dimensional convolutional layer is connected to the output end of the target detection module, and the output end of the global average pooling layer is connected to the output end of the backend module; The back-end module includes: a first gated recurrent unit, a second gated recurrent unit, a third gated recurrent unit, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a fully connected layer, a global average pooling layer, a second batch normalization layer, a second rectified linear activation unit and a softmax output layer connected in sequence; The input end of the first gated recurrent unit is connected to the output end of the global average pooling layer, and the softmax output layer outputs the recognition result; The SE-ResNet-18 network adds a squeeze module and an excitation module for squeezing and excitation processing of the lip image sequence to each residual block of the ResNet network, uses the SE-ResNet-18 network to extract features of the lip image sequence, and inputs the extracted features into the average pooling layer for compression to obtain a compression result.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the lightweight lip reading recognition method according to any one of claims 1 to 5 is implemented.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the lightweight lip reading recognition method according to any one of claims 1 to 5 is implemented.

Images