Lightweight lip language recognition method and device

Abstract

The invention discloses a lightweight lip language recognition method and device. The method comprises the steps of obtaining a to-be-recognized original video; performing data preprocessing on the obtained original video through face tracking detection and face feature point detection technologies to obtain a lip video sequence and a compressed representation of the lip video sequence; expanding a training data set for the obtained lip video sequence data by using a data enhancement algorithm; carrying out feature extraction on the lip movement video sequence by using a 3DConv + 2DResnet convolutional neural network according to the enhanced training data set; performing sequence modeling on a feature vector obtained by performing feature extraction on the lip movement video sequence by using a Conformer time sequence modeling network to obtain a lip language recognition model; performing visual speech decoding on the lip language recognition model obtained through sequence modeling by using a classification network to obtain language characters corresponding to the to-be-recognized video; and carrying out lightweight processing on the lip language recognition model structure by using a model compression algorithm. The lip language recognition performance is improved, and the actual application requirement is met.

Description

Technical Field

[0001] The present application belongs to the technical field of lip reading recognition, and specifically relates to a lightweight lip reading recognition method and device. Background Art

[0002] In recent years, command recognition has become increasingly integrated into people's daily lives. It enables people to communicate with devices quickly and easily by providing intuitive and effective interactions. Lip reading command recognition is a research focus in the field of command recognition. Its goal is to identify the commands described by the speaker's lip movements and to enable interaction with real-world scenarios based on the content of the commands. Lip reading command recognition technology can be applied to mobile terminal command control, such as silent command interaction using smartphones in public places (for example, on the subway), auxiliary voice command recognition in noisy environments, and helping people with language barriers to achieve command interaction with smart devices, thereby improving the efficiency of human-computer interaction.

[0003] Although a lot of progress has been made in the current research on lip reading recognition algorithms, the current instruction content, that is, the limited corpus, is often composed of phrases and sentences. Therefore, higher requirements are placed on the model's feature extraction and sequence modeling capabilities. The sequence modeling capabilities of existing lip reading recognition and classification task models need to be improved. In addition, the structure of existing algorithms is relatively complex, requiring high computational costs and relatively low response speeds. The front-end and back-end networks of lip reading recognition algorithms often require a large number of parameters, which means that the algorithm often consumes large computing resources and high hardware resources to obtain accurate recognition results, and the response speed is also slow. In order to reduce the computational cost of the model, it is necessary to simplify the model structure, reduce its resource usage, and improve the model's response speed without changing the model's performance. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a lightweight lip reading recognition method and device to solve or partially solve the technical problems mentioned in the background technology.

[0005] Based on the above objectives, this application provides a lightweight lip reading recognition method, including:

[0006] Get the original video to be identified;

[0007] The obtained original video is subjected to data preprocessing by face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence;

[0008] Expanding the training data set using a data enhancement algorithm for the obtained lip video sequence data;

[0009] The enhanced training dataset is used to extract features from the lip movement video sequence using a 3DConv+2DResnet convolutional neural network;

[0010] The feature vectors obtained by feature extraction of lip movement video sequences are used to perform sequence modeling using the Conformer temporal modeling network to obtain a lip reading recognition model.

[0011] Using a classification network to decode the visual speech of the lip reading recognition model obtained through sequence modeling to obtain the language and text corresponding to the video to be recognized;

[0012] The lip reading recognition model structure is lightweighted using a model compression algorithm.

[0013] As a preferred solution for the lightweight lip reading recognition method, the data preprocessing is performed using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence, specifically including:

[0014] The RetinaFace tracker is used to detect faces and the face key feature points are located through the face alignment network. The loss function of the RetinaFace tracker is:

[0015]

[0016] Where λ1, λ2, and λ3 correspond to the loss weights of face detection, bounding box regression, key point regression, and dense regression, respectively; p i represents the predicted class probability, represents the true category label; t i represents the predicted bounding box parameters, represents the true bounding box parameters; l i Represents the predicted facial key point position, Indicates the real facial key point position; v i represents the predicted 3D position of face pixels, Indicates the true three-dimensional position of face pixels; L cls is the classification loss function, which is used to measure the difference between the predicted category and the true category; L box is the bounding box regression loss function, which is used to measure the difference between the predicted bounding box and the true bounding box; L pts is the face key point regression loss function, which is used to measure the difference between the predicted key point position and the actual key point position; L mesh is the dense regression branch loss function, which is used to measure the difference between the predicted 3D position of face pixels and the actual position;

[0017] The face is registered to the mean face of the training set, and the rotation and size differences are eliminated through grayscale transformation and affine transformation;

[0018] Centered on the lip, a 96×96 bounding box is used to crop the lip ROI, normalize it, and compress it into a binary file format of a Numpy array.

[0019] As a preferred solution of the lightweight lip reading recognition method, the obtained lip video sequence data is subjected to a data enhancement algorithm to expand the training data set, specifically including:

[0020] Randomly crop the lip ROI;

[0021] Flip the video frame horizontally with a probability of 50%;

[0022] Continuously mask N frames, 0≤N≤N max ,N is sampled by uniform distribution, and the masked frame is replaced by the mean frame;

[0023] Use the Mixup method to generate new samples:

[0024]

[0025] Where, λ~Beta(α,α); x i 、x j are the feature parts of two original input vectors randomly selected from the training data, y i 、y j is the corresponding label part; is the feature of the generated new sample, is the label of the generated new sample.

[0026] As a preferred solution for the lightweight lip reading recognition method, the enhanced training dataset is used to extract features from the lip movement video sequence using a 3DConv+2DResnet convolutional neural network, specifically including:

[0027] A 5×7×7 convolution kernel is used to perform 3D spatiotemporal convolution to extract continuous frame visual information. The output feature map dimension is (B, C, T, H, W); where B represents the number of video sequences in each batch, C represents the number of channels in the frame image, T represents the number of frames in each lip movement video sequence, H represents the height of the frame image, and W represents the width of the frame image.

[0028] The feature map is reduced to (B*T, C, H, W) through dimension merging. The reduced features are input into the 2DResnet-18 residual network, and after four residual blocks and an average pooling layer, fine-grained spatiotemporal feature information is output.

[0029] In the 2DResnet-18 residual network, each residual block is downsampled through a 1×1 convolutional layer, the number of channels is doubled block by block, and the size of the feature map is halved.

[0030] As a preferred solution for the lightweight lip reading recognition method, the feature vectors obtained by extracting features from the lip movement video sequence are sequence modeled using the Conformer temporal modeling network to obtain a lip reading recognition model, specifically including:

[0031] Project the features output by the 2DResnet-18 into a 256-dimensional space, and then inject the transformed features into the relative position information;

[0032] The features are projected into a 2048-dimensional space through the first feed-forward layer in the Conformer module, and then pass through a ReLU activation function, a Dropout layer, and a second linear layer. The second linear layer projects the output into a 256-dimensional space.

[0033] The global dependencies between elements are modeled through the self-attention module in the Conformer module, the output is cascaded to a 256-dimensional space, and then projected to a lower 64-dimensional space;

[0034] Capturing local patterns through the convolutional neural network branch in the Conformer module;

[0035] The vector output by the convolution process is layer-normalized and dropout-operated through the second feed-forward layer in the Conformer module, and connected through the residual network;

[0036] Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a linear layer.

[0037] As a preferred solution for lightweight lip reading recognition, the relative position information is encoded using sine-cosine functions:

[0038]

[0039] Where d is the position encoding dimension, pos is the position in the frame sequence; i is the index of the element in the position encoding vector, which is used to determine the position encoding calculation method of different dimensions; pe positive The position encoding vector representing the positive relative position, pe negative Position encoding vector representing negative relative positions.

[0040] As a preferred solution for lightweight lip reading recognition, the multi-head attention mechanism is used to calculate global dependencies:

[0041]

[0042] Where, d kare the dimensions of the Q and K matrices; Q is the query vector, K is the key vector, and V is the value vector; the softmax function is used to convert the scores into probability distributions so that the sum of the attention weights is 1; QK T Calculate the similarity between the query vector and the key vector.

[0043] As a preferred solution for the lightweight lip reading recognition method, a classification network is used to decode the visual speech of the lip reading recognition model obtained through sequence modeling to obtain the language and text corresponding to the video to be recognized, specifically:

[0044] The 256-dimensional features output by the Conformer are averaged through the fully connected layer, and the category probability is calculated by the softmax function:

[0045]

[0046] Where C is the total number of categories; z i is the unnormalized score of the i-th category node output by the classification network, Softmax(z i ) is the probability value of the i-th category node after calculation by the softmax function. The softmax function converts the output value of the multi-classification into a probability distribution with a distribution range of [0,1] and a probability sum of 1.

[0047] As a preferred solution for lightweight lip reading recognition, a model compression algorithm is used to lightweight the lip reading recognition model structure. Specifically, knowledge distillation technology is used to transfer teacher model knowledge through soft labels. The distillation loss function is:

[0048]

[0049] Where T is the distillation temperature, N is the number of categories; x is the input vector of the sample, y is the true value label of the sample; θ represents the neural network parameter; f i (x; θ) represents the logistic regression value of sample x in the i-th category; P(y|x; θ, T) represents the probability that sample x belongs to category y given the parameters θ and the distillation temperature T.

[0050] The present application also provides a lightweight lip reading recognition device, which adopts the above-mentioned lightweight lip reading recognition method, including:

[0051] A video acquisition unit, used to obtain the original video to be identified;

[0052] A data preprocessing unit, configured to perform data preprocessing on the acquired original video by using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence;

[0053] A data enhancement unit, configured to expand a training data set using a data enhancement algorithm on the obtained lip video sequence data;

[0054] A feature extraction unit is used to extract features from the lip movement video sequence using the enhanced training data set using a 3DConv+2DResnet convolutional neural network;

[0055] A model building unit is used to perform sequence modeling on the feature vectors obtained by extracting features from the lip movement video sequence using a Conformer temporal modeling network to obtain a lip reading recognition model;

[0056] A decoding unit, configured to decode the visual speech of the lip reading recognition model obtained through sequence modeling using a classification network to obtain the language and text corresponding to the video to be recognized;

[0057] The model compression unit is used to perform lightweight processing on the lip reading recognition model structure using a model compression algorithm.

[0058] As a preferred solution of the lightweight lip reading recognition device, in the data preprocessing unit:

[0059] The RetinaFace tracker is used to detect faces and the face key feature points are located through the face alignment network. The loss function of the RetinaFace tracker is:

[0060]

[0061] Where λ1, λ2, and λ3 correspond to the loss weights of face detection, bounding box regression, key point regression, and dense regression, respectively; p i represents the predicted class probability, represents the true category label; t i represents the predicted bounding box parameters, represents the true bounding box parameters; l i Represents the predicted facial key point position, Indicates the real facial key point position; v i represents the predicted 3D position of face pixels, Indicates the true three-dimensional position of face pixels; L cls is the classification loss function, which is used to measure the difference between the predicted category and the true category; L box is the bounding box regression loss function, which is used to measure the difference between the predicted bounding box and the true bounding box; L pts is the face key point regression loss function, which is used to measure the difference between the predicted key point position and the actual key point position; L mesh is the dense regression branch loss function, which is used to measure the difference between the predicted 3D position of face pixels and the actual position;

[0062] The face is registered to the mean face of the training set, and the rotation and size differences are eliminated through grayscale transformation and affine transformation;

[0063] Centered on the lip, a 96×96 bounding box is used to crop the lip ROI, normalize it, and compress it into a binary file format of a Numpy array.

[0064] As a preferred solution of the lightweight lip reading recognition device, in the data enhancement unit:

[0065] Randomly crop the lip ROI;

[0066] Flip the video frame horizontally with a probability of 50%;

[0067] Continuously mask N frames, 0≤N≤N max ,N is sampled by uniform distribution, and the masked frame is replaced by the mean frame;

[0068] Use the Mixup method to generate new samples:

[0069]

[0070] Where, λ~Beta(α,α); x i 、x j are the feature parts of two original input vectors randomly selected from the training data, and yi and yj are the corresponding label parts; is the feature of the generated new sample, is the label of the generated new sample.

[0071] As a preferred solution of the lightweight lip reading recognition device, in the feature extraction unit:

[0072] A 5×7×7 convolution kernel is used to perform 3D spatiotemporal convolution to extract continuous frame visual information. The output feature map dimension is (B, C, T, H, W); where B represents the number of video sequences in each batch, C represents the number of channels in the frame image, T represents the number of frames in each lip movement video sequence, H represents the height of the frame image, and W represents the width of the frame image.

[0073] The feature map is reduced to (B*T, C, H, W) through dimension merging. The reduced features are input into the 2DResnet-18 residual network, and after four residual blocks and an average pooling layer, fine-grained spatiotemporal feature information is output.

[0074] In the 2DResnet-18 residual network, each residual block is downsampled through a 1×1 convolutional layer, the number of channels is doubled block by block, and the size of the feature map is halved.

[0075] As a preferred solution for a lightweight lip reading recognition device, in the model building unit:

[0076] Project the features output by the 2DResnet-18 into a 256-dimensional space, and then inject the transformed features into the relative position information;

[0077] The features are projected into a 2048-dimensional space through the first feed-forward layer in the Conformer module, and then pass through a ReLU activation function, a Dropout layer, and a second linear layer. The second linear layer projects the output into a 256-dimensional space.

[0078] The global dependencies between elements are modeled through the self-attention module in the Conformer module, the output is cascaded to a 256-dimensional space, and then projected to a lower 64-dimensional space;

[0079] Capturing local patterns through the convolutional neural network branch in the Conformer module;

[0080] The vector output by the convolution process is layer-normalized and dropout-operated through the second feed-forward layer in the Conformer module, and connected through the residual network;

[0081] Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a linear layer.

[0082] As a preferred solution for the lightweight lip reading recognition device, in the model building unit, the relative position information is encoded by the sine-cosine function:

[0083]

[0084]

[0085] Where d is the position encoding dimension, pos is the position in the frame sequence; i is the index of the element in the position encoding vector, which is used to determine the position encoding calculation method of different dimensions; pe positive The position encoding vector representing the positive relative position, pe negative Position encoding vector representing negative relative positions.

[0086] As a preferred solution for a lightweight lip reading recognition device, in the model building unit, global dependencies are calculated through a multi-head attention mechanism:

[0087]

[0088] Where, d k are the dimensions of the Q and K matrices; Q is the query vector, K is the key vector, and V is the value vector; the softmax function is used to convert the scores into probability distributions so that the sum of the attention weights is 1; QKT Calculate the similarity between the query vector and the key vector.

[0089] As a preferred solution of the lightweight lip reading recognition device, in the decoding unit:

[0090] The 256-dimensional features output by the Conformer are averaged through the fully connected layer, and the category probability is calculated by the softmax function:

[0091]

[0092] Where C is the total number of categories; z i is the unnormalized score of the i-th category node output by the classification network, Softmax(z i ) is the probability value of the i-th category node after calculation by the softmax function. The softmax function converts the output value of the multi-classification into a probability distribution with a distribution range of [0,1] and a probability sum of 1.

[0093] As a preferred solution for the lightweight lip reading recognition device, the model compression unit adopts knowledge distillation technology to transfer the teacher model knowledge through soft labels. The distillation loss function is:

[0094]

[0095] Where T is the distillation temperature, N is the number of categories; x is the input vector of the sample, y is the true value label of the sample; θ represents the neural network parameter; f i (x; θ) represents the logistic regression value of sample x in the i-th category; P(y|x; θ, T) represents the probability that sample x belongs to category y given the parameters θ and the distillation temperature T.

[0096] From the above, it can be seen that the technical solution provided by this application, using the 3DConv+2DResnet convolutional neural network and the Conformer temporal modeling network, can effectively extract lip movement video features, retain local and global information, improve the recognition accuracy of long sentence commands, and better handle complex command content. Through a variety of data enhancement strategies, such as random cropping, flipping, time masking and Mixup methods, the training data set is expanded, the data diversity is improved, and the model learns more robust features to adapt to different scenarios. Model compression methods such as knowledge distillation are used to reduce the number of model parameters and reduce computing resource requirements. While maintaining accuracy, the response speed is improved, making it easier to deploy on resource-constrained devices such as mobile terminals. The smart home control mobile APP developed based on this lip reading recognition algorithm can realize command control of smart home physical devices and third-party mobile phone applications, bringing users a more convenient and intelligent life experience, especially suitable for public places, noisy environments and scenes such as people with speech disorders. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0098] Figure 1 A schematic diagram of the flow of a lightweight lip reading recognition method provided in an embodiment of the present application;

[0099] Figure 2 A schematic diagram of the overall framework of lip reading recognition provided in an embodiment of the present application;

[0100] Figure 3 Schematic diagram of a feature extraction network based on 3DConv+2DResnet provided in an embodiment of the present application;

[0101] Figure 4 A schematic diagram of a Conformer-based timing modeling network provided in an embodiment of the present application;

[0102] Figure 5 Schematic diagram of the Conformer block provided in an embodiment of the present application;

[0103] Figure 6 Schematic diagram of the knowledge distillation method provided in an embodiment of the present application;

[0104] Figure 7 A schematic diagram of the overall software architecture provided for the embodiments of the present application;

[0105] Figure 8 A schematic diagram of the software application architecture provided in the embodiment of the present application;

[0106] Figure 9 Design an ER diagram for the database of the embodiment of the present application;

[0107] Figure 10 A schematic diagram of the architecture of a lightweight lip reading recognition device provided in an embodiment of the present application;

[0108] Figure 11 This is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0109] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0110] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meanings understood by persons having ordinary skills in the field to which this application belongs. The words "include" or "comprise" and the like used in the embodiments of this application mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects.

[0111] In related technologies, command recognition is mainly divided into gesture command recognition, voice command recognition, and lip reading command recognition. Gesture command recognition is very effective on small command sets. However, with the continuous expansion of gesture sets, gesture design and the scalability of gesture recognition systems have encountered greater challenges, and it has also made learning and memorizing gestures more difficult, thus limiting the widespread application of gesture command recognition technology. Compared with gesture command recognition, since commands are given in natural language, voice command recognition and lip reading command recognition have inherent advantages on large-scale data sets. In the field of voice command recognition, voice assistants are widely used in people's lives. They help people more conveniently access smartphone functions, control smart homes, and control autonomous driving. Although voice input has brought great convenience to people, there are still three major limitations that hinder its practical usability.

[0112] First, due to privacy and security risks, voice commands are not the preferred option in public settings. People may feel embarrassed speaking to their smartphones in front of others. Second, accurate speech recognition can be difficult in noisy environments. Finally, issuing voice commands is impossible for people with speech impairments. Compared to speech recognition, lip reading recognition, because it requires no sound, offers advantages in protecting user privacy, adhering to social norms, assisting speech recognition in noisy environments, and addressing issues affecting people with speech impairments. A user study of silent voice input applications on mobile devices focused specifically on user attitudes regarding social acceptance and error tolerance. The results showed that people perceive silent voice input as more socially acceptable than voice input and are willing to tolerate more errors to maintain privacy and security. This study addresses the issue of command interaction on smartphones. While voice input offers rich expressiveness and enables readily available interactions, it is easily affected by ambient noise, and due to privacy concerns, people often avoid using voice recognition. Silent voice input, on the other hand, is immune to ambient noise, protects user privacy in public settings, and ensures that users can better adhere to social norms. This application focuses on lip reading command recognition and conducts in-depth research and algorithm application in the field of smart home command control.

[0113] In the related art, the existing lip reading recognition model needs to be improved in terms of sequence modeling capabilities. After using three-dimensional convolution and two-dimensional residual networks to extract features of lip movement sequences at the front end, the model that currently performs best on limited corpus public datasets uses a temporal convolutional network (TCN) for sequence modeling. However, in actual needs, the instruction content often contains long sentences, so a network with stronger sequence modeling capabilities is needed to replace the temporal convolutional network. In addition, as the prediction effect of the lip reading recognition algorithm continues to improve, the model itself has become more and more complex and large, often requiring more computing resources and time to process data and generate results, resulting in increased computing costs and the need to rely on high-performance hardware devices to support its operation. Lightweight models can improve the model response speed while maintaining the original recognition accuracy, reduce storage space requirements, and allow it to run on lower-performance hardware, so that it can be more widely used in mobile devices including smartphones and tablets.

[0114] Taking the above into consideration, this application proposes a lightweight lip reading recognition method. By replacing the temporal convolutional network with a larger-scale Conformer temporal modeling network with stronger global feature extraction capabilities, the model's temporal modeling capabilities for long sentences are improved, thereby further improving the model's recognition accuracy. By using a pruning model compression method to reduce the number of parameters in the front-end and back-end networks, the response time and resource usage are reduced while ensuring the original performance of the model as much as possible. Below, the technical solution of this application is explained in detail through specific embodiments.

[0115] See also Figure 1 , this application provides a lightweight lip reading recognition method, including:

[0116] Step S101: Use the terminal camera acquisition device to obtain the original video to be identified.

[0117] In step S101, the main goal is to use the terminal device to collect the video to be recognized, and then the lip reading recognition model predicts the language text based on the video to be recognized.

[0118] Specifically, the videos to be recognized can be recorded by a device (such as a mobile phone or tablet) with a speaker, or they can be excerpts from news media or television programs. The video should include the speaker's face but not the audio. During pre-training, the videos to be recognized primarily come from the public LRW dataset and a dataset of real-person recordings made in the laboratory.

[0119] Step S102: pre-process the acquired original video through face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence.

[0120] In step S102, the main purpose is to extract the lip ROI in the original video, obtain the corresponding lip video sequence, and use its compressed representation as the input of the model.

[0121] The lip video sequence and its compressed representation are obtained according to the original video to be identified, including:

[0122] Use the RetinaFace tracker to detect faces and the face alignment network to detect feature points, automatically locating key facial features such as eyes, nose tip, mouth corners, eyebrows, and contour points of various facial parts;

[0123] Use the face alignment network to locate facial feature points;

[0124] Register the face to the mean face of the training set to eliminate rotation and scale differences;

[0125] Perform matrix transformations such as grayscale transformation and affine transformation on facial feature points;

[0126] According to the lip position, a 96×96 bounding box is used to crop the lip ROI in the original video to obtain the required lip movement video sequence;

[0127] Each frame of the lip movement video sequence is normalized by subtracting the mean and dividing it by the standard deviation of the training set. Finally, the lip movement video sequence is compressed and compressed into a binary file format of a Numpy array.

[0128] In this embodiment, after obtaining the original video to be recognized, data preprocessing is required. The main steps of data preprocessing include face tracking detection, facial feature point detection, face registration, grayscale transformation, affine transformation and other matrix transformations, and finally cropping it into a lip ROI sequence, which is normalized and compressed.

[0129] Specifically, face tracking detection is implemented using the RetinaFace tracker, a robust pixel-level face detector that uses a multi-task learning strategy to simultaneously predict the face score, face bounding box, five facial landmarks (left eye center, right eye center, nose tip, left and right mouth corners), and the 3D position and correspondence of each face pixel. The overall RetinaFace architecture consists of three main components: a feature pyramid, a context module, and a loss module. The specific process includes:

[0130] Features are extracted through the Resnet residual network of the feature pyramid, and then the feature vector is input into the 1×1 convolution adjustment channel, upsampled and added, and finally three feature vectors are output;

[0131] The three feature vectors are respectively input into convolution layers with kernel sizes of 3×3, 5×5, and 7×7 (where 5×5 and 7×7 convolutions are replaced by multiple 3×3 convolutions) to increase the receptive field, thereby introducing more contextual information. The three feature vectors output by the context module are then used to obtain the prediction results, where the prediction results include determining whether the prior box contains an object, face box regression, and face key point regression;

[0132] The loss function of the RetinaFace method includes classification loss (determining whether the frame contains a face), face frame regression loss, face key point regression loss, and loss caused by the dense regression branch (comparing the position differences of the five facial feature points of the face image before and after encoding and decoding). Among them, the loss function of the RetinaFace tracker is:

[0133]

[0134] Where λ1, λ2, and λ3 correspond to the loss weights of face detection, bounding box regression, key point regression, and dense regression, respectively; p i represents the predicted class probability, represents the true category label; t i represents the predicted bounding box parameters, represents the true bounding box parameters; l i Represents the predicted facial key point position, Indicates the real facial key point position; v i represents the predicted 3D position of face pixels, Indicates the true three-dimensional position of face pixels; L cls is the classification loss function, which is used to measure the difference between the predicted category and the true category; L box is the bounding box regression loss function, which is used to measure the difference between the predicted bounding box and the true bounding box; L pts is the face key point regression loss function, which is used to measure the difference between the predicted key point position and the actual key point position; L mesh It is a dense regression branch loss function, which is used to measure the difference between the predicted 3D position of face pixels and the true position.

[0135] In this embodiment, facial feature points are located using a face alignment network (FAN), which is built based on the human pose estimation framework HourGlass. The specific steps include:

[0136] Input the 2D image containing the face, stack the 4-layer HourGlass architecture, and replace all the bottleneck layers with a new parallel hierarchical multi-scale structure to output a heat map of the 2D image;

[0137] Through the 2D-to-3D-FAN network, the RGB image of the given image and the heat map of the 2D image are input, and the corresponding 3D heat map is output, that is, the 2D facial annotation of the given image is converted into 3D facial annotation.

[0138] Face registration refers to registering the face to the mean face of the training set to eliminate rotation and size differences;

[0139] Grayscale conversion compresses the three RGB channels of an image into a single channel. This is an important method for image enhancement. It can increase image contrast, making the image clearer and more distinct. In this example, a weighted average method is used to convert a color image into a grayscale image.

[0140] In this embodiment, affine transformation involves performing matrix transformation operations such as translation, rotation, and scaling on the image to convert the original image into the target image. This example uses affine transformation to achieve face registration, and performs transformations such as rotation and translation to obtain the target video frame transformation matrix and the feature point transformation matrix.

[0141] Image cropping uses a 96×96 bounding box to crop the lip ROI from the lip video sequence, resulting in a lip movement video sequence. During cropping, the average coordinates of the facial landmarks are used as the center, and cropping is performed in the four directions of top, bottom, left, and right to obtain a 96×96 square area containing the lips, thereby obtaining a lip movement video sequence containing the lip ROI. Finally, the lip movement video sequence is normalized and compressed. Each frame of the lip movement video sequence is normalized by subtracting the mean and dividing it by the standard deviation of the training set. Finally, the lip movement video sequence is compressed and compressed into a binary file format of a Numpy array to serve as the input content of the model.

[0142] Step S103: using a data enhancement algorithm to expand the training data set of the obtained lip video sequence data.

[0143] In step S103, a variety of data enhancement strategies are used to improve the diversity of data and the robustness of the model. The data enhancement strategies in step S103 are described in detail below:

[0144] Optionally, the data augmentation strategy is used to expand the model's training data and help the model learn more robust features, including:

[0145] Randomly crop the lip ROI of the training data;

[0146] Flip all frames in the lip video with 50% probability;

[0147] In each round of training, N frames are masked continuously, where N is a time series sampled from 0 to N_max using a uniform distribution;

[0148] For each batch of samples S, they are randomly shuffled into S', and then one sample A and one sample B are randomly sampled from S and S' respectively, and linear combinations are performed to create new training samples.

[0149] In a possible embodiment, after obtaining the binary file of the lip movement video sequence to be identified, the video data is first enhanced. The main method of data enhancement is to first normalize the pixel values ​​of the video frames of the training data with 0 as the mean and 255 as the standard deviation, and then perform random cropping and random flipping operations in sequence, and then normalize the cropped and flipped video frames with 0.421 as the mean and 0.165 as the standard deviation, and then use the time masking method to perform data enhancement. In addition, during the training process, the two input video sequences and their corresponding labels are randomly linearly combined by the Mixup method to create an enhanced training data instance. The random cropping, random flipping, time masking and Mixup methods mainly used in the data enhancement process described in this example are described in detail below.

[0150] Among them, random cropping means cropping a patch of size 88×88 from a random position of the lip ROI of size 96×96 during the training process. This cropping method can reduce the input size of the image, reduce the computational complexity of the neural network, increase the diversity of the data set, and reduce excessive attention to a specific local area in the image, thereby improving the robustness and generalization ability of the model. Random flipping means that during the training process, a video sequence is selected with a probability of 0.5 and all frames of the video sequence are horizontally flipped, thereby increasing the amount of training data and improving the generalization ability of the model. Temporal masking means that during the training process, N consecutive frames are masked for each video sequence, where N is between 0 and N. max A sequence of samples sampled using a uniform distribution between the frames (the maximum temporal masking step used in this example is 15 frames), where each mask frame is replaced by the mean frame of the video sequence. This data augmentation method aims to improve the robustness of the model to small video clips with missing frames.

[0151] In one possible embodiment, the Mixup method is used to randomly select two original input vectors (x i ,y i ), (x j ,y j ), generate new samples by weighted linear interpolation Right now:

[0152]

[0153] Where λ follows the Beta(α,α) distribution. In this example, the order of the original input vectors of each batch S is disrupted to obtain a second "batch" S', and samples (xi ,y i ) and (x j ,y j ) and set α to 0.4. This method achieves linear behavior of the deep model across training samples. From the perspective of Occam's razor, linearity is a good inductive bias. Therefore, through this hybrid operation, the generalization of the model is enhanced.

[0154] Step S104: Using the enhanced training data set to extract features from the lip movement video sequence using the 3DConv+2DResnet convolutional neural network.

[0155] In step S104, the spatiotemporal feature information of the lip movement video sequence is finally extracted through 3D spatiotemporal convolution, dimensionality reduction of 3D vectors to 2D vectors, and input of spatiotemporal features into a 2D Resnet-18 network. This step is described in detail below:

[0156] In step S104, optionally, performing feature extraction on the lip movement video sequence based on the 3DConv+2DResnet convolutional neural network includes:

[0157] A 5×7×7 convolution kernel is used to perform 3D spatiotemporal convolution to effectively extract the visual information of the lip area in consecutive frames and obtain a feature map.

[0158] Reduce the dimension of the feature map from three dimensions (B, C, T, H, W) to two dimensions (B*T, C, H, W), where B represents the number of video sequences in each batch, C represents the number of channels in a frame image, T represents the number of frames in each lip movement video sequence, H represents the height of a frame image, and W represents the width of a frame image. This dimensionality reduction is achieved by multiplying the batch size in the feature vector by the number of frames in each video sequence and merging them into one dimension.

[0159] The spatiotemporal features are input into the 2DResnet-18 residual network for feature extraction and dimensionality reduction to obtain fine-grained spatiotemporal feature information.

[0160] In one possible embodiment, the input video sequence vector first undergoes a 3D convolution operation to obtain the spatiotemporal information of the lip movement video sequence. Three-dimensional Max pooling is then used to reduce the dimensionality of the feature map. Subsequently, the pooled feature vector is reduced from 3D to 2D. Finally, the spatiotemporal feature vector is input into a 2DResnet-18 residual network to extract fine-grained spatiotemporal feature information. The following details the feature extraction process in the front-end module of the lip reading recognition algorithm described in this example.

[0161] During the 3D convolution operation, the batch number, number of channels, number of video frames, image height, and image width of the training data are first obtained. A 3D convolution operation is then performed on the input data, which is 88×88 in length and width. A convolution kernel of 5×7×7 is used, with both the horizontal and vertical sliding steps set to 2, without skipping any samples or color channels. Subsequently, the feature vectors are batch-normalized based on the statistical mean and variance of the training data batch, and the mean and variance parameters are updated. To preserve the features of the entire training data, the mean and variance parameters are updated for each batch of data training, using the following formula:

[0162] mean new =(1-momentum)*mean+momentum*mean t

[0163] val new =(1-momemtum)*val+momentum*val t

[0164] Among them, mean new Indicates the mean value after parameter update for this batch. new Indicates the variance after parameter update for this batch. Momentum indicates the ratio of mean and variance data saved by the network. In this example, the momentum parameter is set to 0.1. Mean indicates the historical mean saved by the network. t Indicates the new mean of the statistics of this batch. val indicates the historical variance saved by the network. t Indicates the new variance of this batch of statistics.

[0165] After batch normalization, the relu activation function is used to help the network fit nonlinear mappings, enhance network sparsity, reduce the computational complexity of the backpropagation process, and effectively prevent the occurrence of gradient vanishing. Finally, the feature map is input into a 3D MAX pooling layer with a convolution kernel size of 1×3×3 to reduce the feature map size and the output data size to 22×22, simplifying the network complexity. In the process of reducing the feature vector from three dimensions to two dimensions, this example sets the number of video sequences per batch B to 32, the number of channels C of the frame image to 64, the number of frames T of each lip movement video sequence to 59, the height H of the frame image to 88, and the width W of the frame image to 88. First, the number of channels C and the number of video frames T in the feature vector are swapped, and then the batch size B and the number of video frames T are multiplied and merged into a dimension B*T to achieve feature map dimensionality reduction.

[0166] Among them, the network depth of the 2DResnet-18 residual network is 18 layers (the weight layer of the network, including the pooling layer, activation function and linear layer), such as Figure 3 As shown, the first layer is the three-dimensional convolution and dimensionality reduction process, and then passes through 4 residual blocks and an average pooling layer, each of which contains two BasicBlock modules, and each BasicBlock module consists of 2 convolution layers with a convolution kernel size of 3*3. When passing through the first residual block, it is necessary to pass through 4 convolution layers, where the convolution kernel size of each convolution layer is 3*3, the step size is set to 1, the number of output channels is 64, and the output data size is 22×22. The first residual block does not change the data size and the number of channels. When passing through the second residual block, it is first necessary to downsample through a convolution layer with a convolution kernel size of 1*1 to reduce the dimensionality of the previous residual block, and then pass through 4 3*3 convolution layers, the number of output channels is 128, and the output data size is 11×11. The number of output channels of this residual block is doubled, and the output data size is halved. When passing through the third residual block, it also goes through the 1*1 convolution layer for downsampling, and the output of the second residual block is reduced in dimensionality. Then it goes through 4 3*3 convolution layers, with the number of output channels being 256 and the output data size being 6×6. The number of output channels of this residual block is doubled, and the output data size is halved. When passing through the fourth residual block, it also goes through the downsampling process, and then goes through 4 3*3 convolution layers, with the number of output channels being 512 and the output data size being 3×3. The number of output channels of this residual block is doubled, and the output data size is halved. Finally, it goes through an average pooling layer to perform an average pooling operation on each channel of the input tensor, compressing the height and width of the input tensor to 1, and extracting the global average of the input features. Finally, the high-dimensional tensor of the input features is flattened into a one-dimensional column vector.

[0167] Step S105: The feature vector obtained by extracting features from the lip movement video sequence is subjected to sequence modeling using a Conformer temporal modeling network to obtain a lip reading recognition model.

[0168] In step S105, the feature vectors extracted by the algorithm front end are sequentially modeled using the network Conformer structure. This structure fully utilizes the advantages of the CNN and Transformer mechanisms to maximize the acquisition of global and local features. This step is described in detail below:

[0169] In step S105, optionally, the step of performing sequence modeling using a Conformer temporal modeling network based on the spatiotemporal feature information extracted by the visual front-end network to maximize the retention of local features and global representation includes:

[0170] The position embedding module projects the features output by the 2DResnet-18 into a 256-dimensional space, and then further injects the transformed features into relative position information;

[0171] The features are projected into a higher 2048-dimensional space through the first feed-forward layer in the Conformer module, and then pass through a ReLU activation function, a Dropout layer, and a second linear layer. The second linear layer projects the output into a 256-dimensional space.

[0172] The global dependencies between elements are modeled through the self-attention module in the Conformer module, the output is cascaded to a 256-dimensional space, and then projected to a lower 64-dimensional space;

[0173] Local patterns are effectively captured through the convolutional neural network branch in the Conformer module.

[0174] The vector output by the convolution process is subjected to layer normalization and dropout operations through the second feed-forward layer in the Conformer module, and is connected through a residual network.

[0175] Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a linear layer, where the total number of categories is 500.

[0176] In a possible embodiment, the using of the self-attention module to model the global dependency relationship between elements includes:

[0177] Mapping the query to a set of key-value pairs through the attention map of the module, where the attention map focuses on different positions of the input temporal features;

[0178] The attention function is executed in parallel on each attention head, and the outputs of multiple heads are cascaded and output into a 256-dimensional space, and then projected again into the final 64-dimensional space.

[0179] Specifically, such as Figure 2 As shown, the feature vector extracted by the front-end feature extraction network is input into the Conformer network to process the time series information, so as to better learn the long-term dependency relationship in the lip movement video sequence. Figure 4 As shown in the figure, the feature vector of the input image first passes through a position embedding module to embed the relative position information of the frame in the sequence. It then passes through 12 Conformer modules, further extracting temporal features through CNN and Transformer networks, and finally passes through a fully connected layer to obtain the predicted output vector. The following details the composition and execution of the position embedding module and Conformer module described in this example.

[0180] The role of the position embedding module is to record global information and embed the relative position information of the frame in the sequence into the input feature vector. Figure 4 As shown in the Positional Encoding module, the 512-dimensional feature vector output by the front end is first projected to a 256-dimensional feature vector through the fully connected layer, and then the position information of the frame in the sequence is encoded through the position encoding module. The position encoding vector is represented by pe, and the relative position information is divided into positive relative position and negative relative position. The positive relative position indicates that the query vector Q is to the right of the key value vector K, and the negative relative position indicates that the query vector Q is to the left of the key value vector. Finally, the positive relative position vector and the negative relative position vector are connected as the final position encoding vector and embedded into the input vector. The position information is obtained by the following calculation formula:

[0181]

[0182]

[0183] Where d is the position encoding dimension, pos is the position in the frame sequence; i is the index of the element in the position encoding vector, which is used to determine the position encoding calculation method of different dimensions; pe positive The position encoding vector representing the positive relative position, pe negative A position encoding vector representing a negative relative position. In this example, the d value is set to 256, 2i represents an even dimension, and 2i+1 represents an odd dimension. There are two advantages to using the above formula to calculate position information. First, when there are sentences longer than all the sentences in the training set in actual applications, the position information of the extra positions can be calculated according to the formula. Second, it can help the model calculate the relative position more easily. For example, for a fixed-length interval l, pe(pos+l) can be calculated from pe(pos) to get l, because sin(A+B)=sinAcosB+cosAsinB, and cos(A+B)=cosAcosB-sinAsinB.

[0184] After embedding position information into the input vector, the next step is a stack of 12 identical Conformer modules. Each Conformer module consists of a first feed-forward layer, a self-attention module, a convolutional module, and a second feed-forward layer stacked in sequence. The following details the components and execution of the Conformer module.

[0185] In the Conformer module, the first feed-forward layer and the second feed-forward layer have the same composition. Figure 5As shown, the input feature vector first passes through the LayerNorm normalization method to convert the input feature vectors of each layer of neurons into feature vectors with the same mean and variance, which can speed up the convergence process. Then, it passes through the first linear layer to project the 256-dimensional input feature vector into a higher 2048-dimensional space. It then passes through the Rectified Linear Unit (ReLU) activation function, followed by a Dropout layer to ignore a certain number of neurons with a probability of 0.1. Finally, it passes through the second linear layer to project the feature vector dimension from 2048 dimensions to 256 dimensions. In addition, half-step residuals are used for connection in each feedforward layer to avoid gradient disappearance and effectively solve the problem of multi-layer network training.

[0186] Subsequently, the multi-head attention module models the global dependencies of the input video sequence. This example uses the multi-head attention mechanism to execute the attention function in parallel on each head, projecting the input vector into a lower 64-dimensional space through different linear projections, and finally cascading the outputs of multiple attention heads into a 256-dimensional space. This example sets the number of attention heads to 4. The self-attention mechanism transforms the input feature vector into a query vector Q, a key vector K, and a value vector V through a fully connected operation, and then calculates the attention matrix after obtaining the Q, K, and V matrices. It is calculated using the following formula:

[0187]

[0188] where d k is the dimension of Q and K matrices. In this example, d k To prevent the inner product from being too large, the inner product of the Q matrix and the K matrix transpose needs to be divided by

[0189] The convolution module then extracts local features from the input video sequence. The input vector first passes through a one-dimensional point-by-point convolution layer with a kernel of 1 and a stride of 1, with 256 input channels and 512 output channels. It then passes through a one-dimensional gated linear unit (GLU), which uses a convolution operation. Its primary goal is to compress all information before the current moment in the time window. The output gate controls the content output based on the position of the temporal information, achieving the desired output effect. The core expression of the algorithm is as follows:

[0190]

[0191] Among them, X represents the network input of this layer, h lThe input vector then passes through a deep convolutional neural network with a convolution kernel of 31 and a stride of 1. Batch normalization is then performed to fix the mean and variance of the input. This is followed by a swish activation layer to help the network fit the nonlinear mapping. The input vector then passes through a one-dimensional pointwise convolution layer again, followed by layer normalization to ensure that the inputs share the same mean and variance. This accelerates model training and prevents overfitting and vanishing gradients.

[0192] Subsequently, the vector output by the convolution process is subjected to layer normalization and Dropout operation (the probability is set to 0.1 in this example) through the second feed-forward layer, and is connected through the residual network.

[0193] Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a fully connected layer, where the total number of categories is 500.

[0194] Step S106: Using a classification network, the lip reading recognition model obtained through sequence modeling is decoded into visual speech to obtain the language and text corresponding to the video to be recognized.

[0195] In step S106, the fully connected layer is used as a decoder to output the category probability of the word to be classified through the softmax function.

[0196] In step S106, the tensor output from step S105 is normalized on the first dimension using the softmax function, that is, the output values ​​of the multi-classification are converted into a probability distribution with a distribution range of [0, 1] and a probability sum of 1. The calculation formula of the Softmax method is as follows:

[0197]

[0198] Where C is the total number of categories; z i is the unnormalized score of the i-th category node output by the classification network, Softmax(z i ) is the probability value of the i-th category node after calculation by the softmax function. The softmax function converts the output value of the multi-classification into a probability distribution with a distribution range of [0,1] and a probability sum of 1.

[0199] Then, the maximum value in the normalized tensor is taken out, and based on its index and label list, the language and text corresponding to the lip movement video sequence to be identified are finally returned.

[0200] Step S107: Using a model compression algorithm to perform lightweight processing on the lip reading recognition model structure.

[0201] In step S107, since the lip reading recognition model has a large number of parameters, it requires a high level of computing resources, which results in a high computational cost and requires high-performance hardware to support its operation. However, in actual applications, users often require the model to have a faster response speed, and the space and resource limitations of mobile devices require the model to occupy less storage space and computing resources. Therefore, the model needs to be lightweight, with fewer parameters and a streamlined structure, so that the accuracy of the original model can be maintained as much as possible while meeting the actual application requirements. This step is described in detail below:

[0202] In step S107, optionally, the model result is lightweighted using a model compression method, including:

[0203] The knowledge distillation method is used to compress the model's time series modeling network, reduce the loss of network performance, and improve the model's response speed.

[0204] In this example, a lip reading recognition model is trained using knowledge distillation, which reduces the number of model parameters and runtime, enabling it to run on mobile devices. The following details the module structure and execution process of the knowledge distillation method described in this example.

[0205] Knowledge distillation refers to transferring the knowledge of a complex teacher model to a small student model. Figure 6 As shown in Figure 2, the teacher sample generates a soft label through model inference, and the soft label guides the model training, forcing the model to make similar predictions for samples of the same category. For sample X, let x be the input vector of sample X and y be the true value label of sample X. Its predicted distribution is as follows:

[0206]

[0207] Where T is the distillation temperature, N is the number of categories; x is the input vector of the sample, y is the true value label of the sample; θ represents the neural network parameter; f i (x; θ) represents the logistic regression value of sample x in category i; P(y|x; θ, T) represents the probability that sample x belongs to category y given the parameters θ and the distillation temperature T. Training the model through knowledge distillation can reduce the size of the lip reading recognition model and improve its recognition accuracy to a certain extent.

[0208] In summary, the present application utilizes a terminal camera acquisition device to obtain the original video to be identified; the obtained original video is pre-processed through face tracking detection and face feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence; the obtained lip video sequence data is expanded with a data enhancement algorithm to a training data set; the enhanced training data set is used to extract features from the lip movement video sequence using a 3DConv+2DResnet convolutional neural network; the feature vector obtained by feature extraction of the lip movement video sequence is sequence modeled using a Conformer temporal modeling network to obtain a lip reading recognition model; the lip reading recognition model obtained through sequence modeling is decoded into visual speech using a classification network to obtain the language and text corresponding to the video to be identified; the lip reading recognition model structure is lightweight using a model compression algorithm. The present application utilizes a 3DConv+2DResnet convolutional neural network and a Conformer temporal modeling network to effectively extract lip movement video features, retain local and global information, improve the recognition accuracy of long sentence instructions, and better handle complex instruction content. Through various data augmentation strategies, such as random cropping, flipping, temporal masking, and Mixup, the training dataset is expanded and data diversity is improved, allowing the model to learn more robust features and adapt to different scenarios. Model compression methods such as knowledge distillation are used to reduce the number of model parameters and computing resource requirements, thereby improving response speed while maintaining accuracy and facilitating deployment on resource-constrained devices such as mobile devices. A smart home control mobile app developed based on this lip reading recognition algorithm can control smart home physical devices and third-party mobile applications, providing users with a more convenient and intelligent living experience. It is particularly suitable for use in public places, noisy environments, and for people with speech impairments.

[0209] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0210] It should be noted that the above describes some embodiments of the present application. In some cases, the actions or steps described can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0211] Based on the same inventive concept, corresponding to any of the above embodiments and methods, the present application also provides a smart home command control mobile terminal APP based on a lightweight lip reading recognition algorithm, which can implement the lip reading recognition command control method described in any of the above embodiments. The development process includes:

[0212] Complete the design and development of the mobile APP through steps such as demand analysis, architecture design, database design, coding testing, and integrated deployment;

[0213] Deploy the data preprocessing program and the lightweight lip reading recognition model on the mobile APP;

[0214] The prediction results of the model are used to control the smart home physical devices and third-party mobile applications through mobile APP.

[0215] In this example, a smart home control app was developed by sequentially performing requirements analysis, architecture design, database design, coding and testing, and integrated deployment to achieve intelligent control of home appliances using a lip reading command recognition method. The following details the execution of these steps in this example.

[0216] The purpose of demand analysis is to understand user needs and clarify the specific functions that the software should implement. Through extensive research on the functional characteristics of current smart home devices, combined with people's current demand for home device command control, the smart home control app includes the following functional modules:

[0217] System management module: including user management, permission management, role management, etc.

[0218] Basic data management module: including equipment information management, home information management (living room, bedroom, etc.), environmental parameter management (temperature, humidity, etc.) and other functions.

[0219] Command recognition module: includes lip reading input, lip reading recognition, visual display and other functions.

[0220] Device control module: Based on the results of lip reading recognition, control related devices to meet user needs.

[0221] Connection control module: includes the function of scanning or identifying surrounding devices. Users can add home devices through operations such as Wi-Fi, code scanning and model search.

[0222] Scene setting module: By customizing and combining any instructions in a scene, users can implement a series of device control operations with just one instruction.

[0223] The overall architecture is as follows Figure 7 As shown in the figure, the software functions described in the above demand analysis are further systematized and modularized. Figure 7As can be seen, the software structure is divided into the basic support layer, the process execution layer, and the data presentation layer. The basic support layer includes the system management module and the basic data module. The system management module includes user management, role management, and permission management. Users can set roles for family members and assign control permissions for certain devices. The basic data module manages device information, command content, home information, environmental parameters, and third-party applications. Device information includes the operating status (e.g., on / off) and attribute information (e.g., AC temperature, washing mode, etc.) of common household devices, such as washing machines, air conditioners, and lights. Command content refers to the user's lip reading commands to control home devices, allowing users to control the devices according to the commands. Home information includes information about the room and its devices (e.g., which devices are in the living room). Environmental parameters include information such as temperature, humidity, lighting, and air quality. Third-party applications refer to applications other than the software itself, such as music and audiobooks. In addition to controlling home devices, users can also use commands to control third-party applications, such as "open music" or "open audiobook." The process execution layer includes the lip reading command recognition module, the home device control module, the connection setup module, and the scene setup module. The lip reading command recognition module consists of three parts: lip reading input, lip reading recognition, and result display. Lip reading input involves using a camera to record a video of the speaker and performing data preprocessing on the input video through the aforementioned processes: face tracking, feature point detection, lip cropping, and normalization. Lip reading recognition involves running the model and performing model inference. Result display involves displaying the model's output command content in text form on the app interface, providing feedback to the user. The home device control module consists of two parts: device query and device control. After the model recognizes the user command, the system queries the relevant devices based on the command, arranges these devices into a queue, and sequentially queries the current status of the devices to execute the user command. If the current status cannot execute the command or the device is damaged, the system will provide information feedback or an abnormality alarm. Device control utilizes technologies such as the Internet of Things to control device status and attribute values ​​according to user needs. The connection control module helps users add devices through methods such as Wi-Fi, scanning device QR codes, Bluetooth connections, or searching for device models. The scene setting module includes scene information settings, trigger condition settings, and command execution settings. The scene information setting includes the scene name (such as going home, going out, etc.), scene icon, and scene location (such as living room, bedroom, etc.). The trigger condition setting refers to the user selecting a preset instruction (such as going home). When the software recognizes the user's trigger instruction, it executes a series of instruction operations. The execution instruction setting means that the user customizes a series of instructions to be executed. The data presentation layer includes two parts: the system portal and data visualization. The system portal includes two modules: common services and message notifications. Common services refer to devices or third-party software that users frequently use. Message notifications include device abnormality alarms, calendar reminders, etc.Data visualization includes visualization of device operating status, environmental parameters, and card-based display of home appliances / devices. Users can directly see device operating status and environmental parameter information on the device or home card.

[0224] Software application architecture such as Figure 8 As shown. This software follows the MVC design pattern and divides the business logic into three parts: the interaction layer, the business logic layer, and the data access layer. After the interaction layer receives a request from another system or user, it parses the request through a processor and then passes the task to be completed to the backend code. After the backend code completes the processing, it feeds the result back to the interaction layer, which then responds with the result. The business logic layer is responsible for the specific implementation of each functional module of the system. It not only accepts access requests from the upper layer, but also calls the data access layer to complete the corresponding functions and feeds the processing results back to the interaction layer. The data access layer is called by the business logic layer and returns the accessed data to the business logic layer after completing the corresponding data operations through the MyBatis framework of the MySQL database.

[0225] Database design includes two parts: entity relationship diagram design and database table design. In the entity relationship diagram, if Figure 9 This section shows the core components of the entity relationship diagram, including entities such as devices, users, and command content. The main entity relationships include those between devices and commands, commands and third-party applications, and scenarios and commands. Because there are many database tables, the following uses third-party application tables and device tables as examples to illustrate database table design.

[0226] Field Name type Is it allowed to be empty? describe APP_ID varchar(30) no Primary key, auto-increment APP_NAME varchar(30) no Third-party application name APP_CODE varchar(30) no Third-party application ID COMMENT varchar(100) yes describe CREATE_TIME date yes Creation time CREATER varchar(30) yes Founder

[0227] Table 1 Third-party application table

[0228] Field Name type Is it allowed to be empty? describe DEVICE_ID varchar(30) no Primary key, auto-increment DEVICE_NAME varchar(30) no Device Name DEVICE_CODE varchar(30) no Equipment Number DEVICE_STATUS varchar(100) yes Device Status DEVICE_FEATURE varchar(30) Device Function CREATE_TIME date yes Creation time CREATER varchar(30) yes Founder

[0229] Table 2 Equipment table

[0230] Coding testing involves implementing software functionality through program coding and system testing. This software's front-end uses HTML, CSS, and JavaScript, utilizing the Vue mobile component library Vant to implement front-end page design and event response. The back-end primarily uses Java, utilizing SpringCloud, MyBatis, and SpringMVC frameworks for implementation. Testing is completed sequentially through module functionality testing and overall testing.

[0231] This software is an Android-based product. Code building, compilation, and execution, as well as real-device simulation and performance testing, are completed using the Android Studio integrated development environment. Finally, the software code is packaged into an APK file, which is the installation package for an Android application and contains all of the application's components, resources, and code. Users install this software on their Android-based mobile devices by downloading and installing the APK file.

[0232] The smart home control APP in the above embodiment is used to enable the computer to execute the lightweight lip reading recognition method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0233] See also Figure 10 Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a lightweight lip reading recognition device, which adopts the lightweight lip reading recognition method of the above-mentioned embodiment, including:

[0234] The video acquisition unit 100 is used to obtain the original video to be identified;

[0235] The data preprocessing unit 200 is used to perform data preprocessing on the acquired original video by using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence;

[0236] A data enhancement unit 300 is configured to expand a training data set using a data enhancement algorithm for the obtained lip video sequence data;

[0237] A feature extraction unit 400 is configured to extract features from the lip movement video sequence using the enhanced training dataset using a 3DConv+2DResnet convolutional neural network;

[0238] The model building unit 500 is used to perform sequence modeling on the feature vectors obtained by extracting features from the lip movement video sequence using a Conformer temporal modeling network to obtain a lip reading recognition model;

[0239] The decoding unit 600 is used to decode the visual speech of the lip reading recognition model obtained through sequence modeling using a classification network to obtain the language and text corresponding to the video to be recognized;

[0240] The model compression unit 700 is used to perform lightweight processing on the lip reading recognition model structure using a model compression algorithm.

[0241] In this embodiment, in the data preprocessing unit 200:

[0242] The RetinaFace tracker is used to detect faces and the face key feature points are located through the face alignment network. The loss function of the RetinaFace tracker is:

[0243]

[0244] Where λ1, λ2, and λ3 correspond to the loss weights of face detection, bounding box regression, key point regression, and dense regression, respectively; p i represents the predicted class probability, represents the true category label; t i represents the predicted bounding box parameters, represents the true bounding box parameters; l i Represents the predicted facial key point position, Indicates the real facial key point position; v i represents the predicted 3D position of face pixels, Indicates the true three-dimensional position of face pixels; L cls is the classification loss function, which is used to measure the difference between the predicted category and the true category; L box is the bounding box regression loss function, which is used to measure the difference between the predicted bounding box and the true bounding box; L pts is the face key point regression loss function, which is used to measure the difference between the predicted key point position and the actual key point position; L mesh is the dense regression branch loss function, which is used to measure the difference between the predicted 3D position of face pixels and the actual position;

[0245] The face is registered to the mean face of the training set, and the rotation and size differences are eliminated through grayscale transformation and affine transformation;

[0246] Centered on the lip, a 96×96 bounding box is used to crop the lip ROI, normalize it, and compress it into a binary file format of a Numpy array.

[0247] In this embodiment, in the data enhancement unit 300:

[0248] Randomly crop the lip ROI;

[0249] Flip the video frame horizontally with a probability of 50%;

[0250] Continuously mask N frames, 0≤N≤N max ,N is sampled by uniform distribution, and the masked frame is replaced by the mean frame;

[0251] Use the Mixup method to generate new samples:

[0252]

[0253] Where, λ~Beta(α,α); x i 、x j are the feature parts of two original input vectors randomly selected from the training data, y i 、y j is the corresponding label part; is the feature of the generated new sample, is the label of the generated new sample.

[0254] In this embodiment, in the feature extraction unit 400:

[0255] A 5×7×7 convolution kernel is used to perform 3D spatiotemporal convolution to extract continuous frame visual information. The output feature map dimension is (B, C, T, H, W); where B represents the number of video sequences in each batch, C represents the number of channels in the frame image, T represents the number of frames in each lip movement video sequence, H represents the height of the frame image, and W represents the width of the frame image.

[0256] The feature map is reduced to (B*T, C, H, W) through dimension merging. The reduced features are input into the 2DResnet-18 residual network, and after four residual blocks and an average pooling layer, fine-grained spatiotemporal feature information is output.

[0257] In the 2DResnet-18 residual network, each residual block is downsampled through a 1×1 convolutional layer, the number of channels is doubled block by block, and the size of the feature map is halved.

[0258] In this embodiment, in the model building unit 500:

[0259] Project the features output by the 2DResnet-18 into a 256-dimensional space, and then inject the transformed features into the relative position information;

[0260] The features are projected into a 2048-dimensional space through the first feed-forward layer in the Conformer module, and then pass through a ReLU activation function, a Dropout layer, and a second linear layer. The second linear layer projects the output into a 256-dimensional space.

[0261] The global dependencies between elements are modeled through the self-attention module in the Conformer module, the output is cascaded to a 256-dimensional space, and then projected to a lower 64-dimensional space;

[0262] Capturing local patterns through the convolutional neural network branch in the Conformer module;

[0263] The vector output by the convolution process is layer-normalized and dropout-operated through the second feed-forward layer in the Conformer module, and connected through the residual network;

[0264] Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a linear layer.

[0265] In this embodiment, the model building unit 500 encodes the relative position information using the sine-cosine function:

[0266]

[0267] Where d is the position encoding dimension, pos is the position in the frame sequence; i is the index of the element in the position encoding vector, which is used to determine the position encoding calculation method of different dimensions; pe positive The position encoding vector representing the positive relative position, pe negative Position encoding vector representing negative relative positions.

[0268] In this embodiment, in the model building unit 500, the global dependency is calculated by a multi-head attention mechanism:

[0269]

[0270] Where, d k are the dimensions of the Q and K matrices; Q is the query vector, K is the key vector, and V is the value vector; the softmax function is used to convert the scores into probability distributions so that the sum of the attention weights is 1; QK T Calculate the similarity between the query vector and the key vector.

[0271] In this embodiment, in the decoding unit 600:

[0272] The 256-dimensional features output by the Conformer are averaged through the fully connected layer, and the category probability is calculated by the softmax function:

[0273]

[0274] Where C is the total number of categories; z i is the unnormalized score of the i-th category node output by the classification network, Softmax(z i ) is the probability value of the i-th category node after calculation by the softmax function. The softmax function converts the output value of the multi-classification into a probability distribution with a distribution range of [0,1] and a probability sum of 1.

[0275] In this embodiment, the model compression unit 700 uses knowledge distillation technology to transfer teacher model knowledge through soft labels. The distillation loss function is:

[0276]

[0277] Where T is the distillation temperature, N is the number of categories; x is the input vector of the sample, y is the true value label of the sample; θ represents the neural network parameter; f i (x; θ) represents the logistic regression value of sample x in the i-th category; P(y|x; θ, T) represents the probability that sample x belongs to category y given the parameters θ and the distillation temperature T.

[0278] The device of the above embodiment is used to implement a corresponding lightweight lip reading recognition method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0279] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a lightweight lip reading recognition method as described in any of the above-mentioned embodiments is implemented.

[0280] Figure 11 8 shows a more specific hardware structure diagram of an electronic device provided in this embodiment. The device may include: a processor 810, a memory 820, an input / output interface 830, a communication interface 840, and a bus 850. The processor 810, the memory 820, the input / output interface 830, and the communication interface 840 are connected to each other within the device via the bus 850.

[0281] The processor 810 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0282] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 820 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0283] The input / output interface 830 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0284] The communication interface 840 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0285] The bus 850 comprises a pathway for transmitting information between the various components of the device (eg, the processor 810 , the memory 820 , the input / output interface 830 , and the communication interface 840 ).

[0286] It should be noted that although the above device only shows the processor 810, the memory 820, the input / output interface 830, the communication interface 840, and the bus 850, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0287] The electronic device of the above embodiment is used to implement a corresponding lightweight lip reading recognition method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0288] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute a lightweight lip reading recognition method as described in any of the above embodiments.

[0289] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0290] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute a lightweight lip reading recognition method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0291] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0292] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0293] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0294] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the scope of protection claimed. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A lightweight lip reading recognition method, characterized in that: include: Get the original video to be identified; The obtained original video is preprocessed using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence; Expanding the training data set using a data enhancement algorithm for the obtained lip video sequence data; The enhanced training dataset is used to extract features from the lip movement video sequence using a 3DConv+2DResnet convolutional neural network; The feature vectors obtained by feature extraction of lip movement video sequences are used to perform sequence modeling using the Conformer temporal modeling network to obtain a lip reading recognition model. Using a classification network to decode the visual speech of the lip reading recognition model obtained through sequence modeling to obtain the language and text corresponding to the video to be recognized; The lip reading recognition model structure is lightweighted using a model compression algorithm.

2. The lightweight lip reading recognition method according to claim 1, characterized in that: The data preprocessing is performed by using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence, specifically including: The RetinaFace tracker is used to detect faces and the face key feature points are located through the face alignment network. The loss function of the RetinaFace tracker is: Where λ1, λ2, and λ3 correspond to the loss weights of face detection, bounding box regression, key point regression, and dense regression, respectively; p i represents the predicted class probability, represents the true category label; t i represents the predicted bounding box parameters, represents the true bounding box parameters; l i Represents the predicted facial key point position, Indicates the real facial key point position; v i represents the predicted 3D position of face pixels, Indicates the true three-dimensional position of face pixels; L cls is the classification loss function, which is used to measure the difference between the predicted category and the true category; L box is the bounding box regression loss function, which is used to measure the difference between the predicted bounding box and the true bounding box; L pts is the face key point regression loss function, which is used to measure the difference between the predicted key point position and the actual key point position; L mesh is the dense regression branch loss function, which is used to measure the difference between the predicted 3D position of face pixels and the actual position; The face is registered to the mean face of the training set, and the rotation and size differences are eliminated through grayscale transformation and affine transformation; Centered on the lip, a 96×96 bounding box is used to crop the lip ROI, normalize it, and compress it into a binary file format of a Numpy array.

3. The lightweight lip reading recognition method according to claim 2, characterized in that: The obtained lip video sequence data is subjected to a data enhancement algorithm to expand the training data set, specifically including: Randomly crop the lip ROI; Flip the video frame horizontally with a probability of 50%; Continuously mask N frames, 0≤N≤N max ,N is sampled by uniform distribution, and the masked frame is replaced by the mean frame; Use the Mixup method to generate new samples: Where, λ~Beta(α,α); x i 、x j are the feature parts of two original input vectors randomly selected from the training data, y i 、y j is the corresponding label part; is the feature of the generated new sample, is the label of the generated new sample.

4. The lightweight lip reading recognition method according to claim 1, characterized in that: The enhanced training dataset is used to extract features from the lip movement video sequence using a 3DConv+2DResnet convolutional neural network, specifically including: A 5×7×7 convolution kernel is used to perform 3D spatiotemporal convolution to extract continuous frame visual information. The output feature map dimension is (B, C, T, H, W); where B represents the number of video sequences in each batch, C represents the number of channels in the frame image, T represents the number of frames in each lip movement video sequence, H represents the height of the frame image, and W represents the width of the frame image. The feature map is reduced to (B*T, C, H, W) through dimension merging. The reduced features are input into the 2DResnet-18 residual network, and after four residual blocks and an average pooling layer, fine-grained spatiotemporal feature information is output. In the 2DResnet-18 residual network, each residual block is downsampled through a 1×1 convolutional layer, the number of channels is doubled block by block, and the size of the feature map is halved.

5. The lightweight lip reading recognition method according to claim 4, characterized in that: The feature vectors obtained by extracting features from the lip movement video sequence are subjected to sequence modeling using a Conformer temporal modeling network to obtain a lip reading recognition model, specifically including: Project the features output by the 2DResnet-18 into a 256-dimensional space, and then inject the transformed features into the relative position information; The features are projected into a 2048-dimensional space through the first feed-forward layer in the Conformer module, and then pass through a ReLU activation function, a Dropout layer, and a second linear layer. The second linear layer projects the output into a 256-dimensional space. The global dependencies between elements are modeled through the self-attention module in the Conformer module, the output is cascaded to a 256-dimensional space, and then projected to a lower 64-dimensional space; Capturing local patterns through the convolutional neural network branch in the Conformer module; The vector output by the convolution process is layer-normalized and dropout-operated through the second feed-forward layer in the Conformer module, and connected through the residual network; Finally, by calculating the mean of the feature vector in the first dimension, the feature vector is mapped from the 256-dimensional space to the 500-dimensional space through a linear layer.

6. The lightweight lip reading recognition method according to claim 5, characterized in that: Relative position information is encoded by sine-cosine functions: Where d is the position encoding dimension, pos is the position in the frame sequence; i is the index of the element in the position encoding vector, which is used to determine the position encoding calculation method of different dimensions; pe positive The position encoding vector representing the positive relative position, pe negative Position encoding vector representing negative relative positions.

7. The lightweight lip reading recognition method according to claim 5, characterized in that: Calculate global dependencies through multi-head attention mechanism: Where, d k are the dimensions of the Q and K matrices; Q is the query vector, K is the key vector, and V is the value vector; the softmax function is used to convert the scores into probability distributions so that the sum of the attention weights is 1; QK T Calculate the similarity between the query vector and the key vector.

8. The lightweight lip reading recognition method according to claim 1, characterized in that: The classification network is used to decode the visual speech of the lip reading recognition model obtained through sequence modeling to obtain the language and text corresponding to the video to be recognized, specifically: The 256-dimensional features output by the Conformer are averaged through the fully connected layer, and the category probability is calculated by the softmax function: Where C is the total number of categories; z i is the unnormalized score of the i-th category node output by the classification network, Softmax(z i ) is the probability value of the i-th category node after calculation by the softmax function. The softmax function converts the output value of the multi-classification into a probability distribution with a distribution range of [0,1] and a probability sum of 1.

9. The lightweight lip reading recognition method according to claim 1, characterized in that: The lip reading recognition model structure is lightweighted using a model compression algorithm. Specifically, knowledge distillation technology is used to transfer teacher model knowledge through soft labels. The distillation loss function is: Where T is the distillation temperature, N is the number of categories; x is the input vector of the sample, y is the true value label of the sample; θ represents the neural network parameter; f i (x; θ) represents the logistic regression value of sample x in the i-th category; P(y|x; θ, T) represents the probability that sample x belongs to category y given the parameters θ and the distillation temperature T.

10. A lightweight lip reading recognition device, using the lightweight lip reading recognition method according to any one of claims 1 to 9, characterized in that: include: A video acquisition unit, used to obtain the original video to be identified; A data preprocessing unit, configured to perform data preprocessing on the acquired original video by using face tracking detection and facial feature point detection technology to obtain a lip video sequence and a compressed representation of the lip video sequence; A data enhancement unit, configured to expand a training data set using a data enhancement algorithm on the obtained lip video sequence data; A feature extraction unit is used to extract features from the lip movement video sequence using the enhanced training data set using a 3DConv+2DResnet convolutional neural network; A model building unit is used to perform sequence modeling on the feature vectors obtained by extracting features from the lip movement video sequence using a Conformer temporal modeling network to obtain a lip reading recognition model; A decoding unit, configured to decode the visual speech of the lip reading recognition model obtained through sequence modeling using a classification network to obtain the language and text corresponding to the video to be recognized; The model compression unit is used to perform lightweight processing on the lip reading recognition model structure using a model compression algorithm.

Images