A lip speech recognition method based on partial convolution and multi-scale feature extraction

Abstract

This invention discloses a lip-reading recognition method based on partial convolution and multi-scale feature extraction, comprising the following steps: S1: Obtain a word-level lip-reading recognition video dataset and perform facial landmark detection on each video; S2: Preprocess the lip-reading recognition dataset to obtain a cropped, continuously grayscale lip-reading video sequence; S3: Divide the data in the preprocessed dataset; S4: Apply data augmentation methods such as horizontal flipping, mixup, temporal masking, and random cropping to the training set data; S5: Extract short-term temporal and spatial features from the data in the training set; S6: Extract short-term and long-term temporal features from the lip-reading recognition dataset using a Res2-TCN network; S7: Input the sequence used for result classification into a fully connected layer for classification; S8: Calculate the model loss based on the lip-reading recognition results. This invention ensures that the model can achieve more accurate lip-reading recognition with very low computational cost.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, specifically to a lip-reading recognition method based on partial convolution and multi-scale feature extraction. Background Technology

[0002] Lip reading, also known as visual speech recognition, refers to the ability to identify spoken content solely by the movement of a speaker's lips in the absence of a spoken signal. Its research encompasses multiple fields, including pattern recognition, computer vision, speech recognition, and natural language processing. In recent years, lip reading has become a hot topic, with broad application prospects in areas such as speech recognition in noisy environments, assisting the hearing impaired, human-computer interaction, and public safety.

[0003] Due to the rapid development of deep learning technology and the emergence of convolutional neural networks, deep learning-based lip reading recognition has gradually replaced traditional lip reading methods. While these models have achieved good classification results, their high computational cost and demanding hardware resources make them difficult to implement on mobile devices or resource-constrained devices. Recent research on lip reading focuses on improving its efficiency to enable wider application. Their approach primarily involves replacing the original feature extraction network with lightweight networks such as MobileNet, ShuffleNet, EfficientNet, and GhostNet. By using deep convolutions, group convolutions, and Ghost modules instead of standard convolutional layers for feature extraction, significant reductions in computational and storage costs are achieved. However, the high memory access volume makes these operations inefficient, and these lightweight networks often employ additional data operations such as concatenation and channel shuffling. The latency introduced by these operations is fatal for small models. Therefore, achieving truly efficient networks requires ensuring both reduced computational cost and high inference speed. Summary of the Invention

[0004] The purpose of this invention is to provide a lip-reading recognition method based on partial convolution and multi-scale feature extraction, thereby solving the aforementioned technical problems.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A lip-reading recognition method based on partial convolution and multi-scale feature extraction includes the following steps:

[0007] S1: Obtain a word-level lip reading recognition video dataset, perform facial landmark detection on each video, and extract the lip region image to obtain the lip region set of each video frame, which constitutes the lip reading recognition dataset D.

[0008] S2: Preprocess the lip-reading dataset to obtain a cropped, continuously grayscale lip-reading video sequence;

[0009] S3: Divide the data in the preprocessed dataset to obtain the training set, validation set, and test set;

[0010] S4: Use data augmentation methods such as horizontal flipping, Mixup, temporal masking, and random cropping on the training set data;

[0011] S5: Short-term temporal and spatial features are extracted from the training set data using 3D convolution and an improved FasterNet network.

[0012] S6: The Res2-TCN network is used to extract short-term and long-term temporal features from the lip-reading dataset to obtain sequences for result classification;

[0013] S7: Input the sequence used for result classification into the fully connected layer for classification to obtain the lip reading recognition result;

[0014] S8: Calculate the model's loss based on the lip-reading results, input the data from the validation set into the model, and continuously adjust the model's parameters. Training is complete when the loss function value is minimized. The formula for calculating the cross-entropy loss function is as follows:

[0015]

[0016] Where p is the probability prediction value, q = 1 - p, and y is the true label value.

[0017] As a further aspect of the present invention: Step 1 specifically includes the following steps:

[0018] S11: Use the Dlib facial landmark detection algorithm to extract 68 facial landmarks from each video sequence in the lip-reading dataset and align each frame with the reference average face;

[0019] S12: To reduce interference from other facial areas, a lip region of interest of size 96×96 is centrally cropped from the aligned face image of size 224×224.

[0020] As a further aspect of the present invention: step 2 specifically includes the following steps:

[0021] S21: Since color images do not provide additional assistance to recognition and increase the computational load of the model, the original video frame images are converted to grayscale to reduce the impact of color difference on recognition.

[0022] S22: Randomly crop the face image from 96×96 to 88×88.

[0023] As a further aspect of the present invention: step 4 specifically includes the following steps:

[0024] S41: Horizontally flip each video frame in the training set with a probability of 0.5;

[0025] S42: To reduce the risk of overfitting, the Mixup data augmentation method is used during training. MixUp constructs a new training sample and its corresponding label by linear interpolation between two samples.

[0026] S43: Use the TimeMask data augmentation method on the training set to randomly select a portion of the video frames in the training set and replace them with the average value of all frames;

[0027] S44: Use Cutout as an additional data augmentation method on the training set. Cutout uses a fixed-size square region, filled with all zeros, and allows the square region to be outside the image. Parts that exceed the boundary will be truncated. Cutout has two parameters: n_holes and length, where n_holes represents the number of blocks to be cut and length represents the side length of the cut square.

[0028] As a further aspect of the present invention: in step S5, the construction of the 3D convolutional and FasterNet networks specifically includes the following steps:

[0029] S51: For input data I∈R 1×t×h×w After performing a 3D convolution with a kernel size of 5×7×7, short-term spatiotemporal features are extracted, resulting in output data O∈R. c×t×h×w Where c represents the number of channels, t represents the number of frames, h represents the height, and w represents the width;

[0030] S52: For input features of 3D convolution, max pooling is used to downsample the features to reduce computation and model complexity; for input features I∈R c×t×h×w The output features can be obtained after 3D max pooling.

[0031] S53: Features after 3D max pooling need to be dimensionality compressed before being input into the 2D-FasterNet network. The output features are obtained after transformation. The time dimension and batch size are merged.

[0032] S54: The transformed 2D data is input into the FasterNet network to extract the spatial features of the lips. FasterNet consists of four stages, each stage composed of multiple FasterNet Blocks. Each FasterNet Block consists of one partial convolution (PConv) and two pointwise convolutions (PWConv). For the input features... The output feature O∈R is obtained after passing through the FasterNet network. c'×h'×w' ;

[0033] S55: The output features of the FasterNet network need to be transformed into a one-dimensional feature vector through global average pooling, for the input features I∈R c'×h'×w' The transformed one-dimensional vector O∈R is obtained. c'×t .

[0034] As a further aspect of the present invention: step S6 specifically includes the following steps:

[0035] S61: The obtained one-dimensional feature vector is input into Res2-TCN to extract the temporal features of lip movement; where Res2-TCN consists of four Res2-TCN Blocks, each Res2-TCN Block contains three sub-modules with different dilation rates, where the dilation rate d∈{1,2,5}; each sub-module consists of two temporal convolutional layers, and a one-dimensional convolution with a filter size of 1 is embedded after each temporal convolutional layer to aggregate feature information from different channels of the temporal convolutional layer;

[0036] S62: First, divide the input feature vector into s uniform feature vector subsets, and use x i Let be the subset of features, where i∈{1,2,…,s}; compared with the input feature vector, each feature subset x i They have the same size, but only 1 / s of channels; except for x s In addition, each x i Each has a corresponding one-dimensional convolution with a kernel size of 3 and an inflation rate of d, denoted as K. i (); use y i K represents i The output of (); feature subsets and K i-1 The outputs of () are summed and then fed into K. i (); In order to increase s while decreasing the parameter, x is omitted. s A 1D convolution; therefore, y i It can be written as:

[0037]

[0038] Finally, all the yi The concatenation of these features forms the output feature y, which can be expressed by the formula: y = [y1, y2, ..., y s ].

[0039] S63: For input feature I∈Rc'×t, after passing through Res2-TCN, output feature O∈Rc”×t can be obtained, where the number of input channels and output channels remains unchanged.

[0040] As a further aspect of the present invention: in step S7, the specific steps are as follows:

[0041] S71: The output features of Res2-TCN are averaged over time to obtain the output features O∈Rc”;

[0042] S72: The obtained features are input into a fully connected layer for classification. The input dimension is 512 and the output dimension is 500, where 500 is the total number of word categories.

[0043] S73: Feed the output of the fully connected layer into the SoftMax layer to obtain the final word prediction probability.

[0044] The beneficial effects of this invention are as follows: This invention is a lip-reading recognition method based on partial convolution and multi-scale feature extraction. It uses a multi-scale temporal convolutional network to perform short-term and long-term temporal modeling of the input video sequence and employs several effective data augmentation methods. This solves the problems of high computational cost and slow inference speed in lip-reading recognition models. It ensures that the model can achieve more accurate lip-reading recognition with very low computational cost. Attached Figure Description

[0045] The invention will now be further described with reference to the accompanying drawings.

[0046] Figure 1 This is a schematic diagram of the model structure of a lip-reading recognition method based on partial convolution and multi-scale feature extraction according to the present invention;

[0047] Figure 2 This is a schematic diagram of the model structure of the FasterNet network in this invention;

[0048] Figure 3 This is a comparison diagram of different TCN structures described in this invention;

[0049] Figure 4 This is a schematic diagram of the model structure of the Res2-TCN network described in this invention;

[0050] Figure 5 This is a flowchart illustrating a lip-reading recognition method based on partial convolution and multi-scale feature extraction according to the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figure 1-5 As shown, this invention is a lip-reading recognition method based on partial convolution and multi-scale feature extraction. Depending on the function, the lip-reading recognition method mainly consists of a front-end network for spatial feature extraction and a back-end network for temporal feature extraction. The proposed lip-reading recognition model structure is as follows: Figure 1 As shown;

[0053] In this embodiment, for a given video sequence, we extract 29 consecutive frames. These frames are first processed by a 3D convolution with a kernel size of 5×7×7 to extract short-term spatiotemporal features. Then, they are processed by FasterNet to extract rich spatial features to obtain the feature vector of the sequence. Finally, we use the output of the first part as the input of Res2-TCN, which extracts multi-scale temporal features of lip movement. Then, the final prediction result is obtained through the SoftMax layer.

[0054] In this invention, we use the lightweight model FasterNet for lip feature extraction. The first step is to construct the PConv module, the structure of which is as follows: Figure 2 As shown in the bottom right corner, this module utilizes feature redundancy to reduce computation and memory access. Specifically, it applies regular convolution to a portion of the input channels for spatial feature extraction, leaving the remaining channels unprocessed, and then concatenates the two as the output; for an input I∈R c×h×w Using c filters W∈R k×k Calculate the output O∈R c×h×w The number of memory accesses for a regular convolution is:

[0055] h×w×2c+k 2 ×c 2 ≈h×w×2c;

[0056] FLOPs are: h × w × k 2 ×c 2 .

[0057] As a variant of regular convolution, depthwise classifiable convolution has been widely used in building lightweight networks. Depthwise classifiable convolution mainly consists of channel-wise convolution (DWConv) and pointwise convolution (PWConv). In DWConv, each kernel convolves only one input channel and operates on only one output channel, thus keeping the input and output channels of the feature map unchanged. This operation results in lower FLOPs for DWConv compared to regular convolution; DWConv's FLOPs are only:

[0058] h×w×k 2 ×c;

[0059] Although DWConv has fewer FLOPs than regular convolution, it is common to add a PWConv before and after DWConv. First, PWConv is used to expand the width of the network. The number of channels c of DWConv or the network width is increased to c' (c'>c) to compensate for the decrease in accuracy.

[0060] For example, in MobileNetv2, the width of DWConv is increased by 6 times, and in EfficientNetv2, the width is increased by 4 times. Compared to ordinary convolutions, DWConv requires significantly more memory accesses, which correspondingly slows down the model's computation speed, a fatal flaw for I / O-bound devices. The memory accesses of DWConv are as follows:

[0061] h×w×2c'+k 2 ×c'≈h×w×2c';

[0062] For deep neural networks, the features extracted by each layer are very rich and even redundant, thus ensuring a comprehensive understanding of the input data. In this invention, we use a simple PConv module to reduce the computational cost and memory access of the model. PConv's FLOPs are only:

[0063]

[0064] Where cp represents the number of channels performing PConv, and when the block ratio is... At that time, PConv's FLOPs were only 1 / 16 of those of regular convolutions, and the number of memory accesses was only 1 / 4 of that of regular convolutions, that is:

[0065]

[0066] To fully and effectively utilize information from all channels, we incorporate pointwise convolution PWConv into the FasterNet Block. The structure of the FasterNet Block is as follows: Figure 2As shown in the bottom left corner.

[0067] Each FasterNet Block consists of one PConv and two PWConv. The input data is first processed through PConv for feature extraction, then through PWConv. Increasing the network width also compensates for the loss of accuracy. We only use normalization and activation layers between the two PWConv to maintain feature diversity and achieve lower latency. Finally, residual connections are used to reuse the input features.

[0068] We improved the residual network using FasterNet Blocks. The structure of FasterNet is as follows: Figure 2 As shown, FasterNet has four stages, each consisting of multiple FasterNet Blocks. In this invention, the FasterNetBlock is repeated [2,2,2,2]. A merging layer is used between two FasterNet Blocks for feature downsampling and to widen the network. Finally, a global average pooling layer is used to obtain the feature vector as the input to the backend temporal feature extraction network.

[0069] Extracting multi-scale temporal information is beneficial for networks to identify subtle differences in lip movements. In this invention, we use a novel multi-scale temporal convolutional network (Res2-TCN) to extract short-term and long-term lip movement information. First, we construct temporal convolutional layers. Unlike conventional multi-scale convolutions, we aim for stronger multi-scale temporal feature extraction capabilities while maintaining a similar computational cost. Specifically, we replace a set of 3×1 filters with smaller filters and use filter banks that connect different layers across stages to transfer shallow features to deeper features and extend the temporal length. Since our proposed TCN module involves residual-like connections within a single residual block, we name it Res2-TCN.

[0070] Figure 3 The diagram illustrates a standard TCN, a multi-scale TCN, and our proposed Res2-TCN. First, we uniformly divide the input feature vector into s subsets of feature vectors, using x... i Let be the subset of features, where i∈{1,2,…,s}; compared with the input feature vector, each feature subset x i They have the same size, but only 1 / s of channels. (Except for x) s In addition, each x i Each has a corresponding one-dimensional convolution with a kernel size of 3 and an inflation rate of d, denoted as K. i (); We use y i K represents i The output of (). Feature subset and K i-1The outputs of () are added and then fed into K i (); To reduce the parameters while increasing s, we omit x s 1D convolution of. Therefore, y i can be written as:

[0071]

[0072] Note that due to the existence of the horizontal cross-stage connection, each 1D convolution operation K i () may obtain temporal features from all feature subsets {x j , j < i}. Each time a feature subset passes through the 1D convolution operation, the output result can have a larger temporal receptive field than x j ; Due to this combined effect, the output of the temporal convolution layer can contain temporal information of different lengths. Finally, all the y i are concatenated as the output feature y. Expressed by the formula as:

[0073]

[0074] y = [y1, y2,..., y s ;

[0075] As Figure 4 shown, each Res2-TCNBlock contains 3 sub-modules with different dilation rates, where the dilation rate d ∈ {1, 2, 5}, and each sub-module consists of two temporal convolution layers. We embed a 1D convolution with a filter size of 1 after each temporal convolution layer to aggregate the feature information from different channels of the temporal convolution layer. For the input feature I ∈ R c×t , after passing through the Res2-TCN Block, the output feature O ∈ R c×t can be obtained, where the number of input channels and output channels remains unchanged.

[0076] Res2-TCN is composed of four Res2-TCN Blocks. For the input feature I ∈ R c×t , after passing through Res2-TCN, the output feature O ∈ R c×t can be obtained, where the number of input channels and output channels remains unchanged. Then it is fed into the temporal pooling layer and the fully connected layer to obtain the final prediction result. Res2-TCN can better distinguish the subtle movement differences of the lips, and without losing the model accuracy, the computational complexity and the number of parameters of Res2-TCN are less than 1 / 3 of the ordinary multi-scale TCN.

[0077] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A lip-reading recognition method based on partial convolution and multi-scale feature extraction, characterized in that, Includes the following steps: S1: Obtain a word-level lip reading recognition video dataset, perform facial landmark detection on each video, and extract the lip region image to obtain the lip region set of each video frame, which constitutes the lip reading recognition dataset D. S2: Preprocess the lip-reading dataset to obtain a cropped, continuously grayscale lip-reading video sequence; S3: Divide the data in the preprocessed dataset to obtain the training set, validation set, and test set; S4: Data augmentation methods such as horizontal flipping, mixup, temporal masking, and random cropping are used on the training set data; S5: Short-term temporal and spatial features are extracted from the training set data using 3D convolution and an improved FasterNet network. S6: The Res2-TCN network is used to extract short-term and long-term temporal features from the lip-reading dataset to obtain sequences for result classification; S7: Input the sequence used for result classification into the fully connected layer for classification to obtain the lip reading recognition result; S8: Calculate the model's loss based on the lip-reading results, input the data from the validation set into the model, and continuously adjust the model's parameters. Training is complete when the loss function value is minimized. The formula for calculating the cross-entropy loss function is as follows: ； Where p is the probability prediction value, q=1-p, and y is the true label value; In step S5, the construction of the 3D convolutional and FasterNet networks specifically includes the following steps: S51: For input data After passing through a 3D convolution with a kernel size of 5×7×7, short-term spatiotemporal features are extracted to obtain the output data. Where c represents the number of channels, t represents the number of frames, h represents the height, and w represents the width; S52: For input features of 3D convolution, max pooling is used to downsample the features to reduce computation and model complexity; for input features The output features are obtained after 3D max pooling. ; S53: Features after 3D max pooling need to be dimensionality compressed before being input into the 2D-FasterNet network. The output features are obtained after transformation. The time dimension is merged with the batch size; S54: The transformed 2D data is input into the FasterNet network to extract spatial features of the lips. FasterNet consists of four stages, each stage comprising multiple FasterNet Blocks. Each FasterNet Block consists of one partial convolution and two pointwise convolutions. For the input features... The output features are obtained after passing through the FasterNet network. ; S55: The output features of the FasterNet network need to be converted into a one-dimensional feature vector through global average pooling, while the input features... This yields a transformed one-dimensional vector. ; Step S6 specifically includes the following steps: S61: The obtained one-dimensional feature vector is input into Res2-TCN to extract temporal features of lip movement; where Res2-TCN consists of four Res2-TCN Blocks, each Res2-TCN Block contains three sub-modules with different dilation rates, where the dilation rate... Each submodule consists of two temporal convolutional layers, with a one-dimensional convolutional layer of filter size 1 embedded after each temporal convolutional layer to aggregate feature information from different channels of the temporal convolutional layer; S62: First, divide the input feature vector into s uniform feature vector subsets, and use x i To indicate, among which Compared to the input feature vector, each feature subset x i They have the same size, but only 1 / s of channels; except for x s In addition, each x i Each has a corresponding one-dimensional convolution with a kernel size of 3 and an inflation rate of d, denoted as K. i (); use y i K represents i The output of (); feature subsets and K i-1 The outputs of () are summed and then fed into K. i (); In order to increase s while decreasing the parameter, x is omitted. s A 1D convolution; therefore, y i Written as: ； Finally, all the y i The concatenation of these features forms the output feature y, which can be expressed by the formula: ; S63: For input features The output features are obtained after passing through Res2-TCN. The number of input channels and output channels remains unchanged.

2. The lip-reading recognition method based on partial convolution and multi-scale feature extraction according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11: Use the Dlib facial landmark detection algorithm to extract 68 facial landmarks from each video sequence in the lip-reading dataset and align each frame with the reference average face; S12: To reduce interference from other facial areas, a lip region of interest of size 96×96 is centrally cropped from the aligned face image of size 224×224.

3. The lip-reading recognition method based on partial convolution and multi-scale feature extraction according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21: Since color images do not provide additional assistance to recognition and increase the computational load of the model, the original video frame images are converted to grayscale to reduce the impact of color difference on recognition. S22: Randomly crop the face image from 96×96 to 88×88.

4. The lip-reading recognition method based on partial convolution and multi-scale feature extraction according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41: Horizontally flip each video frame in the training set with a probability of 0.5; S42: To reduce the risk of overfitting, the Mixup data augmentation method is used during training. MixUp constructs a new training sample and its corresponding label by linear interpolation between two samples. S43: Use the TimeMask data augmentation method on the training set to randomly select a portion of the video frames in the training set and replace them with the average value of all frames; S44: Use Cutout as an additional data augmentation method on the training set. Cutout uses a fixed-size square region, filled with all zeros, and allows the square region to be outside the image. Parts that exceed the boundary will be truncated. Cutout has two parameters: n_holes and length, where n_holes represents the number of blocks to be cut and length represents the side length of the cut square.

5. The lip-reading recognition method based on partial convolution and multi-scale feature extraction according to claim 1, characterized in that, In step S7, the specific steps are as follows: S71: The output features of Res2-TCN are subjected to time averaging to obtain the output features. ; S72: The obtained features are input into a fully connected layer for classification. The input dimension is 512 and the output dimension is 500, where 500 is the total number of categories of the word. S73: Feed the output of the fully connected layer into the SoftMax layer to obtain the final word prediction probability.

Images