A 3D facial lip-reading method based on neural networks

Abstract

This invention relates to a neural network-based 3D facial lip-reading recognition method, belonging to the field of image processing. The method involves processing and fusing RGB image information and depth map information of the face, extracting features, extracting sequence information using a temporal convolutional network, optimizing using an attention mechanism, and finally obtaining the word probability distribution for each image frame through a classifier. This invention achieves higher lip-reading recognition accuracy and, by utilizing depth image information unaffected by illumination intensity, exhibits stronger robustness.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a three-dimensional facial lip-reading method based on neural networks. Background Technology

[0002] Lip reading is a technique that identifies the content of a speaker's speech by analyzing facial and lip regions in a sequence of images, thus obtaining speech data from image data. Current lip reading methods primarily rely on processing the mouth region in two-dimensional facial images as the basis for judgment. This involves determining the location of key points in the two-dimensional lip image or directly extracting lip region image features using neural networks to obtain lip feature information for different pronunciations. However, whether using two-dimensional key points or two-dimensional images to process lip region changes, these methods only utilize two-dimensional information. With the development of depth sensing technology, obtaining two-dimensional facial images and depth data has become simpler, achievable through depth cameras, binocular stereo matching, and LiDAR. For lip reading tasks, a depth image of the facial lip region can be input along with the two-dimensional facial lip region image as a feature base. Currently, lip reading methods based on two-dimensional images are susceptible to the influence of lighting intensity, while depth images, as absolute depth information, are not affected by lighting intensity and have greater robustness. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] The technical problem to be solved by this invention is how to provide a three-dimensional facial information lip reading method based on neural networks, so as to solve the problem that lip reading methods based on two-dimensional images are affected by the intensity of light.

[0005] (II) Technical Solution

[0006] To address the aforementioned technical problems, this invention proposes a three-dimensional facial information lip-reading recognition method based on a neural network. This neural network includes a three-dimensional facial information extraction part, a temporal convolutional network, an attention mechanism module, and a classification layer.

[0007] 3D face information extraction section:

[0008] Use 3D convolutional layers to process the input RGB image sequence of a human face. and depth image sequences The process involves processing the image, where N is the number of input image frames, to obtain the preliminary features of the face RGB image and depth image. After 3D convolution processing, the image features are fused using the Add operation, and then ResNet-18 is used for feature encoding.

[0009] Temporal convolutional network part:

[0010] Temporal convolutional networks employ a combination of causal convolution and dilated convolution; the filter of a temporal convolutional network is defined as F. Temporal =(E1,E2,…,E K ), where K is the number of layers in the temporal convolutional network, and the input sequence is X = (x1, x2, ..., x...). N ), where N is the length of the input sequence, representing N input images. The input sequence is the feature sequence output by ResNet18, and d represents the dilation rate. For x in the input sequence t Features, in x t The expressions for dilated convolution and causal convolution of size d at point d are:

[0011]

[0012] Temporal convolutional networks include causal convolution and dilated convolution, and finally residual connections. The output of the 3D face information after processing by ResNet-18 is f. r The result after processing by the temporal convolutional network is:

[0013] f TCN =F Temporal (f r )+f r ;

[0014] Attention mechanism module:

[0015] After processing by the temporal convolutional network, the output temporal series features are processed by the attention mechanism module to distinguish which face image frames the feature information is concentrated on, and the output vector v of the attention mechanism module is obtained.

[0016] Category layer section:

[0017] Finally, the output vector of the attention mechanism module is fed into the softmax classifier for classification, outputting the probability distribution of each word corresponding to each frame sequence.

[0018] Furthermore, the method includes training and inference components.

[0019] Furthermore, the acquisition of training data requires obtaining RGB and depth images of the face, while also annotating human language information.

[0020] Furthermore, training data is acquired using Kinect's RGBD camera or a stereo depth camera.

[0021] Furthermore, the preliminary features of the face RGB image and depth image are as follows:

[0022] f RGB =3DConv(I RGB )

[0023] f Depth =3DConv(I Depth ).

[0024] Furthermore, in the ResNet-18 residual network, assuming the current layer is layer l, the relationship between the features of deeper layers L and the features of layer l can be expressed as follows:

[0025]

[0026] Where L is the number of layers in the ResNet, F r For Resnet_block, W i This represents the parameters of the i-th Resnet_block, where fi is the feature of the i-th layer;

[0027] The ResNet-18 processing procedure is represented by function F. res18 (x, W) res18 ), where x is the input feature, W res18 The network parameters in ResNet-18, after processing by ResNet-18, are represented as follows:

[0028] f r =F res18 (f RGB +f Depth W res18 ).

[0029] Furthermore, the temporal convolutional network has 4 layers, with a dilation rate of 1 for the first hidden layer, a dilation rate of 2 for the second hidden layer, and a dilation rate of 4 for the output layer.

[0030] Furthermore, the processing procedure of the attention mechanism in the attention mechanism module is expressed as follows:

[0031] u = tanh(w*f TCN +b)

[0032] a = softmax(u T ,u)

[0033] v=∑af TCN

[0034] Where u is the hidden layer unit of the attention mechanism module, w is the parameter of the attention mechanism module, b is the bias of the attention mechanism module, a is the attention vector, and v is the output vector of the attention mechanism module.

[0035] Furthermore, the output of the classification layer is as follows:

[0036] y = softmax(q*v + c)

[0037] Where q is the parameter of the classifier, c is the bias in the classifier, and y is the probability distribution of the word corresponding to each frame sequence in the output.

[0038] Furthermore, by recognizing the image of the lips, words and phrases are output, and the input image is processed by a neural network to obtain the probability distribution of the candidate words.

[0039] (III) Beneficial Effects

[0040] This invention proposes a three-dimensional facial lip-reading method based on neural networks. Compared with existing technologies, the proposed solution utilizes three-dimensional information of the lip region of the face to design a temporal convolutional network for three-dimensional facial feature extraction and language sequence prediction. Compared with lip-reading methods based on two-dimensional images, this invention achieves higher lip-reading accuracy. Furthermore, by utilizing depth image information that is unaffected by illumination intensity, the method of this invention exhibits stronger robustness. Attached Figure Description

[0041] Figure 1 This is a diagram of the neural network structure of the present invention;

[0042] Figure 2 This is a diagram of the temporal convolutional neural network structure of the present invention. Detailed Implementation

[0043] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0044] The purpose of this invention is to propose a three-dimensional facial lip-reading recognition method based on neural networks. This method utilizes the depth image of the lip region of the face and improves the accuracy and robustness of the lip-reading recognition method by using three-dimensional information.

[0045] A neural network-based 3D facial lip-reading method, including training and inference components.

[0046] First, the training data acquisition requires obtaining RGB images and depth images of the face, while also annotating human language information. Training data acquisition uses a camera similar to Kinect's RGBD camera or a stereo depth camera (Kinect's RGBD camera can obtain RGB images and depth maps; stereo depth cameras can also obtain RGB images and depth maps).

[0047] The neural network structure of the method of this invention is as follows: Figure 1 As shown, it includes a 3D face information extraction part, a temporal convolutional network, an attention mechanism module, and a classification layer.

[0048] 3D face information extraction section:

[0049] Use 3D convolutional layers to process the input RGB image sequence of a human face. and depth image sequences The process involves processing N, where N is the number of input image frames, to obtain preliminary features of the face's RGB image and depth image, represented as follows:

[0050] f RGB =3DConv(I RGB )

[0051] f Depth =3DConv(I Depth )

[0052] Image features after 3D convolution are fused using an Add operation (i.e., + operation) and then ResNet-18 is used for feature encoding.

[0053] In the ResNet-18 residual network, assuming the current layer is layer l, the relationship between the features of deeper layers L and the features of layer l can be expressed as follows:

[0054]

[0055] Where L is the number of layers in the ResNet, F r For Resnet_block, W i f represents the parameter of the i-th Resnet_block. i For the i-th layer features;

[0056] The ResNet-18 processing procedure is represented by function F. res18 (x,W res18 ), where x is the input feature, W res18 For network parameters in ResNet-18, Figure 1 The features in the data after ResNet-18 processing are represented as follows:

[0057] f r =F res18 (f RGB +f Depth W res18 )

[0058] Temporal convolutional network part:

[0059] Temporal convolutional networks (TCNNs) are network structures capable of processing time series data. Compared to RNNs and LSTMs, they have the advantage of lower memory consumption. TCNNs combine causal convolution and dilated convolution. Causal convolution is characterized by the fact that the value of the previous layer at time t depends only on the values ​​of the next layer at time t and earlier, representing a strict time constraint. To capture longer dependencies, holes are injected into standard convolutions to increase the receptive field. The hyperparameter `dilation_rate` is added to represent the spacing between convolutional kernels. The structure of a TCNN combining causal and dilated convolutions is shown below. Figure 2 As shown.

[0060] Define the filter of the temporal convolutional network as F. Temporal =(E1,E2,…,E K ), where K is the number of layers in the temporal convolutional network, and the input sequence is X = (x1, x2, ..., x...). N ), where N is the length of the input sequence, representing N input images. The input sequence is the feature sequence output by ResNet18, and d represents the dilation rate. For x in the input sequence t Features, in x t The expressions for dilated convolution and causal convolution of size d at point d are:

[0061]

[0062] Temporal convolutional networks include causal convolution and dilated convolution, and finally residual connections. The output of the 3D face information after processing by ResNet-18 is f. r The result after processing by the temporal convolutional network is:

[0063] f TCN =F Temporal (f r )+f r

[0064] like Figure 2 As shown, the temporal convolutional network has 4 layers, with dilation_rate = 1 for the first hidden layer, dilation_rate = 2 for the second hidden layer, and dilation_rate = 4 for the output layer.

[0065] Attention mechanism module:

[0066] After processing by the temporal convolutional network, the output temporal series features are processed by the attention mechanism module to distinguish which face image frames the feature information is concentrated on, and the output vector v of the attention mechanism module is obtained.

[0067] The processing of the attention mechanism is represented as follows:

[0068] u = tanh(w*f TCN +b)

[0069] a = softmax(u T ,u)

[0070] v=∑af TCN

[0071] Where u is the hidden layer unit of the attention mechanism module, w is the parameter of the attention mechanism module, b is the bias of the attention mechanism module, a is the attention vector, and v is the output vector of the attention mechanism module.

[0072] Category layer section:

[0073] Finally, the output vector of the attention mechanism module is fed into the softmax classifier for classification, outputting the probability distribution of each word corresponding to each frame sequence.

[0074] The output is,

[0075] y = softmax(q*v + c)

[0076] Where q represents the classifier parameters, c represents the classifier bias, and y represents the probability distribution of the words corresponding to each output frame sequence. Lip reading recognition identifies images of the lips and outputs word sentences. A neural network processes the input image to obtain the probability distribution of candidate words.

[0077] The above describes an embodiment of the present invention, a lip-reading recognition method based on three-dimensional facial information, which includes processing and fusing RGB image information and depth map information of the face, extracting features, extracting sequence information through a temporal convolutional network, optimizing using an attention mechanism, and finally obtaining the word probability distribution corresponding to each image frame through a classifier. The overall neural network structure described above is within the scope of protection of this invention.

[0078] Compared with existing technologies, the technical solution proposed in this invention utilizes the three-dimensional information of the lip region of the face to design a temporal convolutional network for three-dimensional facial information feature extraction and language sequence prediction. Compared with lip reading recognition methods based on two-dimensional images, this invention can achieve higher lip reading recognition accuracy. At the same time, by utilizing depth image information that is not affected by light intensity, the method of this invention has stronger robustness.

[0079] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional facial lip-reading recognition method based on neural networks, characterized in that, The neural network includes a 3D face information extraction part, a temporal convolutional network, an attention mechanism module, and a classification layer; 3D face information extraction section: Use 3D convolutional layers to process the input RGB image sequence of a human face. and depth image sequences The process involves processing the image, where N is the number of input image frames, to obtain the preliminary features of the face RGB image and depth image. After 3D convolution processing, the image features are fused using the Add operation, and then ResNet-18 is used for feature encoding. Temporal convolutional network part: Temporal convolutional networks employ a combination of causal convolution and dilated convolution; the filter of a temporal convolutional network is defined as F. Temporal =(E1,E2,…,E K ), where K is the number of layers in the temporal convolutional network, and the input sequence is X = (x1, x2, ..., x...). N ), where N is the length of the input sequence, representing N input images. The input sequence is the feature sequence output by ResNet18, and d represents the dilation rate. For x in the input sequence t Features, in x t The expressions for dilated convolution and causal convolution of size d at point d are: Temporal convolutional networks include causal convolution and dilated convolution, and finally residual connections. The output of the 3D face information after processing by ResNet-18 is f. r The result after processing by the temporal convolutional network is: f TCN =F Temporal (f r )+f r ; Attention mechanism module: After processing by the temporal convolutional network, the output temporal series features are processed by the attention mechanism module to distinguish which face image frames the feature information is concentrated on, and the output vector v of the attention mechanism module is obtained. Category layer section: Finally, the output vector of the attention mechanism module is fed into the softmax classifier for classification, outputting the probability distribution of each word corresponding to each frame sequence.

2. The 3D facial lip-reading recognition method based on neural networks as described in claim 1, characterized in that, This method includes training and inference components.

3. The 3D facial lip-reading recognition method based on neural networks as described in claim 2, characterized in that, The collection of training data requires obtaining RGB images and depth images of human faces, while also annotating human language information.

4. The 3D facial lip-reading recognition method based on neural networks as described in claim 3, characterized in that, Training data is acquired using Kinect's RGBD camera or a stereo depth camera.

5. The 3D facial lip-reading recognition method based on neural networks as described in claim 1, characterized in that, The initial features of the RGB and depth images of a human face are: f RGB =3DConv(I RGB ) f Depth =3DConv(I Depth )。 6. The 3D facial lip-reading method based on neural networks as described in any one of claims 1-5, characterized in that, In the ResNet-18 residual network, assuming the current layer is layer l, the relationship between the features of deeper layers L and the features of layer l can be expressed as follows: Where L is the number of layers in the ResNet, F r For Resnet_block, W i f represents the parameter of the i-th Resnet_block. i For the i-th layer features; The ResNet-18 processing procedure is represented by function F. res18 (x, W) res18 ), where x is the input feature, W res18 The network parameters in ResNet-18, after processing by ResNet-18, are represented as follows: f r =F res18 (f RGB +f Depth ,W res18 )。 7. The 3D facial lip-reading recognition method based on neural networks as described in claim 6, characterized in that, The temporal convolutional network has 4 layers, with dilation_rate = 1 for the first hidden layer, dilation_rate = 2 for the second hidden layer, and dilation_rate = 4 for the output layer.

8. The 3D facial lip-reading recognition method based on neural networks as described in claim 6, characterized in that, The attention mechanism processing procedure of the attention mechanism module is expressed as follows: u=tanh(w*f TCN +b) a=softmax(u T ,u) v=∑af TCN Where u is the hidden layer unit of the attention mechanism module, w is the parameter of the attention mechanism module, b is the bias of the attention mechanism module, a is the attention vector, and v is the output vector of the attention mechanism module.

9. The 3D facial lip-reading method based on neural networks as described in claim 8, characterized in that, The output of the classification layer is: y = softmax(q*v + c) Where q is the parameter of the classifier, c is the bias in the classifier, and y is the probability distribution of the word corresponding to each frame sequence in the output.

10. The 3D facial lip-reading method based on neural networks as described in claim 9, characterized in that, By recognizing the image of the lips, the system outputs word sentences. The input image is then processed by a neural network to obtain the probability distribution of the candidate words.

Images