A face occlusion detection method

Abstract

The present invention relates to a face occlusion detection algorithm, comprising the following steps: face image preprocessing, cutting a plurality of collected face images according to face position information and face key point coordinates to obtain a plurality of face features images as a basic data set; data augmentation, augmenting the basic data set; data labeling and division, classifying and labeling the augmented images of different categories, and using the labeled data as training samples, including a training set, a validation set and a test set; building a convolutional neural network model; setting hyperparameters, training the convolutional neural network model, determining the hyperparameters, configuring a training set for simulation training, using a validation set to test the model, observing whether the model accuracy fluctuates significantly, if the accuracy fluctuates significantly, returning to set the hyperparameters, otherwise outputting the model as a face occlusion detection model; testing the model, verifying the accuracy, using a test set to determine the model accuracy, and evaluating the generalization ability of the optimal hyperparameter training model.

Description

Technical Field

[0001] The invention relates to a face occlusion detection method and belongs to the field of face recognition. Background Art

[0002] At present, most face occlusion detection is done by extracting image features through feature extraction algorithms, and then using classifiers to classify the features. The traditional method is to use feature extraction algorithms such as SIFT (Scale-invariant Feature Transform) and HOG (Histogram of Oriented Gradient), extract features, and then process them in different ways, and then use classifiers such as support vector machines to train them to obtain a classifier model.

[0003] In recent years, the rise of deep neural networks has provided another solution. Among them, convolutional neural networks are suitable for image recognition and classification. By building a convolutional neural network, training a large amount of complete face data, learning weight parameters, extracting image features, and directly outputting classification results.

[0004] Traditional face occlusion detection uses the entire face image as input data, and the training of convolutional neural networks requires a large amount of data to learn the positions of facial features and facial feature occlusion information to achieve accuracy. The accuracy is low and the training data is large. Therefore, the present invention provides a face occlusion detection method for solving the above problem. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a face occlusion detection method, which has the characteristics of high accuracy and less model fitting training data.

[0006] The technical solution of the present invention is as follows: a face occlusion detection method, comprising the following steps: face image preprocessing, collecting a plurality of face images, traversing all face images, and performing image segmentation processing on each face image according to face position information and face key point coordinates to obtain a plurality of images of facial features as a basic data set; data augmentation, performing data augmentation on the face images in the basic data set to obtain augmented images, wherein the data augmentation methods include brightness adjustment, image angle rotation and mirror processing; data annotation and classification, classifying the augmented images, including occlusion, eyes, mouth and nose, annotating the augmented images of different categories, and using the annotated data as training samples, including a training set, a validation set and a test set; building a convolutional neural network model, wherein the convolutional neural network includes an input layer, a hidden layer and an output layer. Out of the layer, train the convolutional neural network through training samples, and output the convolutional neural network model; set hyperparameters, train the convolutional neural network model, the setting of the hyperparameters includes the initial learning rate, the size of a single batch of training data and the number of iterations of the training data, determine the hyperparameters, configure the training set data for simulation training, use the validation set to test the model accuracy, observe whether the model accuracy fluctuates significantly and verify the generalization ability of the model, if the accuracy fluctuates significantly, return to setting the hyperparameters, adjusting the learning rate, the size of a single training batch and the number of iterations of the training data, otherwise output the result as the face occlusion detection model; test the face occlusion detection model, verify the accuracy, use the test set to determine the model accuracy, and evaluate the generalization ability of the optimal hyperparameter training model; obtain the facial features information to be recognized, and use the trained face occlusion detection model to perform face occlusion detection.

[0007] Specifically, the convolutional neural network extracts image features by stacking networks, including a shallow network and a deep network. The shallow network extracts edge, texture and color features, and the deep network extracts specific contour and structural features.

[0008] Specifically, the convolutional neural network weights are shared. When the convolutional neural network calculates the convolution kernel and the feature map, it calculates with different parts of the feature map through a sliding window method, and the calculation process shares the weights.

[0009] Specifically, the image data of the convolutional neural network is divided into sub-graphs of different regions, and the sub-graphs of different regions have the same features and are detected using the same set of weights.

[0010] Specifically, the facial features information is obtained through the Dlib face detection algorithm.

[0011] Specifically, by calling the Dlib face detection algorithm, it is found that the left eye area among the facial key points is a square with a width of the distance between points 36 and 39, the right eye area is a square with a width of the distance between points 42 and 45, the nose area is a square with a frame of the distance between points 28 and 33, and the mouth area is a square with a width of points 48 and 54.

[0012] Specifically, the weights are back-propagated and updated using a loss function during training by a convolutional neural network, and the convolutional neural network extracts feature vectors. The loss function is obtained by calculating the cross entropy between the label of the feature vector and the annotated label of the augmented image through a softmax function.

[0013] Specifically, a mobilenet, vggnet or resnet network structure is used to replace the hidden layer of the convolutional neural network.

[0014] The present invention has the following beneficial effects:

[0015] 1. The present invention segments the facial features before face occlusion detection, making recognition more targeted, reducing training data and improving recognition accuracy;

[0016] 2. The present invention uses convolutional neural network to extract features, which reduces the manual feature design process and reduces labor costs compared to traditional feature engineering for feature extraction;

[0017] 3. Compared with the traditional method using SIFT and HOG feature extraction algorithms, the present invention has better generalization ability;

[0018] 4. The convolutional neural network used in the present invention has the characteristics of sparse connection and weight sharing, and is more suitable for processing image data;

[0019] 5. Compared with the fully connected network, the convolutional neural network used in the present invention uses convolutional layers and pooling layers instead of fully connected layers. Only some nodes between network layers are connected, which reduces network parameters, speeds up network training, and reduces the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of face occlusion detection of the present invention;

[0021] Figure 2 A schematic diagram of key points of a human face according to the present invention;

[0022] Figure 3 A flowchart of training a convolutional neural network according to the present invention;

[0023] Figure 4 Schematic diagram of the convolutional neural network structure of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] See also Figures 1 to 4 A face occlusion detection method comprises the following steps: face image preprocessing, collecting a plurality of face images, traversing all face images, and performing image segmentation processing on each face image according to face position information and face key point coordinates, obtaining a plurality of images of face features as a basic data set; data augmentation, performing data augmentation on the face images in the basic data set to obtain augmented images, wherein the data augmentation methods include brightness adjustment, image angle rotation and mirror processing; data labeling and classification, classifying the augmented images, including occlusion, eyes, mouth and nose, labeling the augmented images of different categories, and using the labeled data as training samples, including training set, verification set and test set; building a convolutional neural network model, wherein the convolutional neural network includes an input layer, a hidden layer and an output layer, through The convolutional neural network is trained with training samples, and a convolutional neural network model is output; hyperparameters are set to train the convolutional neural network model, wherein the hyperparameter settings include the initial learning rate, the size of a single batch of training data, and the number of iterations of the training data; the hyperparameters are determined, the training set data is configured for simulation training, and the model accuracy is tested using the validation set to observe whether the model accuracy fluctuates significantly and to verify the generalization ability of the model; if the accuracy fluctuates significantly, the hyperparameters are set, the learning rate, the single training batch size, and the number of iterations of the training data are adjusted, otherwise the result is output as a face occlusion detection model; the face occlusion detection model is tested to verify the accuracy, the model accuracy is determined using the test set, and the generalization ability of the model trained with the optimal hyperparameters is evaluated; the facial features information to be identified is obtained, and face occlusion detection is performed using the trained face occlusion detection model.

[0026] Furthermore, the convolutional neural network extracts image features by stacking networks, including a shallow network and a deep network. The shallow network extracts edge, texture and color features, and the deep network extracts specific contour and structural features.

[0027] Furthermore, the convolutional neural network weights are shared. When the convolutional neural network calculates the convolution kernel and the feature map, it calculates with different parts of the feature map through a sliding window method, and the calculation process shares the weights.

[0028] Furthermore, the image data of the convolutional neural network is divided into sub-graphs of different regions, and the sub-graphs of different regions have the same features and are detected using the same set of weights.

[0029] Furthermore, the facial features information is obtained through the Dlib face detection algorithm.

[0030] Furthermore, by calling the Dlib face detection algorithm, it is calculated that the left eye area among the facial key points is a square with a width of the distance between points 36 and 39, the right eye area is a square with a width of the distance between points 42 and 45, the nose area is a square with a frame of the distance between points 28 and 33, and the mouth area is a square with a width of points 48 and 54.

[0031] Furthermore, the weights are back-propagated and updated using a loss function during training by a convolutional neural network, and the convolutional neural network extracts feature vectors. The loss function is obtained by calculating the cross entropy between the label of the feature vector and the annotated label of the augmented image through a softmax function.

[0032] Furthermore, a mobilenet, vggnet or resnet network structure is used to replace the hidden layer of the convolutional neural network.

[0033] Furthermore, the specific structure of the convolutional neural network includes INPUT->CONV1->CONV2->MAXPOOL->CONV3->MAXPOOL->CONV4->CONV5->DENSE->OUTPUT, wherein INPUT is the input layer data, the image size is 48*48*3, CONV1 is the first convolution layer, the size is 11*11*96, the step size is 1, and CONV2 is the second convolution layer, the size is 5* 5*128, with a step size of 1, the MAXPOOL is the maximum pooling layer with a step size of 2, the CONV3 is the third convolutional layer with a size of 3*3*128 and a step size of 1, the CONV4 is the fourth convolutional layer with a size of 3*3*64 and a step size of 1, the CONV5 is the fifth convolutional layer with a size of 3*3*64 and a step size of 1, the DENSE is a dense layer with an output size of 4*1*1, and the OUTPUT is the output layer, which is the feature vector extracted by the convolutional neural network.

[0034] Furthermore, the augmented images of different categories are labeled using one-hot encoding. The DENSE outputs a 4*1*1 one-dimensional vector. The probability of each position is calculated by the softmax function. If the probability result is [0.10.1 0.7 0.1], the closest position is [0 0 1 0]. The one-hot encodings of the augmented images of different categories are occlusion [1 0 0 0], eyes [0 1 0 0], mouth [0 0 1 0], and nose [0 0 0 1].

[0035] Furthermore, the division ratio of the training set, validation set, and test set is 8:1:1.

[0036] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A face occlusion detection method, characterized in that: The following steps are involved: Face image preprocessing: collect multiple face images, traverse all face images, and perform image segmentation on each face image according to face position information and face key point coordinates to obtain multiple images of facial features as the basic data set; Data augmentation, performing data augmentation on the face images in the basic data set to obtain augmented images, wherein the data augmentation methods include brightness adjustment, image angle rotation, and mirror processing; Data labeling and classification: classifying the augmented images, including occlusion, eyes, mouth and nose, labeling the augmented images of different categories, and using the labeled data as training samples, including training sets, validation sets and test sets; Build a convolutional neural network model. The convolutional neural network includes an input layer, a hidden layer, and an output layer. Train the convolutional neural network through training samples and output the convolutional neural network model. Setting hyperparameters and training the convolutional neural network model. The hyperparameter settings include the initial learning rate, the size of a single batch of training data, and the number of iterations of the training data. Determine the hyperparameters, configure the training set data for simulation training, use the validation set to test the model accuracy, observe whether the model accuracy fluctuates significantly, and verify the generalization ability of the model. If the accuracy fluctuates significantly, return to setting the hyperparameters, adjust the learning rate, the size of a single training batch, and the number of iterations of the training data, otherwise output the result as the face occlusion detection model; Test the face occlusion detection model, verify the accuracy, use the test set to determine the model accuracy, and evaluate the generalization ability of the model trained with the optimal hyperparameters; Obtain the facial features information of the person to be identified, and use the trained face occlusion detection model to perform face occlusion detection; in: The convolutional neural network extracts image features by stacking networks, including a shallow network and a deep network. The shallow network extracts edge, texture and color features, and the deep network extracts specific contour and structural features. The image data of the convolutional neural network is divided into sub-graphs of different regions, and the sub-graphs of different regions have the same features and are detected using the same set of weights; The specific structure of the convolutional neural network model includes INPUT->CONV1->CONV2->MAXPOOL->CONV3->MAXPOOL->CONV4->CONV5->DENSE->OUTPUT, wherein INPUT is the input layer data, the image size is 48*48*3, CONV1 is the first convolution layer, the size is 11*11*96, the step size is 1, CONV2 is the second convolution layer, the size is 5*5*128, the step size is 1, MAXPOOL is the maximum pooling layer, the step size is 2, and CONV3 is the third convolution layer, the size is 3*3 *128, the step size is 1, the CONV4 is the fourth convolution layer, the size is 3*3*64, the step size is 1, the CONV5 is the fifth convolution layer, the size is 3*3*64, the step size is 1, the DENSE is a dense layer, the output size is 4*1*1, the OUTPUT is the output layer, which is the feature vector extracted by the convolutional neural network; the labeling of the augmented images of different categories is specifically labeling the augmented images of different categories using one-hot encoding, the DENSE outputs a one-dimensional vector of 4*1*1, and the probability of each position is calculated by the softmax function. If the probability result is [0.1 0.1 0.70.1], the closest position is [0 0 1 0], and the one-hot encodings of the augmented images of different categories are occlusion [10 0 0], eyes [0 1 0 0], mouth [0 0 1 0], and nose [0 0 0 1].

2. A face occlusion detection method according to claim 1, characterized in that: The convolutional neural network weights are shared. When the convolutional neural network calculates the convolution kernel and the feature map, it calculates with different parts of the feature map through a sliding window method, and the calculation process shares the weights.

3. A face occlusion detection method according to claim 1, characterized in that: Get facial features information through Dlib face detection algorithm.

4. The face occlusion detection method according to claim 3, characterized in that: By calling the Dlib face detection algorithm, it is found that the left eye area among the facial key points is a square with a width of the distance between points 36 and 39, the right eye area is a square with a width of the distance between points 42 and 45, the nose area is a square with a frame of the distance between points 28 and 33, and the mouth area is a square with a width of points 48 and 54.

5. The face occlusion detection method according to claim 2, characterized in that: The weights are back-propagated and updated using a loss function during training via a convolutional neural network, which extracts feature vectors. The loss function is obtained by calculating the cross entropy between the label of the feature vector and the annotated label of the augmented image using a softmax function.

6. A face occlusion detection method according to claim 1, characterized in that: The hidden layer of the convolutional neural network is replaced by a mobilenet, vggnet or resnet network structure.

Images