FaceNet and attention mechanism-based shielded face recognition method

Abstract

The invention relates to an occluded face recognition method based on FaceNet and an attention mechanism. The method comprises the steps of data set preprocessing, face recognition network construction, network training and testing. According to the method, an MTCNN backbone network is adopted to carry out face region detection and face key point detection, an attention mechanism is used to focus the network on an unshielded part, a CNN neural network is adopted to map a face image to a 128-dimensional Euclidean space, and the similarity degree of two persons is judged according to the Euclidean distance between two portraits. The invention aims to solve the problem of how to obtain the face features of the non-shielding face part, so that the performance and robustness of a face recognition task are improved, and the face detection effect in a complex shielding scene is improved.

Description

Technical Field

[0001] The invention relates to an occluded face recognition method based on FaceNet and an attention mechanism, and belongs to the technical field of computer vision. Background Art

[0002] The face recognition task is an important part of modern security systems and identity authentication technology. It uses advanced image processing and pattern recognition technology to identify and verify the identity of individuals. As a contactless recognition technology, face recognition has good adaptability, flexibility and scalability compared to fingerprint recognition, iris recognition, etc. In recent years, with the rapid development of computer vision and deep learning technology, face recognition has evolved from simple feature comparison to complex biometric analysis. Its application scope has also expanded from traditional security fields to many daily life scenarios such as mobile payment, social media, and smart device unlocking. It has great practical significance in maintaining public information security and promoting generative life.

[0003] However, in non-constrained scenarios, faces are inevitably occluded, causing conventional face detection and recognition algorithms to experience varying degrees of performance degradation. The reason for this is that, on the one hand, the integrity of facial information is destroyed, key information is lost, and the model is difficult to extract sufficient effective features; on the other hand, facial occlusions are noise data, so the features extracted by the model also contain noise, which leads to missed detections by the face detection algorithm and the inability of the face recognition algorithm to correctly identify identity information. These problems have greatly affected the performance of the face recognition system. Summary of the invention

[0004] In order to overcome the defects of the prior art, the present invention provides an occluded face recognition method based on FaceNet and attention mechanism, which uses the MTCNN backbone network to detect face regions and face key points, uses the attention mechanism to focus the network on the unoccluded parts, uses the CNN neural network to map the face image to a 128-dimensional Euclidean space, and judges the similarity between two people based on the Euclidean distance of the two portraits. The present invention aims to solve the problem of how to obtain the facial features of the non-occluded face part, thereby improving the performance and robustness of the face recognition task and improving the face detection effect in complex occlusion scenes.

[0005] An occluded face recognition method based on FaceNet and attention mechanism includes the following steps:

[0006] Step 1: Dataset preprocessing: Select CASIA-Webface and LFW as datasets, send the datasets to the MTCNN network for face detection, alignment and cropping, perform mask processing on the preprocessed CASIA-Webface and LFW datasets to generate CASIA-Webface_Mask and LFW_Mask, and use CASIA-Webface and CASIA-Webface_Mask as training sets in a 1:1 mixing mode, and select LFW and LFW_Mask datasets as test sets;

[0007] Step 2: Build a face recognition network: Replace the original backbone network of FaceNet with the MobileNetV3 network, and replace the original Squeeze-and-Excitation attention mechanism of the MobileNetV3 network structure with the CBAM attention mechanism, giving higher weights to non-occluded areas and lower weights to occluded features, thereby enhancing more important facial features;

[0008] Step 3: Train the network: Input the preprocessed data set into the face recognition network and train the model using the TripletLoss loss function;

[0009] Step 4: Test: Input the preprocessed LFW and LFW_Mask datasets into the face recognition network, compare them with other images, and calculate the evaluation indicators.

[0010] The step 1 specifically includes:

[0011] The face dataset is input into the P-Net network to generate face candidate regression frames, and the NMS algorithm is used to merge highly overlapping candidate frames; the generated face candidate regression frames are input into the R-Net network for stricter screening, and the NMS algorithm is used to output the most accurate face candidate frames; the screened face candidate frames are input into the 0-Net network for more precise screening, and the NMS algorithm is used to output the most accurate face candidate frames; all the most accurate face candidate frames are aligned and cropped to an image size of 128×128. The processed CASIA-Webface and LFW datasets are input into MaskTheFace, and masks are added to the faces in the dataset for occlusion processing. CASIA-Webface and CASIA-Webface_Mask are used as training sets in a 1:1 mixing manner, and LFW and LFW_Mask datasets are selected as test sets.

[0012] The step 2 specifically includes:

[0013] FaceNet directly learns the Euclidean features of the input face image by training a deep convolutional neural network, mapping the face image to a 128-dimensional Euclidean space. The length of the spatial distance indicates the similarity of the face image, where the spatial distance between different faces of the same person is small, while the spatial distance between images of different faces is large. The recognition process of FaceNet is mainly divided into four parts. First, the face image data is processed and batched into the network backbone; then, the input face data is feature extracted, where the backbone network can be regarded as a black box and can be replaced by other feature extraction networks; then, the extracted face feature information is normalized through the L2 paradigm; finally, the model is optimized using a specific ternary loss function.

[0014] The original backbone network of FaceNet can achieve high accuracy for normal face recognition, but it cannot solve the problem of face recognition with occlusion. The present invention uses the MobileNetV3 network to replace the original Inception-RestNetV1 backbone network in FaceNet as the backbone extraction network. MobileNetV3 is a lightweight neural network model that retains the depth-separable convolution of MobileNetV1 and inherits the residual structure of MobileNetV2. It uses the residual structure of the linear bottleneck as the basic convolution module, which can increase the width and expression ability of the network by expanding the channel, while avoiding the problem of high-dimensional information being lost after the activation function. In addition, the MobileNetV3 network redesigns the time-consuming layer structure and updates the activation function. MobileNetV3 uses 1×1 convolutional layers and 3×3 convolutional layers to replace the original version of 5×5 convolution. This change significantly reduces the number of parameters. In order to further improve the performance of the model, MobileNetV3 introduces the Swish activation function to ensure the accuracy and lightness of the network.

[0015] The Squeeze-and-Excitation attention mechanism in MobileNetV3 only considers the information between channels and ignores the position information, which is very important for face positioning and recognition. This paper uses CBAM to replace the SE attention mechanism in MobileNetV3. CBAM provides a more comprehensive and effective feature extraction capability for the model by combining channel attention and spatial attention.

[0016] The step three specifically includes:

[0017] FaceNet uses a ternary loss function to train the network to output a 128-dimensional feature vector, and controls the distance between positive and negative samples by setting a threshold. Face images are input in the form of a triplet, namely the target sample (anchor), the positive sample of the same person as the target sample, and the negative sample of a different person from the target sample. For FaceNet, the loss function is expressed as:

[0018]

[0019]

[0020] in is the target sample, is a positive sample, is a negative sample, a is the threshold that fully separates positive and negative samples, T is the set of possible triplets in the data set, and n is the number of elements in the set.

[0021] The goal of the training optimization is to minimize L so that the distance between a particular person's facial image and all other images of that person is smaller than the distance between any image of any other person.

[0022]

[0023] The step 4 specifically includes:

[0024] The characteristic Euclidean distance of a pair of images is expressed as D(x i ,x j ), which is the face pair of all the faces of the same person All faces of different people are d is the dimension.

[0025]

[0026] The set of the same person and classified as the same person (True Accept):

[0027]

[0028] A set of people who are not the same person but are classified as the same person (False Accept)

[0029]

[0030] Verification accuracy VAL(d) and false acceptance rate FAR(d):

[0031]

[0032] Generally, when FAR is a small value (1e-3...1e-6), the larger the VAL value is, the more accurate the face verification is. The calculation process is to get the distance threshold d when FAR is equal to the specified value, and then calculate the VAL value.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention relates to an occluded face recognition method based on FaceNet and an attention mechanism, which uses MTCNN for face detection, alignment, and cropping, and uses an attention mechanism to focus the network on unoccluded face parts. At the same time, a MobileNetV3 network is used to replace the original backbone network in FaceNet as a backbone extraction network, thereby ensuring the real-time and lightweight nature of the network. In addition, the present invention also uses a CBAM attention mechanism to replace the SE attention mechanism, so that the model can better capture key information in the feature map, thereby improving the model performance.

[0035] The present invention applies the MTCNN model to perform face detection, alignment, and cropping, and then fits the image to face recognition, which can reduce the impact of useless background or noise on the performance of the face recognition model.

[0036] The present invention uses the MobileNetV3 network as the backbone network in the FaceNet network model. The network adopts deep separable convolution, which reduces the amount of calculation and the number of parameters of the model, improves the reasoning speed of the model, and also maintains the recognition accuracy. MobileNetV3 uses a new activation function h-swish, which can provide better nonlinear characteristics and help improve face recognition performance.

[0037] The present invention uses the CBAM attention mechanism instead of the SE attention mechanism of MobileNetV3 so that the model can better capture the key information in the feature map, thereby improving the model performance. CBAM uses a dual attention mechanism to make the model better adaptable to different data sets and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 This is a flow chart of the MTCNN algorithm of an occluded face recognition method based on FaceNet and attention mechanism of the present invention;

[0040] Figure 2 The FaceNet structure diagram of the occluded face recognition method based on FaceNet and attention mechanism of the present invention;

[0041] Figure 3 This is a bneck structure diagram of the MobileNetV3 in the occluded face recognition method based on FaceNet and attention mechanism of the present invention;

[0042] Figure 4 A CBAM attention mechanism diagram of an occluded face recognition method based on FaceNet and attention mechanism of the present invention;

[0043] Figure 5 A diagram of the training process of a triplet loss function of an occluded face recognition method based on FaceNet and an attention mechanism of the present invention;

[0044] Figure 6 This is an overall framework diagram of an occluded face recognition method based on FaceNet and attention mechanism in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] An occluded face recognition method based on FaceNet and attention mechanism includes the following steps:

[0047] Step 1: Dataset preprocessing: Select CASIA-Webface and LFW as datasets, send the datasets to the MTCNN network for face detection, alignment and cropping, perform mask processing on the preprocessed CASIA-Webface and LFW datasets to generate CASIA-Webface_Mask and LFW_Mask, and use CASIA-Webface and CASIA-Webface_Mask as training sets in a 1:1 mixing mode, and select LFW and LFW_Mask datasets as test sets;

[0048] Specifically, Figure 1, the CASIA and LFW face datasets are input into the fully connected convolutional neural network P-Net to generate face candidate frames and obtain regression vectors, and the NMS algorithm is used to merge highly overlapping candidate frames; the generated face candidate frames are then input into the R-Net network for stricter screening, and the NMS algorithm is used to calculate the IOU of the overlapping area between the face candidate frame and the target, and compared with the threshold to eliminate the face candidate frames that do not meet the area score, and output the filtered regression characteristics; the filtered face candidate frames are input into the O-Net network to output the most accurate face candidate frames using the NMS algorithm; finally, all the most accurate face candidate frames are aligned and cropped to an image of 128×128 size.

[0049] On the official github website, I found a MaskTheFace package, which can add different types of masks to face pictures. The generated face images wearing masks are also clearer and have better robustness.

[0050] Step 2: Build a face recognition network: Replace the original backbone network of FaceNet with the MobileNetV3 network, and replace the original Squeeze-and-Excitation attention mechanism of the MobileNetV3 network structure with the CBAM attention mechanism, giving higher weights to non-occluded areas and lower weights to occluded features, thereby enhancing more important facial features;

[0051] Specifically, Figure 2 , the recognition process of FaceNet is mainly divided into four parts. First, the face image data is processed and batched into the network backbone; then the input face data is feature extracted, where the backbone network can be regarded as a black box and can be replaced by other feature extraction networks; then, the extracted face feature information is normalized through the L2 paradigm; finally, the model is optimized using a specific ternary loss function.

[0052] The present invention uses the MobileNetV3-small model to extract facial features. The input of MobileNetV3-small is a 3-channel image. First, it passes through a common convolution layer to convert the 3 channels of the input into 16 channels. Then, a series of bottleneck layers (Bottleneck, Bneck) are used to extract and compress features. Finally, a global average pooling layer is added to set the height and width of the output feature map to 1, and retain the feature information of all channels. Subsequently, two 1×1 convolutional layers convert the output characteristics of the global average pooling layer into the probability distribution of 1000 categories and the classification results of the model. The bneck structure in MobileNetV3 is as follows: Figure 3 .

[0053] The SE attention mechanism module is used in the MobileNetV3 network structure, and the SE module focuses on the channel dimension of the feature map and ignores the spatial dimension of the target information. The present invention uses the CBAM module instead of the SE module, such as Figure 4 ,The CBAM module forms an attention map at two locations, channel and space, and performs element-wise multiplication on the attention map and the input feature map of each dimension, so that the extracted features are more comprehensive and more accurate.

[0054] Using the backbone feature extraction network, we can obtain a feature layer, perform global average pooling on it, and fully connect the flattened feature layer with 128 neurons. At this time, we use a feature vector with a length of 128 to replace the input image. This feature vector with a length of 128 is the condensed version of the input image.

[0055] L2 regularization is responsible for mapping the features of the deep network to a hyperspherical space. Assume that the face feature vector obtained from the deep network is [x1, x2, x3]. After regularization, the feature vector becomes:

[0056]

[0057] Assuming the center of the sphere is (0, 0, 0), the distance from the eigenvector to the center of the sphere is:

[0058]

[0059] In this way, the problem of calculating the similarity of face images is transformed into the problem of calculating the distance between coordinates.

[0060] MobileNetV3 uses the h-swish activation function, which is optimized from the swish activation function. The h-swish function solves the time loss of calculation and derivation, improves the network's reasoning speed and quantization process, and makes the network perform better.

[0061] The h-swish activation function is

[0062]

[0063] R ( x ) =min ( max ( x,0 ) ,6)

[0064] Step 3: Train the network: Input the preprocessed data set into the face recognition network and train the model using the TripletLoss loss function;

[0065] Specifically, FaceNet uses a ternary loss function to train the network to output a 128-dimensional feature vector, and controls the distance between positive and negative samples by setting a threshold. Face images are input in the form of a triplet, namely, a target sample (anchor), a positive sample that is the same person as the target sample, and a negative sample that is not the same person as the target sample. Figure 5 , optimize the network model.

[0066] The goal of the training optimization is to minimize L so that the distance between a particular person's facial image and all other images of that person is smaller than the distance between any image of any other person.

[0067]

[0068] Step 4: Test: Input the preprocessed LFW and LFW_Mask datasets into the face recognition network, compare them with other images, and calculate the evaluation indicators.

[0069] Specifically, the processed LFW and LFW_Mask datasets are respectively input into the face recognition network to verify the recognition effect of the model on occluded faces.

[0070] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. An occluded face recognition method based on FaceNet and attention mechanism, characterized by: The following steps are involved: Step 1: Dataset preprocessing: Select CASIA-Webface and LFW as datasets, send the datasets to the MTCNN network for face detection, alignment and cropping, perform mask processing on the preprocessed CASIA-Webface and LFW datasets to generate CASIA-Webface_Mask and LFW_Mask, use CASIA-Webface and CASIA-Webface_Mask as training sets in a 1:1 mixing manner, and select LFW and LFW_Mask datasets as test sets; Step 2: Build a face recognition network: Replace the original backbone network of FaceNet with the MobileNetV3 network, and replace the original Squeeze-and-Excitation attention mechanism of the MobileNetV3 network structure with the CBAM attention mechanism, giving higher weights to non-occluded areas and lower weights to occluded features, thereby enhancing more important facial features; Step 3: Train the network: Input the preprocessed data set into the face recognition network and train the model using the Triplet Loss function; Step 4: Test: Input the preprocessed LFW and LFW_Mask datasets into the face recognition network, compare them with other images, and calculate the evaluation indicators.

2. The occluded face recognition method based on FaceNet and attention mechanism according to claim 1, characterized in that: The step 1 specifically includes: The face dataset is input into the P-Net network to generate face candidate regression frames, and the NMS algorithm is used to merge highly overlapping candidate frames; the generated face candidate regression frames are input into the R-Net network for stricter screening, and the NMS algorithm is used to output the most accurate face candidate frames; the screened face candidate frames are input into the 0-Net network for precise screening, and the NMS algorithm is used to output the most accurate face candidate frames; all the most accurate face candidate frames are aligned and cropped to an image size of 128×128, and the processed CASIA-Webface and LFW datasets are input into MaskTheFace, masks are added to the faces in the dataset for occlusion processing, CASIA-Webface and CASIA-Webface_Mask are used as training sets in a 1:1 mixing manner, and LFW and LFW_Mask datasets are selected as test sets.

3. The occluded face recognition method based on FaceNet and attention mechanism according to claim 1, characterized in that: The step 2 specifically includes: FaceNet directly learns the Euclidean features of input face images by training a deep convolutional neural network, mapping the face images to a 128-dimensional Euclidean space. The length of the spatial distance represents the similarity of the face images. The recognition process of FaceNet is divided into four parts. First, the facial image data is processed and batched into the network backbone; then the input facial data is extracted, where the backbone network can be regarded as a black box and can be replaced by other feature extraction networks; then the extracted facial feature information is normalized through the L2 paradigm, and the model is optimized using a specific ternary loss function.

4. The occluded face recognition method based on FaceNet and attention mechanism according to claim 1, characterized in that: The step three specifically includes: FaceNet uses a ternary loss function to train the network to output a 128-dimensional feature vector. The distance between positive and negative samples is controlled by setting a threshold. Face images are input in the form of triplets, namely, the target sample, the positive sample of the same person as the target sample, and the negative sample of a different person from the target sample. For FaceNet, the loss function is expressed as: in is the target sample, is a positive sample, is a negative sample, a is the threshold that makes positive and negative samples fully separated, T is the set of possible triplets in the data set, and n is the number of elements in the set; The goal of training optimization is to minimize L to ensure that the distance between a particular person's facial image and all other images is smaller than the distance between any images of any other person; 5. The occluded face recognition method based on FaceNet and attention mechanism according to claim 1, characterized in that: The step 4 specifically includes: The characteristic Euclidean distance of a pair of images is represented by D ( x i ,x j ), which is the face pair of all the faces of the same person All faces of different people are d is the dimension; The set is the same person and is classified as the same person: A collection of people that are not the same person but are classified as the same person: Verification accuracy VAL(d) and false acceptance rate FAR(d): When FAR is a small value, the larger the VAL value is, the more accurate the face verification is.

Images