A method for facial recognition payment using face masks

Abstract

A method for face recognition payment using masks is proposed, supporting face detection and registration even when masks are obscuring the face, ensuring the accuracy and robustness of face recognition. A special product database is constructed. Silent liveness detection technology is employed: Silent liveness detection is used to detect faces obscured by masks, effectively avoiding interference from attacks using photos or videos. A lightweight network is used for mask recognition: To avoid excessive computational load, a lightweight network is selected for mask recognition, improving the system's speed and efficiency. The network model is improved, enabling fast and accurate recognition of faces wearing masks. The loss function is improved: To address the face recognition problem under mask obscuration, the loss function is improved, increasing the system's recognition accuracy and robustness while reducing the system's data requirements.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, specifically to a method for facial recognition payment using face masks. Background Technology

[0002] With the widespread adoption of mobile payments, facial recognition technology has been widely used in payment scenarios. Wearing masks in public places has become commonplace. Traditional facial recognition technology suffers from a certain false recognition rate and low accuracy when recognizing faces wearing masks, which poses challenges to the use of payment systems. To address this issue, people have begun to experiment with facial recognition technology for people wearing masks to improve the security and usability of payment systems.

[0003] However, existing facial recognition technologies for people wearing masks have many problems. First, traditional feature extraction methods are not suitable for faces wearing masks because masks obscure some facial features. Second, facial recognition networks need to be able to distinguish between masked and unmasked states, which requires high classification accuracy. Finally, existing facial recognition networks are typically large and require significant computational resources, which is unacceptable for mobile applications. Summary of the Invention

[0004] In order to overcome the shortcomings of the above technologies, this invention provides a method that improves the accuracy of facial recognition using masks and enables payment functionality.

[0005] The technical solution adopted by this invention to overcome its technical problems is:

[0006] A method for facial recognition payment using face masks includes the following steps:

[0007] a) Obtain a facial image of someone wearing a mask;

[0008] b) Preprocess the facial images of people wearing masks;

[0009] c) Perform face location detection and face registration on the face image;

[0010] d) Perform liveness detection on the image after face location detection. If the face in the image is a real face, proceed to step e); if the face in the image is not a real face, end the transaction.

[0011] e) Establish a mask-wearing face recognition network, input the image after real face location detection into the mask-wearing face recognition network, and obtain whether the face wearing a mask passes the detection and whether the face not wearing a mask passes the detection.

[0012] Furthermore, in step a), facial images of people wearing masks and not wearing masks are obtained from the open-source dataset CelebA or a dataset that we collect and annotate ourselves.

[0013] Furthermore, the preprocessing in step b) includes image data augmentation, normalization, and partitioning the training and test sets. The image data augmentation includes random flipping, random brightness adjustment, and contrast adjustment.

[0014] Furthermore, in step c), the MTCNN algorithm is used to perform face location detection and face registration on the face image. Furthermore, in step d), the CDCN++ algorithm is used to perform liveness detection on the image after face location detection. Furthermore, step e) includes the following steps:

[0015] e-1) The mask-wearing face recognition network consists of a MobileNetV3 network, a GlobalAvgPooling2D layer, a Flatten function, a Mask branch, and a NoMask branch;

[0016] e-2) Input the real face image into the MobileNetV3 network of the masked face recognition network, and output the feature map;

[0017] e-3) The feature map is input into the GlobalAvgPooling2D layer for global average pooling, and then flattened using the Flatten function to obtain the feature vector;

[0018] e-4) The Mask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer in sequence. The feature vector is input into the Mask branch, and the floating-point number of the masked face is output.

[0019] e-5) The NoMask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer. The feature vector is input into the Mask branch, and the output is the floating-point number of the face without a mask.

[0020] e-6) Map the floating-point numbers of faces wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face wearing a mask has failed the detection. Map the floating-point numbers of faces not wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face not wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face not wearing a mask has failed the detection.

[0021] Furthermore, in step e-2), the real face image is input into the MobileNetV3 network, and the output is a feature map of size 224×224 containing three color channels: red, green, and blue.

[0022] In step e-4), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.5.

[0023] Furthermore, in step e-5), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.5.

[0024] Furthermore, the following steps are performed after step e):

[0025] f-1) Construct the loss function L for the face recognition branch using the cross-entropy loss function. face ;

[0026] f-2) Construct the loss function L for the mask recognition branch using the cross-entropy loss function. mask ;

[0027] f-3) Using the formula L=λ1×L face +λ2×L mask The final loss L is calculated by adding λ3×regularization, where regularization is the regularization term, and λ1, λ2, and λ3 are all hyperparameters.

[0028] f-4) The final loss L is used to optimize the masked face recognition network in step e) using the Adam optimizer. The beneficial effects of this invention are: It employs the MTCNN algorithm for face detection and registration, combining face region detection with face keypoint detection. It uses the CDCN++ monocular silent liveness detection algorithm to detect whether a face is real. It uses MobileNetV3 as the backbone neural network, adding a mask recognition branch to quickly and accurately identify faces wearing masks. The loss function is improved, combining the classifier results from the face recognition branch and the mask recognition branch, and a regularization term is added to ensure model stability. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 This is a diagram of the face recognition network structure for the mask according to the present invention. Detailed Implementation

[0031] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 The present invention will be further described below.

[0032] A method for facial recognition payment using face masks includes the following steps:

[0033] a) Obtain facial images of people wearing masks.

[0034] b) Preprocess the facial images of people wearing masks.

[0035] c) Perform face location detection and face registration on the face image.

[0036] d) Perform liveness detection on the image after face location detection. If the face in the image is a real face, proceed to step e); otherwise, end the transaction.

[0037] e) Establish a mask-wearing face recognition network, input the image after real face location detection into the mask-wearing face recognition network, and obtain whether the face wearing a mask passes the detection and whether the face not wearing a mask passes the detection.

[0038] By improving the network structure, the accuracy of masked facial recognition was enhanced, and payment functionality was implemented. The development of this system mainly involved the following technical challenges: face detection and registration under mask occlusion, silent liveness detection under masks, mask recognition, and loss function improvement. These challenges required the use of deep learning algorithms to overcome the impact of mask occlusion on facial recognition while ensuring accuracy and robustness, thus enabling convenient payment operations. Therefore, developing a deep learning-based masked facial recognition payment system not only requires a deep understanding of deep learning algorithms but also necessitates designing facial recognition algorithms specifically for mask-occluded situations, based on real-world scenarios, to achieve a secure, efficient, and convenient payment method.

[0039] Supports face detection and registration even when masked, ensuring accuracy and robustness of face recognition by building a special product database. Employs silent liveness detection technology: Silent liveness detection is used to detect faces hidden by masks, effectively avoiding interference from attacks using photos and videos. Uses a lightweight network for mask recognition: To avoid excessive computational load, a lightweight network is used for mask recognition, improving system speed and efficiency. Improved network model enables fast and accurate recognition of masked faces. Improved loss function: For the face recognition problem under mask occlusion, the loss function is improved, increasing the system's recognition accuracy and robustness while reducing the system's data requirements.

[0040] Example 1:

[0041] In step a), obtain facial images of people wearing masks and not wearing masks from the open-source dataset CelebA or a self-collected and labeled dataset. The dataset contains facial images and their corresponding mask annotations.

[0042] Example 2:

[0043] The preprocessing in step b) includes image data augmentation, normalization, and partitioning the training and test sets. In the mask recognition task, operations such as cropping, rotating, and translating the face are performed to increase data diversity. The image data augmentation includes random flipping, random brightness, and contrast adjustment to further improve the robustness of the model.

[0044] Example 3:

[0045] Step c) uses the MTCNN algorithm to perform face location detection and face registration on the face image. Face location detection is an important component of this mask-wearing face recognition payment system, mainly used to determine the location of faces in the image and perform subsequent face recognition and mask detection. To achieve efficient and accurate face detection, this system uses the MTCNN algorithm. MTCNN is a deep learning-based face detection algorithm. Its main feature is that it decomposes the face detection task into multiple cascaded sub-tasks, and applies a convolutional neural network (CNN) for feature extraction and classification in each sub-task, thereby achieving face detection in complex scenes.

[0046] MTCNN combines face region detection with facial landmark detection. To balance performance and accuracy, and avoid the significant performance overhead of traditional approaches like sliding window and classifier, MTCNN first uses a small model to generate candidate bounding boxes with a certain probability of success. Then, it uses a more complex model for finer classification and higher-precision bounding box regression, recursively executing this step. This approach forms a three-layer network: P-Net, R-Net, and O-Net, achieving fast and efficient face detection. In the input layer, an image pyramid is used to scale the initial image, and P-Net generates a large number of candidate bounding boxes. R-Net then performs initial selection and bounding box regression on these boxes, eliminating most negative examples. Finally, the more complex and accurate O-Net discriminates and regresses bounding boxes on the remaining boxes.

[0047] Example 4:

[0048] In step d), the CDCN++ algorithm is used to perform liveness detection on the image after face location detection. The CDCN++ algorithm is a liveness detection algorithm based on convolutional neural networks (CNN), which can effectively detect the liveness of biometric information and avoid fraudulent activities such as forgery or photo attacks.

[0049] The algorithm mainly consists of two parts: extraction of the face image of the person wearing a mask as input and liveness detection. In the extraction part of the face image of the person wearing a mask as input, the algorithm uses a technique called "dynamic convolution" to extract effective feature information from the face image. At the same time, considering the importance of features of different facial parts, a context-guided method is adopted to enhance the feature representation of different locations.

[0050] In the liveness detection part, the algorithm analyzes multiple dimensions of information in a face image to determine whether it is a real human face. Specifically, the algorithm uses two convolutional neural networks to process the temporal and spatial information of the face image respectively, and obtains the final liveness detection result through feature fusion.

[0051] The silent liveness detection feature of this mask analyzes and judges multi-dimensional information from the facial image of the person wearing the mask, which can effectively prevent attackers from using photos, masks, and other fraudulent means to conduct illegal transactions, thereby ensuring the security and reliability of the payment system.

[0052] In this system, we use the CDCN++ algorithm as the core algorithm for silent liveness detection of face masks to ensure the security of the payment process. We also employ data preprocessing, face detection, face registration, data augmentation, and feature extraction techniques to improve the system's accuracy and robustness.

[0053] Example 5:

[0054] Step e) includes the following steps:

[0055] (e-1) The masked face recognition component is the core of this system. Its purpose is to identify and verify the identity of faces that have passed silent liveness detection while wearing masks. This invention employs a face recognition model based on the MobileNetV3 backbone network, incorporating a masked face recognition branch to improve the accuracy of face recognition when masks are worn. Specifically, the masked face recognition network consists of a MobileNetV3 network, a GlobalAvgPooling2D layer, a Flatten function, a Mask branch, and a NoMask branch. Previously, we used the MTCNN algorithm to detect and locate faces in images, and used face registration technology to align the detected faces to reduce the impact of factors such as lighting and facial expressions on face recognition. Then, we used the CDCN++ algorithm to perform silent liveness detection on the aligned faces, determining whether the wearer is wearing a mask and whether they are a real person. Based on this, for faces that pass the detection, we use a face recognition model based on the MobileNetV3 backbone network for identity verification.

[0056] e-2) Input the real face image into the MobileNetV3 network of the masked face recognition network, and output the feature map.

[0057] e-3) The feature map is input into the GlobalAvgPooling2D layer for global average pooling, and then flattened using the Flatten function to obtain the feature vector.

[0058] e-4) The Mask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer. The feature vector is input into the Mask branch, and the floating-point number of the masked face is output.

[0059] e-5) The NoMask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer. The feature vector is input into the Mask branch, and the output is a floating-point number of the face without a mask.

[0060] e-6) Map the floating-point numbers of faces wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face wearing a mask has failed the detection. Map the floating-point numbers of faces not wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face not wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face not wearing a mask has failed the detection.

[0061] Example 6:

[0062] MobileNetV3 can adapt to different input sizes, so we adjust the network input size according to the specific application scenario. In the mask recognition task, considering the balance between real-time performance and accuracy, the real face image in step e-2) is input into the MobileNetV3 network, and the output is a feature map of size 224×224 containing three color channels: red, green, and blue.

[0063] Example 7:

[0064] In step e-4), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.5.

[0065] Example 8:

[0066] In step e-5), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.5.

[0067] Example 9:

[0068] To improve the model's performance and stability, we adopted an improved loss function. This loss function combines the classifier results from the face recognition branch and the mask recognition branch, and adds a regularization term. Specifically, the loss function consists of the following three parts: a. Face recognition loss: The loss function for the face recognition branch uses the cross-entropy loss function, which measures the model's performance when recognizing faces. b. Masked face recognition loss: The loss function for the mask recognition branch uses the cross-entropy loss function, which measures the model's performance when recognizing whether a mask is being worn. c. Regularization term: We added a regularization term to prevent overfitting. This term calculates the difference between the two classifiers and adds it to the loss function. Specifically, the following steps are also performed after step e):

[0069] f-1) Construct the loss function L for the face recognition branch using the cross-entropy loss function. face .

[0070] f-2) Construct the loss function L for the mask recognition branch using the cross-entropy loss function. mask .

[0071] f-3) Using the formula L=λ1×L face +λ2×L mask The final loss L is calculated by adding λ3×regularization, where regularization is the regularization term, and λ1, λ2, and λ3 are hyperparameters used to adjust the weights of different parts.

[0072] f-4) The final loss L is used to optimize the masked face recognition network in step e) using the Adam optimizer. This final loss L optimizes the performance of both face recognition and mask recognition simultaneously, and reduces overfitting. Finally, after extensive experimental adjustments, our designed masked face recognition model achieved good performance on the test set, with an accuracy rate exceeding 98.5%, meeting the needs of practical applications. It should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for facial recognition payment using a face mask, characterized in that, Includes the following steps: a) Obtain a facial image of someone wearing a mask; b) Preprocess the facial images of people wearing masks; c) Perform face location detection and face registration on the face image; d) Perform liveness detection on the image after face location detection. If the face in the image is a real face, proceed to step e); if the face in the image is not a real face, end the transaction. e) Establish a mask-wearing face recognition network, input the image after real face location detection into the mask-wearing face recognition network, and obtain whether the face wearing a mask passes the detection and whether the face not wearing a mask passes the detection; Step e) includes the following steps: e-1) The mask-wearing face recognition network consists of a MobileNetV3 network, a GlobalAvgPooling2D layer, a Flatten function, a Mask branch, and a NoMask branch; e-2) Input the real face image into the MobileNetV3 network of the masked face recognition network, and output the feature map; e-3) The feature map is input into the GlobalAvgPooling2D layer for global average pooling, and then flattened using the Flatten function to obtain the feature vector; e-4) The Mask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer in sequence. The feature vector is input into the Mask branch, and the floating-point number of the masked face is output. e-5) The NoMask branch consists of the first Dense layer, the first Dropout layer, the first ReLU function layer, the second Dense layer, the second Dropout layer, and the second ReLU function layer. The feature vector is input into the Mask branch, and the output is the floating-point number of the face without a mask. e-6) Map the floating-point numbers of faces wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face wearing a mask has failed the detection. Map the floating-point numbers of faces not wearing masks to a binary classification between [0,1] using the sigmoid function. A binary classification value of 0 indicates that the face not wearing a mask has passed the detection, and a binary classification value of 1 indicates that the face not wearing a mask has failed the detection.

2. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step a), obtain facial images of people wearing masks and not wearing masks from the open-source dataset CelebA or a dataset you have collected and labeled yourself.

3. The method for facial recognition payment using a mask according to claim 1, characterized in that: The preprocessing in step b) includes image data augmentation, normalization, and dividing the training and test sets. The image data augmentation includes random flipping, random brightness adjustment, and contrast adjustment.

4. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step c), the MTCNN algorithm is used to detect the face location and register the face in the face image.

5. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step d), the CDCN++ algorithm is used to perform liveness detection on the image after face location detection.

6. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step e-2), the real face image is input into the MobileNetV3 network, and the output is a feature map of size 224×224 containing three color channels: red, green, and blue.

7. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step e-4), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.

5.

8. The method for facial recognition payment using a mask according to claim 1, characterized in that: In step e-5), the first Dense layer has 256 neurons, the second Dense layer has 128 neurons, and the dropout rate of both the first and second Dropout layers is 0.

5.

9. The method for facial recognition payment using a mask according to claim 1, characterized in that, This also includes performing the following steps after step e): f-1) Construct the loss function for the face recognition branch using the cross-entropy loss function. ; f-2) Construct the loss function for the mask recognition branch using the cross-entropy loss function. ; f-3) via formula The final loss was calculated. regularization is the regularization term. , , All are hyperparameters; f-4) uses the final loss The masked face recognition network in step e) is optimized using the Adam optimizer.

Images