Face recognition method based on MTCNN algorithm

Abstract

The invention provides a face recognition method based on an MTCNN algorithm. The face recognition method comprises the steps that collected samples are labeled; training the face region detection model to obtain a face region detection training model; and applying the face region detection training model. According to the method, the identification success rate is improved, user information is stored in a server of a company, public opinion events caused by leakage of the user information are prevented, algorithm optimization and offset modification are carried out according to different scenes of the company, and the purpose of adapting to scene use is achieved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a face recognition method based on MTCNN algorithm. Background Art

[0002] Facial recognition technology can be widely used in public security, finance, airports, subways and other important fields for natural comparison and identification of personnel identities. Whether it is face payment at convenience stores or attendance punching for office workers, the application scenarios of facial recognition are increasing, and the requirements for the accuracy and security of facial recognition are getting higher and higher.

[0003] Different facial recognition algorithms result in uneven technology and security cannot be guaranteed.

[0004] Different face recognition algorithms lead to uneven technology and unguaranteed security. It is easy to leak personal information and the recognition success rate cannot be guaranteed. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a face recognition method based on the MTCNN algorithm that overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a face recognition method based on the MTCNN algorithm is provided, and the face recognition method comprises:

[0007] Label the collected samples;

[0008] Training the face region detection model to obtain a face region detection training model;

[0009] Apply the face region detection training model.

[0010] Optionally, the labeling of the collected samples specifically includes:

[0011] Use of sample annotation tools;

[0012] Face area detection sample annotation;

[0013] Facial feature point calibration sample annotation.

[0014] Optionally, the training of the face area detection model specifically includes:

[0015] Face region detection model training and face region detection model evaluation;

[0016] Facial feature point calibration model training and facial feature point calibration model evaluation;

[0017] Face comparison model training and face comparison model evaluation.

[0018] Optionally, the face area detection model training specifically includes:

[0019] Collect multiple face images, and mark the multiple face images with a unique label to obtain a face dataset;

[0020] Preprocessing the face data set to obtain a preprocessed face data set;

[0021] Use the OpenCV library's face detector to locate and crop the face area;

[0022] A face recognition model is trained based on the preprocessed face dataset.

[0023] Optionally, the training of the face recognition model according to the preprocessed face data set specifically includes: using the MTCNN algorithm to train the model in OpenCV.

[0024] Optionally, in OpenCV, using the MTCNN algorithm to train the model specifically includes:

[0025] Use MTCNN algorithm to quickly identify loading models;

[0026] Use OpenCV's VideoCapture class to open and read camera data, and use the MTCNN algorithm ProposalNetwork to optimize the face data read by the camera;

[0027] In each frame, a face detector is used to detect the face area;

[0028] Compare the detected face with the previously trained model and call the MTCNN algorithm OutputNetwork to calibrate the key points;

[0029] Extract feature vectors from detected face regions;

[0030] Based on the matching results, a bounding box is drawn in the image and the recognition results are displayed.

[0031] It will be put into a loop to realize real-time face recognition function.

[0032] Optionally, extracting the feature vector from the detected face region specifically includes: converting the face image into a grayscale image and applying a histogram equalization preprocessing method;

[0033] Use the trained face recognition model to match the extracted feature vectors with known face tags.

[0034] Optionally, applying the face area detection training model specifically includes:

[0035] Real-time video acquisition, real-time image capture, real-time face detection, real-time facial feature point calibration, real-time facial feature point alignment, real-time face comparison, real-time blink recognition and real-time mouth opening recognition.

[0036] The present invention provides a face recognition method based on the MTCNN algorithm, the face recognition method comprising: labeling the collected samples; training the face region detection model to obtain the face region detection training model; applying the face region detection training model. The recognition success rate is improved, and the user information is stored in the company's own server to prevent public opinion events caused by user information leakage. At the same time, the algorithm is optimized and the offset is modified according to different scenarios of the company to achieve the purpose of adapting to the scene use.

[0037] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 A system business architecture diagram corresponding to a face recognition method based on the MTCNN algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0042] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, the face recognition method based on the MTCNN algorithm includes: sample labeling, model training and model application.

[0044] Sample annotation, including the use of sample annotation tools, facial region detection sample annotation, and facial feature point calibration sample annotation;

[0045] Model training, including face region detection model training, face region detection model evaluation, face feature point calibration model training, face feature point calibration model evaluation, face comparison model training and face comparison model evaluation; Model training process: In order to train the face recognition model, you need a labeled face dataset. Collect a sufficient number of face images and mark each face with a unique label. Before training, the dataset needs to be preprocessed. Use the face detector of the OpenCV library to locate and crop the face area. Make sure all images have the same size and format.

[0046] Using the preprocessed dataset, start training the face recognition model.

[0047] In OpenCV, the MTCNN algorithm is used to train the model.

[0048] After training is completed, the trained model is saved to disk for subsequent calls.

[0049] (1) Loading the face recognition model: Before calling the face recognition function, first load the previously trained model and use the MTCNN algorithm to quickly identify the loaded model. Use the API provided by OpenCV to load the model file from the disk.

[0050] (2) Initialize the camera: Before performing face recognition, you need to initialize the camera to obtain the video stream. Use OpenCV's VideoCapture class to open and read the camera data, and use the MTCNN algorithm ProposalNetwork to optimize the face data read by the camera.

[0051] (3) Face detection and recognition: In each frame, a face detector is used to detect the face area. Then, the detected face is compared with the previously trained model, and the MTCNN algorithm OutputNetwork is called to calibrate the key points.

[0052] (4) Feature extraction and matching: Extract feature vectors from the detected face regions. This can be achieved by converting the face image to a grayscale image and applying preprocessing methods such as histogram equalization. Then, use the trained face recognition model to match the extracted feature vectors with known face labels.

[0053] (5) Display results: Based on the matching results, a bounding box can be drawn in the image and the recognition results can be displayed. A rectangular box is drawn around the recognized face area and the corresponding face label or identity information is displayed in the box.

[0054] (6) Real-time face recognition: Put the above steps into a loop to achieve real-time face recognition. Continuously read images from the camera, perform face detection, feature extraction and matching, and then display the recognition results.

[0055] Model applications include real-time video acquisition, real-time image capture, real-time face detection, real-time facial feature point calibration, real-time facial feature point alignment, real-time face comparison, real-time blink recognition, and real-time mouth opening recognition.

[0056] Algorithm Introduction:

[0057] MTCNN is defined as a multi-task cascade convolutional neural network. The network was the best performer when it was first created. Although it is not the best performer now, the network is of great significance. It is the first time that face detection and facial feature point location are combined, and the facial feature points obtained can be used to realize face correction.

[0058] The algorithm consists of three stages:

[0059] In the first stage, candidate frames are quickly generated through CNN.

[0060] In the second stage, a more complex CNN is used to refine the candidate windows and discard a large number of overlapping windows.

[0061] In the third stage, a more powerful CNN is used to implement the retention and removal of candidate windows and to regress the five facial key points.

[0062] The first stage is to use a convolutional neural network called PNet (Proposal Network) to obtain candidate windows and boundary regression vectors. At the same time, the candidate windows are calibrated according to the bounding box. Then non-maximum suppression is used to remove overlapping windows. The second stage is to use the R-Net (Refine Network) convolutional neural network to train the pictures containing candidate windows determined by P-Net in R-Net, and finally use the fully connected network for classification. The candidate windows are fine-tuned using the bounding box vector, and finally the overlapping windows are removed using the non-maximum suppression algorithm. In the third stage, Onet (Output Network) convolutional neural network is used. This network has one more convolution layer than R-Net and has similar functions to R-Net, except that the positions of 5 facial key points are calibrated while removing overlapping candidate windows.

[0063] Before being processed by the three convolutional networks, the MTCNN network first performs a multi-scale transformation to scale a face image into images of different sizes, thus forming an image pyramid. These images of different scales are then used as input data for three stages of training, allowing the MTCNN to detect faces of different sizes.

[0064] Loss function:

[0065] The MTCNN feature descriptor mainly consists of three parts: face-non-face binary classifier, bounding box regression, and face feature points. The loss functions of these three parts are introduced below. First, the face must be classified, that is, the binary classifier that determines whether the area contains a face. The cross entropy loss function of face classification is as follows:

[0066]

[0067] Among them, is the probability of a face appearing, and is the true label of the area. In addition to determining whether there is a face in the area, we also hope that the area demarcated by the face is as accurate as possible. This is naturally a regression problem. MTCNN uses the common bounding box regression to achieve this. The bounding box regression uses the Euclidean distance as the loss function of the distance metric, as shown in the following formula:

[0068]

[0069] Among them, is the bounding box coordinates predicted by the network, is the actual bounding box coordinates, that is, a four-tuple representing a rectangular area, the specific form is as follows:

[0070] Similar to the boundary regression process, facial key point localization is also a regression problem. The loss function of this step is still to calculate the deviation between the predicted key point position and the actual position, and the distance metric uses the Euclidean distance. The loss function in the key point localization process is as follows:

[0071]

[0072] Among them, is the prediction result, is the actual key point position. Since a total of 5 facial key points need to be predicted, each point has 2 coordinate values, so it is a 10-tuple.

[0073] The final loss that MTCNN wants to optimize is:

[0074]

[0075] Among them, N is the number of training samples, represents the importance of the task, is the sample label, and is the above loss function. When training pnet and rnet, the author did not add the landmark regression task, and the loss_weight ratio of classification and face frame regression was 1:0.5. Onet added landmark regression, and the loss_weight ratio of classification, face frame regression and key point regression was 1:0.5:0.5.

[0076] During the training process, in order to achieve better results, the MTCNN authors only backpropagate the gradients of the first 70% of samples each time to ensure that only valid data is transmitted.

[0077] Beneficial effects: Use algorithms to improve the success rate of recognition, and save user information in the company's own server to prevent public opinion incidents caused by user information leakage. At the same time, optimize the algorithm and modify the offset according to the company's different scenarios to achieve the purpose of adapting to the scenario.

[0078] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A face recognition method based on MTCNN algorithm, characterized in that: The face recognition method comprises: Label the collected samples; Training the face region detection model to obtain a face region detection training model; Apply the face region detection training model.

2. The face recognition method based on the MTCNN algorithm according to claim 1, characterized in that: The labeling of the collected samples specifically includes: Use of sample annotation tools; Face area detection sample annotation; Facial feature point calibration sample annotation.

3. The face recognition method based on the MTCNN algorithm according to claim 1, characterized in that: The training of the face area detection model specifically includes: Face region detection model training and face region detection model evaluation; Facial feature point calibration model training and facial feature point calibration model evaluation; Face comparison model training and face comparison model evaluation.

4. The face recognition method based on the MTCNN algorithm according to claim 3 is characterized in that: The face area detection model training specifically includes: Collect multiple face images, and mark the multiple face images with a unique label to obtain a face dataset; Preprocessing the face data set to obtain a preprocessed face data set; Use the OpenCV library's face detector to locate and crop the face area; A face recognition model is trained based on the preprocessed face dataset.

5. The face recognition method based on the MTCNN algorithm according to claim 4, characterized in that: The training of the face recognition model according to the preprocessed face data set specifically includes: using the MTCNN algorithm to train the model in OpenCV.

6. The face recognition method based on the MTCNN algorithm according to claim 5, characterized in that: In OpenCV, the MTCNN algorithm is used to train the model, specifically including: Use MTCNN algorithm to quickly identify loading models; Use OpenCV's VideoCapture class to open and read camera data, and use the MTCNN algorithm ProposalNetwork to optimize the face data read by the camera; In each frame, a face detector is used to detect the face area; Compare the detected face with the previously trained model and call the MTCNN algorithm Output Network to calibrate the key points; Extract feature vectors from detected face regions; Based on the matching results, a bounding box is drawn in the image and the recognition results are displayed. It will be put into a loop to realize real-time face recognition function.

7. The face recognition method based on the MTCNN algorithm according to claim 6, characterized in that: The extracting of the feature vector from the detected face region specifically includes: converting the face image into a grayscale image and applying a histogram equalization preprocessing method; Use the trained face recognition model to match the extracted feature vectors with known face tags.

8. The face recognition method based on the MTCNN algorithm according to claim 1, characterized in that: The applying of the face area detection training model specifically includes: Real-time video acquisition, real-time image capture, real-time face detection, real-time facial feature point calibration, real-time facial feature point alignment, real-time face comparison, real-time blink recognition and real-time mouth opening recognition.

Images