Mobile terminal face recognition and photo quality detection method and system based on deep learning

Abstract

The invention relates to the field of mode recognition, in particular to a mobile terminal face recognition and photo quality detection method and system based on deep learning, and the method comprises the steps: collecting an entrance video image of an examinee in an examination room in real time, and obtaining a face image I; inputting the image I into an improved RetinaFace network for face detection to obtain a face positioning frame and face region feature point coordinates; performing multi-dimensional quality detection on the face image; carrying out image quality enhancement on the face image passing the quality detection and then carrying out alignment processing; inputting the aligned face image into an improved MobileNetV3 network to extract a depth feature, and obtaining a face feature vector; calculating the cosine similarity between the face feature vector and a feature vector in a preset face feature library; and judging whether the person is the same person. The problems that the recognition failure rate is high and frequent recollection is needed due to poor image quality are effectively solved, and the smoothness and efficiency of examinee entrance verification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and pattern recognition, in particular to a mobile terminal face recognition and photo quality detection method and system based on deep learning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, face recognition technology based on deep learning has become an important part of identity authentication field due to its uniqueness of biological characteristics, and has shown high efficiency, accuracy and intelligent application potential in examination entry verification, security anti-counterfeiting and other scenarios.

[0003] In the examination scene, the camera is used to collect real-time video or photos of the examinee, and the identity verification is automatically completed through face recognition technology, which has gradually replaced the traditional manual verification method. This method not only can effectively prevent irregular behavior, but also significantly improves the verification efficiency.

[0004] However, in actual application, due to the performance limitation of mobile terminal device and the complex and changeable examination environment, the collected images often have a series of quality problems, such as uneven illumination, overexposure or underexposure, blurred face, non-frontal pose, closed eyes, and improper face proportion in the image. These quality problems will seriously interfere with the accuracy of face detection and feature extraction, directly leading to the decrease or failure of recognition, so that the system has to frequently require re-collection of images, which affects the entry efficiency.

[0005] Therefore, the prior art lacks an end-to-end solution that can be integrated into a mobile device, automatically assess and enhance the image quality before recognition, and thus guarantee the reliability of high-precision face recognition. How to cooperatively complete photo quality detection and high-precision recognition under the premise of ensuring speed has become a technical problem to be solved in the field. SUMMARY

[0006] To solve the above problems, the present application provides a mobile terminal face recognition and photo quality detection method and system based on deep learning.

[0007] In a first aspect, the present application provides a mobile terminal face recognition and photo quality detection method based on deep learning, comprising the following steps: S1, collecting the video images of the examinees in the examination room in real time, and sequentially performing Gaussian filter smoothing and histogram equalization processing on the collected images to obtain preprocessed images I containing human faces; S2, inputting the image I into the improved RetinaFace network for face detection, if the detection is successful, outputting the face positioning frame and the face region feature point coordinates including the left and right eye, nose, left and right mouth corner five feature point coordinates; if the detection fails, returning to S1; S3, cropping a face image from the image I based on the face positioning frame; and performing multi-dimensional quality detection on the cropped face image; the multi-dimensional quality detection at least includes face size compliance detection, closed eye detection and head posture detection; if any detection fails, returning to S1; S4, performing image quality enhancement processing on the face image passing the quality detection to obtain an enhanced face image; S5, aligning the enhanced face image to a standard template based on face region feature point coordinates through affine transformation to obtain an aligned face image; S6, inputting the aligned face image into an improved MobileNetV3 network to extract deep features to obtain a face feature vector; S7, calculating the cosine similarity of the face feature vector and the feature vector in the preset face feature library; S8, if the similarity is greater than or equal to a set threshold, determining that the person is the same, otherwise determining that the person is different and outputting a prompt information.

[0008] The image quality detection is taken as a necessary pre-process for face recognition, forming a complete closed-loop system, effectively solving the problem of high recognition failure rate and frequent re-collection caused by poor image quality, greatly improving the smoothness and efficiency of examinee entry verification. Through multi-dimensional quality detection and image enhancement, it is ensured that the image input into the recognition module is compliant and clear, which improves the accuracy of subsequent feature extraction and comparison from the source, and overcomes the interference of factors such as light, posture and blur. By using the improved lightweight network, high precision is ensured while ensuring that the algorithm can run in real time on mobile devices such as mobile phones and tablets, meeting the actual needs of speed in the examination scene.

[0009] As a further limitation of the technical scheme of the application, the improved RetinaFace network is trained by the following method: An annotated face image dataset is used to train the network, and the annotation information of the dataset includes face positioning frame and coordinates of five feature points of left eye, right eye, nose, left corner of mouth and right corner of mouth; During the training process, the network parameters are optimized by minimizing the multi-task loss function L through the back propagation algorithm, and the multi-task loss function L is:

[0010] Wherein, is the face classification loss, represents the probability that the predicted i th Anchor is a face, represents the true value; is the face frame regression loss, wherein and representing the position of the predicted frame corresponding to the positive sample Anchor and the position of the real labeled frame, is a face key point regression loss, is a dense regression loss, is a coefficient for balancing the weights of each loss term.

[0011] By introducing data containing key point labeling and a multi-task loss function for training, the network can not only accurately locate the face frame, but also simultaneously regress the facial feature points.

[0012] As a further limitation of the technical scheme of the application, the improved RetinaFace network in S2 takes MobileNetV3 as the backbone network, combines the feature pyramid network FPN for multi-scale feature fusion to obtain a fused feature map, and introduces an SSH module on the fused feature map to enhance the context information perception ability; The specific steps for face detection include: S21, inputting the image I into the MobileNetV3 network to obtain multi-scale features; S22, inputting the multi-scale features into the feature pyramid network FPN for feature fusion to obtain a multi-scale fused feature map; S23, introducing an SSH module on the fused feature map to enhance the feature expression ability through context modeling to obtain an enhanced feature map for prediction; S24, predicting the face classification confidence, face bounding box regression offset and face key point regression offset on the enhanced feature map in parallel; after non-maximum suppression NMS processing of the prediction results, outputting the final face positioning frame and face region feature point coordinates.

[0013] The combination of MobileNetV3 backbone network, FPN and SSH module significantly enhances the network's detection ability and context understanding ability for multi-scale faces on the basis of lightweight, and improves the detection rate in complex examination room environments. Through the NMS post-processing step, a large number of overlapping redundant false detection frames are effectively filtered out, ensuring that each face output corresponds to only one set of optimal positioning frame and key point coordinates, providing clean and accurate input for the subsequent process.

[0014] As a further limitation of the technical scheme of the application, the steps of S24 include: S241, presetting a plurality of anchor frames at each pixel point of the enhanced feature map; using a single-level detector design, predicting the face classification confidence, face bounding box regression offset and face key point regression offset for each anchor frame in parallel on the enhanced feature map through a convolution layer; S242, apply the predicted face bounding box regression offset and the face key point regression offset to the corresponding preset anchor box, and decode to obtain a preliminary prediction result, wherein the preliminary prediction result comprises a preliminary face positioning box, a corresponding classification confidence, and a preliminary face region feature point coordinate; S243, apply a non-maximum suppression algorithm to all preliminary prediction results, filter out redundant preliminary prediction results based on the overlap degree of the preliminary face positioning box and the classification confidence, and output a final face positioning box, a classification confidence, and a face region feature point coordinate.

[0015] As a further limitation of the technical scheme of the application, the multi-dimensional quality detection in S3 specifically comprises: S31, cutting out a corresponding face region image from the image I based on the face positioning box; S32, calculating the width and height of the cut-out face image and the width and height proportions of the original image; if the width and height proportions are both within a preset first interval, and the width-to-height ratio is within a second interval, it is determined that the head portrait size is compliant; S33, inputting the cut-out face image into a pre-trained CNN network to obtain the closed-eye probability values of the left and right eyes; if the closed-eye probability values of the two eyes are both less than a set threshold, it is determined that the eyes are open, i.e., compliant; S34, inputting the face region feature point coordinate into a pre-trained 3D key point regression network to predict the yaw angle, pitch angle, and roll angle of the head; if each angle is within a corresponding preset compliant range, it is determined that the head posture is compliant; S35, if the detection results of S32, S33, and S34 are all compliant, then the quality detection is passed.

[0016] Through the quantitative detection of three dimensions of size, open eyes, and posture, it is automatically and objectively determined whether the collected image is qualified, completely replacing subjective human judgment, and ensuring the source quality of identity authentication data. The closed-eye detection can effectively prevent the behavior of using photos to cheat; the head posture detection ensures that the captured face is a front face, greatly improving the accuracy of subsequent feature extraction and comparison; the size detection avoids recognition errors caused by too small or too large face.

[0017] As a further limitation of the technical scheme of the application, the step S4 comprises: S41, dividing the face image into grid blocks of a set pixel, calculating the gray mean value of each grid block, calculating the standard deviation of all grid block mean values, and if the standard deviation is greater than a first threshold, it is determined that the light is uneven, and a Retinex algorithm is used for illumination compensation; The Retinex algorithm is used for uneven images. The specific steps are: The image is decomposed into a reflection component and a brightness component and illumination component The illumination component is estimated using Gaussian filtering:

[0018] wherein,

[0019] represents convolution, and G is a Gaussian kernel; The reflection component is recovered: and finally, the image is exponentiated to obtain the illumination-compensated image, wherein ε is a constant. S42, the gray mean value μ of the whole face region is calculated, if μ is less than the lower limit value of the preset range, gamma correction is used to enhance the brightness, if μ is greater than the upper limit value of the preset range, gamma correction is used to reduce the brightness;

[0020] S43, the Laplace gradient variance of the face image is calculated, if the gradient variance is less than the second threshold value, it is determined that the image is blurred, and a non-local mean denoising algorithm or a blind deblurring algorithm is used for clear processing; S44, after the processing of S41-S43, the enhanced face image is obtained. The face images collected under different illumination conditions are normalized to a standard brightness range. The blur detection and deblurring algorithm based on Laplace variance automatically repairs the blur caused by inaccurate focusing or motion, and improves the image clarity. The features of the enhanced image are more prominent and stable, which directly improves the robustness of the feature extraction module and the overall performance of the recognition system.

[0021] As a further limitation of the technical scheme of the application, the step of S5 includes:

[0022] S51, five feature point coordinates of a predefined standard face template are obtained to obtain standard coordinates P_std; S52, the corresponding five feature point coordinates in the face image obtained by S2 are read to obtain input coordinates P_input; S53, an affine transformation matrix M is calculated according to the standard coordinates and the input coordinates;

[0023] S54, the affine transformation matrix M is used to perform geometric transformation on each pixel in the face image after quality enhancement, and the pixel value of the transformed image is calculated by using bilinear interpolation, and finally the aligned standard size face image is obtained.

[0024] ​All face images are geometrically normalized to a standard template by affine transformation, which eliminates the face geometric deformation caused by slight changes in posture. This makes the extracted feature vector pose-independent and more focused on the features of the identity itself, greatly improving the accuracy of feature comparison and the reliability of the recognition system.

[0025] As a further limitation of the technical scheme of the application, in S6, the improvement of the MobileNetV3 network includes: Using h-swish activation function instead of the original ReLU activation function; Introducing SE attention mechanism in the bottleneck module; Adding a depth separable convolution layer after the last three convolution layers; Using a global depth separable convolution layer to replace the global average pooling layer.

[0026] The h-swish activation function reduces the computational overhead; the SE attention mechanism makes the network pay more attention to feature channels with strong discriminability; and the depth separable convolution and the operation of replacing GAP enhance the representation ability of fine-grained features. These improvements make the feature extraction network extract 128-dimensional feature vectors with higher discriminability while maintaining high-speed operation on mobile terminals, thereby significantly improving the accuracy of face recognition while ensuring speed.

[0027] As a further limitation of the technical scheme of the application, the face feature library in S7 is constructed by the following method: a) The data source is the standard face photo submitted by the examinee during registration; b) The standard face photo is sequentially processed through steps S1 to S6, and the corresponding face feature vector is extracted, and the similarity with the existing face feature vector in the face feature library is calculated. If the similarity with any existing face feature vector exceeds the preset threshold, it is determined as duplicate or low-quality data and filtered out; otherwise, c) is executed; c) The extracted face feature vector is associated with the examinee ID and the storage timestamp, and stored in the face feature library, and an index is established for all face feature vectors in the face feature library.

[0028] Filtering duplicate / low-quality data before storage ensures that each template in the feature library is of high quality and unique, improving the accuracy of the comparison stage. The structured storage method (association of ID and timestamp) and the establishment of vector index support efficient management of large-scale examinee data and millisecond-level real-time retrieval, meeting the needs of practical applications.

[0029] In the second aspect, the technical scheme of the application also provides a mobile terminal face recognition and photo quality detection system based on deep learning, which comprises: An image acquisition and preprocessing module is configured to acquire a video image of an examinee entering an examination room in real time, and sequentially perform Gaussian filter smoothing and histogram equalization on the acquired image to obtain a preprocessed image I containing a human face; A face detection module including an improved RetinaFace network is configured to receive the image I and perform face detection. If the detection is successful, a face positioning frame and face region feature point coordinates including left and right eye, nose, and left and right corner of mouth are output. If the detection fails, the image acquisition and preprocessing module is triggered to work again. An image quality detection module is configured to crop a face image from the image I based on the face positioning frame, and perform multi-dimensional quality detection on the cropped face image. The multi-dimensional quality detection includes at least face size compliance detection, closed eye detection, and head posture detection. If any of the detection fails, the image acquisition and preprocessing module is triggered to work again. An image enhancement module is configured to perform image quality enhancement on the face image that passes the quality detection to obtain an enhanced face image. A face alignment module is configured to align the enhanced face image to a standard template based on the face region feature point coordinates through affine transformation to obtain an aligned face image. A feature extraction module including an improved MobileNetV3 network is configured to extract deep features of the aligned face image to obtain a face feature vector. A feature comparison module is configured to calculate a cosine similarity between the face feature vector and a feature vector in a face feature library. An identity authentication module is configured to determine that the person is the same when the cosine similarity is greater than or equal to a set threshold, and otherwise determine that the person is different and output a prompt information.

[0030] As can be seen from the above technical solutions, the present application has the following advantages: the present application optimizes each link from image acquisition and preprocessing, face detection and positioning, image quality detection and enhancement, face alignment, feature extraction to feature comparison, forms a mobile face recognition and photo quality detection system, effectively solves the problem of high recognition failure rate and frequent re-acquisition caused by poor image quality, greatly improves the smoothness and efficiency of examinee entry verification. Through close cooperation and collaborative optimization of each link, the system can stably operate in complex actual application scenarios, effectively solves the problems of light change, posture diversity, and image resolution in the prior art, and significantly improves the accuracy and efficiency of face recognition. In the training process, the present application uses a face data set containing different light, posture, and expression, and applies data enhancement methods such as random cropping, horizontal flipping, brightness, and contrast disturbance in the training process, so that the model can better adapt to image changes in various actual scenarios. Attached Figure Description

[0031] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a method provided in one embodiment of the present invention.

[0033] Figure 2 Diagram of the improved RetinaFace network structure.

[0034] Figure 3 A flowchart illustrating a method provided in another embodiment of the present invention.

[0035] Figure 4 A block diagram of a system provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0038] like Figure 1 As shown, this embodiment of the invention provides a mobile face recognition and photo quality detection method based on deep learning, including the following steps: S1. Real-time acquisition of video images of candidates entering the examination room, and sequential Gaussian filtering smoothing and histogram equalization processing of the acquired images to obtain preprocessed image I containing human faces; In this step, Gaussian filtering and histogram equalization are used to suppress noise in the image, resulting in a relatively ideal image I containing a face, with a size of [size missing]. ; S2, input the image I into the improved RetinaFace network for face detection, if the detection is successful, output the face positioning frame and the face region feature point coordinates, including the left and right eyes, nose, left and right corners of the mouth five feature point coordinates; if the detection fails, return to S1; In this step, the improved RetinaFace network uses the labeled data to automatically adjust all parameters (weights) in the network through the optimization algorithm, so that the predicted value output by the network is closer and closer to the true labeled value. This process is completed once on a powerful server (such as a Linux system equipped with a GPU), and then the trained network model is deployed to the mobile end for inference. The specific training steps are as follows: (1) Data preparation and loading Input: labeled face data set. Each sample contains: an image, the coordinates of one or more real face frames , and the coordinates of the 5 key points (left eye, right eye, nose, left corner of the mouth, and right corner of the mouth) corresponding to each real face frame.

[0039] Operation: Write a data loader to read out the images and their corresponding label information in batches (Batch) and send them to the GPU.

[0040] (2) Forward propagation Input: a batch of images.

[0041] Operation: input the image into the improved RetinaFace network.

[0042] Output: The network will output four prediction values for each anchor frame on the feature map: Classification prediction: the probability that each anchor frame is a face or background.

[0043] Boundary box regression prediction: the offset of each anchor frame .

[0044] Key point regression prediction: the offset of the 5 key points of each anchor frame.

[0045] Dense regression prediction: more detailed pixel-level positioning information.

[0046] (3) Anchor frame matching and label assignment Operation: Calculate the overlap (IoU) of all preset anchor frames and all real face frames in the image.

[0047] Positive sample anchor frame: the IoU of any real face frame is greater than the high threshold (such as 0.5). They are assigned a positive classification label (=1), and are associated with the real frame with the largest IoU and its key points.

[0048] Negative sample anchor boxes: Anchor boxes whose IoU with all ground truth boxes are below a low threshold (e.g. 0.3). They are assigned a negative classification label (=0).

[0049] Ignore sample anchor boxes: Anchor boxes whose IoU is between the high and low thresholds. They are not involved in the calculation of classification and regression loss in training.

[0050] Output: Each anchor box has a clear classification label, and the coordinates of the target ground truth box and key points it needs to regress to.

[0051] (4) Loss calculation Operation: Use the multi-task loss function L to calculate the total error between the predicted value of the current batch and the true label assigned in step (3).

[0052] The multi-task loss function L is:

[0053] where, is the face classification loss, represents the probability that the i th Anchor is a face, represents the true value; is the face box regression loss, where and represent the position of the predicted box corresponding to the positive sample Anchor and the position of the true annotation box, is the face key point regression loss, is the dense regression loss, is the coefficient for balancing the weights of each loss term. Only calculate the classification error on positive and negative sample anchor boxes. This embodiment uses cross-entropy loss. Only calculate the bounding box regression error on positive sample anchor boxes. This embodiment uses Smooth L1 Loss. Only calculate the key point regression error on positive sample anchor boxes. This embodiment uses Smooth L1 Loss. Calculate the more dense regression error.

[0054] Output: A scalar value representing the total error of the current network prediction, which is the loss value L.

[0055] (5) Backpropagation and parameter optimization Backpropagation: Calculate the gradient of the loss value L with respect to the millions of parameters in the network. The gradient indicates how each parameter should be fine-tuned to reduce the loss.

[0056] Parameter update: using the Adam optimizer, according to the calculated gradient and the preset learning rate, according to the rule of new parameter = old parameter - learning rate x gradient, update all parameters in the network.

[0057] (6) Loop iteration Repeat steps (2)-(5) until the performance of the model on the validation set no longer improves or reaches the preset training round (Epoch). Each Epoch will traverse the training dataset once.

[0058] (7) Model saving After training, save the final optimized network parameters (model weights) as a file. This saved file is the improved RetinaFace network model that can be deployed to the mobile end for face detection.

[0059] S3, cut out a face image from the image I based on the face positioning frame; and perform multi-dimensional quality detection on the cut-out face image; the multi-dimensional quality detection at least includes face size compliance detection, closed eye detection and head posture detection; if any one detection fails, return to S1; In the embodiment of the application, the multi-dimensional quality detection in S3 specifically includes: S31, cutting out a corresponding face region image from the image I based on the face positioning frame; S32, calculating the width, height, width ratio and height ratio of the cut-out face image to the original image; if the width ratio and height ratio are both within a preset first interval, and the width to height ratio is within a second interval; determine that the avatar size is compliant; S33, inputting the cut-out face image into a pre-trained CNN network to obtain the closed eye probability values of the left eye and the right eye; if the closed eye probability values of the two eyes are both less than a set threshold, it is determined that the eyes are open, that is, compliant; S34, inputting the face region feature point coordinates into a pre-trained 3D key point regression network to predict the yaw angle, pitch angle and roll angle of the head; if each angle is within the corresponding preset compliance range, it is determined that the head posture is compliant; S35, if the detection results of S32, S33 and S34 are all compliant, the quality detection is passed.

[0060] That is, the face positioning frame (upper left corner and lower right corner coordinates) and the original image size h x w obtained by S2.

[0061] Calculate the positioning frame pixel size: width , height . Based on the pixel size of the detection frame and the relative image ratio, evaluate the compliance of the avatar size in the image.

[0062] The proportion of the face frame relative to the original image: Compliance determination: Based on the examination room standards (such as the face accounting for no less than 15% and no more than 80%, and the aspect ratio between 0.8 and 1.2), if the requirements are met, proceed to the next step; otherwise, return to S1. The cropped face image and five feature points (left and right eyes, nose, and left and right corners of the mouth) are input into the lightweight CNN network structure MobileNet, which outputs the probability of both eyes being closed (p_left, p_right). A threshold is set: if p < 0.3, it is considered as open eyes; otherwise, it is considered as closed eyes.

[0063] Based on five 2D feature points, the head Euler angles (yaw, pitch, and roll) are predicted using a pre-trained 3D keypoint regression network.

[0064] Posture compliance range: Based on the standard set threshold (e.g., yaw∈[-15°,15°], pitch∈[-10°,10°], roll∈[-5°,5°]), exceeding this range is judged as an abnormal posture. If both eyes are open and the posture is normal, the image quality detection is successful, and the process continues to S4; otherwise, it proceeds to S1.

[0065] S4. Perform image quality enhancement processing on the face images that pass the quality inspection to obtain the enhanced face images; the specific steps are as follows: S41. Divide the face image into a 3×3 grid, calculate the gray value of each grid block, and calculate the standard deviation of the mean of all grid blocks. If the standard deviation is greater than the first threshold (set to 20 here), it is determined that the lighting is uneven, and the Retinex algorithm is used for lighting compensation. The Retinex algorithm is used for non-uniform images. The specific steps are as follows: Image Decomposed into reflection components and light component :

[0066] Using Gaussian filtering to estimate illumination components:

[0067] in, Represents convolution, where G is the Gaussian kernel; Restore the reflection component: Finally, the image is obtained by indexation, where ε is a constant.

[0068] S42. Calculate the average grayscale value μ of the entire face area. If μ < the lower limit of the preset range, gamma correction is used to enhance the brightness; if μ > the upper limit of the preset range, gamma correction is used to reduce the brightness. Here, the preset range: μ∈ [80, 180], σ∈ [40, 80] (adjust according to the standard of the test room illumination). If too dark (μ<80): adopt gamma correction (γ<1, such as γ=0.7) to enhance brightness.

[0069] If too bright (μ>180): adopt gamma correction (γ>1, such as γ=1.5) to reduce brightness.

[0070] S43, calculate the Laplacian gradient variance of the face image, if the gradient variance is less than a second threshold (set to 100 in this embodiment), determine that the image is blurred, and adopt a non-local mean denoising algorithm or a blind deblurring algorithm for sharpening processing; S44, obtain the enhanced face image after the processing of S41-S43.

[0071] S5, based on the face region feature point coordinates, align the enhanced face image to a standard template through affine transformation to obtain an aligned face image; the specific steps are as follows: S51, obtain five feature point coordinates of a predefined standard face template to obtain standard coordinates (left and right eyes, nose, left and right corners of the mouth, set according to dataset statistics, such as normalized to 150x150 pixels).

[0072] S52, read five feature point coordinates in the face image obtained by S2 to obtain input coordinates .

[0073] S53, calculate an affine transformation matrix M according to the standard coordinates and the input coordinates; Affine transformation formula:

[0074] where M is a 2x3 matrix [[a,b,c], [d,e,f]].

[0075] Solve M by least squares method: Substitute 5 sets of point pairs into the equation to construct an overdetermined equation group Ax=b, where x=[a,b,c,d,e,f]^T.

[0076] Solve to obtain the M matrix.

[0077] S54, use the affine transformation matrix M to perform geometric transformation on each pixel in the face image after quality enhancement, and calculate the pixel value of the transformed image by bilinear interpolation, to finally obtain an aligned standard size face image.

[0078] For each pixel in the face image , the transformed coordinates are calculated by M : ; The pixel values are filled by bilinear interpolation to obtain the aligned face image.

[0079] S6, the aligned face image is input into the improved MobileNetV3 network to extract deep features, and a face feature vector is obtained; the improvement of the MobileNetV3 network includes: Use h-swish activation function instead of original ReLU activation function; Introduce SE attention mechanism in bottleneck module; Add a depth separable convolution layer after the last three convolution layers; Use global depth separable convolution layer instead of global average pooling layer.

[0080] The improved MobileNetV3 network adopts the following strategies during training: The initial learning rate is 1e-3, and the cosine annealing strategy is used for learning rate decay; The training data uses a face dataset containing different light, posture and expression; Pre-train the basic MobileNetV3 on ImageNet, and then fine-tune it on the face dataset. Random cropping, horizontal flipping, brightness and contrast disturbance data are applied during training.

[0081] In S6, the improved MobileNetV3 network model for face feature extraction is designed as shown in Table 1. The network model uses separable convolution instead of average pooling layer, that is, a 7x7x512 (512 represents the number of input feature map channels) separable convolution layer is used instead of global average pooling. This allows the network to learn different weights on its own. The separable convolution layer (global depthwise convolution) here, global represents global, depthwise represents depthwise, that is, channel-by-channel convolution. The separable convolution is used in the following way: a 7x7x512 convolution kernel is used instead of a 7x7x512x512 convolution kernel. In addition, the channel expansion multiple is reduced to facilitate the extraction of detailed features. The training process uses an Arcface loss function, whose formula is as follows:

[0082] In the formula, the value of the correct label is The cosine function is monotonically decreasing in (0, 1), and m, will make the value becomes smaller, so that the loss will become very large, angle distance than the cosine distance in the impact of the angle is more obvious; the angle between the feature vector of i samples and the center vector of its real class , s is the scale factor. the angle between the feature vector and the real class center vector, s is the scale factor.

[0083] Table 1: Improved MobileNetV3 face feature extraction network structure

[0084] S7, calculate the face feature vector and the cosine similarity of the feature vector in the preset face feature library; The face feature similarity calculation method used is cosine similarity, and the n-dimensional vector calculation formula is as follows:

[0085] S8, if the similarity is greater than or equal to the set threshold, it is determined that it is the same person, otherwise it is determined that it is different people and output prompt information.

[0086] In some embodiments, the improved RetinaFace network in S2 uses MobileNetV3 as the backbone network, combines the feature pyramid network FPN for multi-scale feature fusion to obtain a fused feature map, and introduces an SSH module on the fused feature map to enhance the context information perception ability; The specific steps of face detection include: S21, input the image I into the MobileNetV3 network to obtain multi-scale features; S22, input the multi-scale features into the feature pyramid network FPN for feature fusion to obtain a multi-scale fused feature map; S23, introduce an SSH module on the fused feature map, enhance the feature expression ability through context modeling to obtain an enhanced feature map for prediction; S24, predict the face classification confidence, face bounding box regression offset and face key point regression offset on the enhanced feature map in parallel; after non-maximum suppression NMS processing on the prediction results, output the final face positioning frame and face region feature point coordinates.

[0087] In some embodiments, the step S24 includes: S241, preset a plurality of anchor boxes at each pixel point of the enhanced feature map; adopt a single-level detector design, and predict the face classification confidence, face bounding box regression offset and face key point regression offset for each anchor box on the enhanced feature map in parallel through a convolution layer; S242, applying the predicted face bounding box regression offset and the face key point regression offset to the corresponding preset anchor box, decoding to obtain a preliminary prediction result, the preliminary prediction result including a preliminary face positioning box, a corresponding classification confidence and a preliminary face region feature point coordinate; S243, applying a non-maximum suppression algorithm to all preliminary prediction results, filtering out redundant preliminary prediction results based on the overlap degree of the preliminary face positioning box and the classification confidence, and outputting a final face positioning box, a classification confidence and a face region feature point coordinate.

[0088] It should be noted that in the above S2, the improved RetinaFace network model for face detection is designed as shown in Figure 2 As shown in the figure, the backbone network adopts MobileNetV3 as a multi-stage feature extraction network, which is convenient for mobile terminal use. Compared with MobileNetV2, it improves the target detection accuracy and efficiency, and then adopts FPN network which predicts independently at each layer. The top layer feature is fused with the low layer feature through upsampling, and SSH improves the face detection by introducing context modeling in the feature map. The whole network adopts single-stage design, which performs dense face position and scale sampling on the feature pyramid. Compared with the two-stage cascade method, this design has obtained good performance and speed improvement, and the network is trained by using the multi-task loss of strong supervision and self-supervision as the loss function.

[0089] In some embodiments, the face feature library in S7 is constructed by the following method: a) The data source is the standard face photo submitted by the examinee when registering; b) The standard face photo is processed through the steps of S1 to S6 in turn, the corresponding face feature vector is extracted, the similarity with the existing face feature vector in the face feature library is calculated, if the similarity with any existing face feature vector exceeds the preset threshold, it is judged as repeated or low-quality data and filtered out, not stored in the library; otherwise, c) is executed; c) The extracted face feature vector is associated with the examinee ID and the storage timestamp, and stored in the face feature library, and an index is established for all face feature vectors in the face feature library to speed up the retrieval. When the examinee's information changes (such as changing the photo), the old feature vector is deleted, and the new vector is extracted and stored again to ensure data consistency. The feature vector is checked for quality (such as similarity threshold filtering, comparison with existing vectors in the library to avoid repeated or low-quality data) before storage.

[0090] Specifically, the model training environment used was a Linux system. The improved RetinaFace face detection network model was trained on a dataset containing approximately 30,000 images and 400,000 high-precision face localization boxes. The improved MobileNetV3 face feature extraction network model was trained on a dataset containing approximately 50,000 individuals, each with approximately 12 face photos from different scenes. The candidate's registration photo was processed using the above steps to obtain a 128-dimensional feature vector, which was used to construct a face feature database and assign labels. A face similarity threshold of 0.85 was set. If the similarity of the face feature vector between the collected on-site photo and the registration photo was greater than or equal to 0.85, they were considered to be the same person, and face recognition passed. If the similarity was less than 0.85, they were considered not to be the same person, and face recognition failed. Actual candidate entry verification monitoring video with a resolution of 1080×720 was selected, and a strategy of sampling 10 frames per second was used as the input image sequence, taking a single candidate target as an example. Figure 3 As shown, the specific implementation steps are as follows: Step 1: Take the video image of the examinee entering the examination room with a resolution of 1080×720 and sample it at 10 frames per second as the input, and use Gaussian filtering smoothing and histogram equalization to suppress the noise of the image. Step 2: Input the pre-processed video image of the examinee entering the examination room into the pre-trained improved RetinaFace face detection network model, and output the target examinee's face bounding box and the coordinates of five feature points of the face region: left and right eyes, nose, and left and right corners of the mouth. If the face detection is successful, continue to step 3; otherwise, go back to step 1. Step 3: Crop the face image using the face localization bounding box obtained in Step 2 and perform image quality detection. The evaluation criteria are: the pixel size of the detection bounding box must satisfy an aspect ratio > 0.8 and < 1.2; the head's relative proportion to the image must be > 15% for the face image size to be compliant. Use a CNN probabilistic prediction algorithm to classify the eye image, calculate the probability of closed eyes to detect if the eyes are closed, and use a 3D keypoint regression network algorithm to predict the pose angle to detect if the head pose is normal. If the image quality detection is successful, continue to Step 4; otherwise, proceed to Step 1. Step 4: Perform light uniformity detection and compensation, brightness and blur detection and correction on the face image to obtain the processed face image; Step 5: Align the face image using affine transformation based on the coordinates of five feature points: left and right eyes, nose, and left and right corners of the mouth, to obtain the aligned face image. Change the size of the face image to 112×112. Step 6: The aligned 112×112 face image is input into the pre-trained improved MobileNetV3 network to extract face features and obtain a 128-dimensional feature vector of the face. Step 7: The obtained face feature vector is compared with the vectors in the face feature library to obtain a face feature similarity, and the feature similarity is calculated in a cosine similarity manner; Step 8: The face feature similarity comparison result is processed, and when the similarity is greater than or equal to a set threshold of 0.85, the output result is the same person, and when the similarity is less than 0.85, the output result is not the same person.

[0091] As Figure 4 shown, the embodiment of the application also provides a mobile terminal face recognition and photo quality detection system based on deep learning, comprising: An image acquisition and preprocessing module is configured to acquire real-time video images of examinees entering an examination room, and sequentially perform Gaussian filter smoothing and histogram equalization processing on the acquired images to obtain a preprocessed human face image I; A face detection module comprising an improved RetinaFace network is configured to receive the image I and perform face detection, and if the detection is successful, output a face positioning frame and face region feature point coordinates including left and right eye, nose, and left and right corner of the mouth; if the detection fails, trigger the image acquisition and preprocessing module to work again; An image quality detection module is configured to crop a face image from the image I based on the face positioning frame, and perform multi-dimensional quality detection on the cropped face image; the multi-dimensional quality detection at least includes face size compliance detection, closed eye detection, and head posture detection; if any of the detection fails, trigger the image acquisition and preprocessing module to work again; An image enhancement module is configured to perform image quality enhancement processing on the face image that passes the quality detection to obtain an enhanced face image; A face alignment module is configured to align the enhanced face image to a standard template based on the face region feature point coordinates through affine transformation to obtain an aligned face image; A feature extraction module comprising an improved MobileNetV3 network is configured to perform deep feature extraction on the aligned face image to obtain a face feature vector; A feature comparison module is configured to calculate the cosine similarity between the face feature vector and the feature vectors in the face feature library; An identity authentication module is configured to determine that the two persons are the same when the cosine similarity is greater than or equal to a set threshold, and otherwise determine that the two persons are different and output a prompt information.

[0092] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims define the scope of the application. Without departing from the spirit and scope of the application as disclosed herein, the application can be used with or in conjunction with other systems, devices, applications, components, apparatuses, software, methodologies, or the like.

Claims

1. A method for mobile face recognition and photo quality detection based on deep learning, characterized in that, The method comprises the following steps: S1, collecting a video image of an examinee entering an examination room in real time, and sequentially performing Gaussian filter smoothing and histogram equalization processing on the collected image to obtain a preprocessed image I containing a human face; S2, inputting the image I into an improved RetinaFace network for face detection, if the detection is successful, outputting a face positioning frame and face region feature point coordinates including left and right eye, nose, left and right corner of the mouth five feature point coordinates; if the detection fails, returning to S1; S3, cropping a face image from the image I based on the face positioning frame; and performing multi-dimensional quality detection on the cropped face image; the multi-dimensional quality detection at least includes face size compliance detection, closed eye detection and head posture detection; if any one detection fails, returning to S1; S4, performing image quality enhancement processing on the face image passing the quality detection to obtain an enhanced face image; S5, based on the face region feature point coordinates, aligning the enhanced face image to a standard template through affine transformation to obtain an aligned face image; S6, inputting the aligned face image into an improved MobileNetV3 network to extract deep features to obtain a face feature vector; S7, calculating the cosine similarity of the face feature vector and the feature vector in the preset face feature library; S8, if the similarity is greater than or equal to a set threshold, it is determined that the same person, otherwise it is determined that different people and output a prompt information. 2.The method of claim 1, wherein, The improved RetinaFace network is obtained by training in the following way: Using a labeled face image dataset to train the network, the labeling information of the dataset includes face positioning frame and coordinates of five feature points of left eye, right eye, nose, left corner of the mouth and right corner of the mouth; During the training process, the network parameters are optimized by minimizing the multi-task loss function L through the back propagation algorithm, and the multi-task loss function L is: wherein, is a face classification loss, represents the probability that the i-th Anchor is a face, i is a true value; is a face bounding box regression loss, wherein and represent the position of the predicted bounding box and the position of the true bounding box corresponding to the positive sample Anchor, is a face key point regression loss, is a dense regression loss, is a coefficient for balancing the weights of each loss term.​ 3.The method of claim 2, wherein, The improved RetinaFace network in S2 uses MobileNetV3 as the backbone network, combines feature pyramid network FPN for multi-scale feature fusion to obtain a fused feature map, and introduces SSH module on the fused feature map to enhance the context information perception ability; The specific steps of face detection include: S21, inputting the image I into the MobileNetV3 network to obtain multi-scale features; S22, inputting the multi-scale features into the feature pyramid network FPN for feature fusion to obtain a multi-scale fused feature map; S23, introducing the SSH module on the fused feature map to enhance the feature expression ability through context modeling to obtain an enhanced feature map for prediction; S24, predicting face classification confidence, face bounding box regression offset and face key point regression offset on the enhanced feature map in parallel; after non-maximum suppression NMS processing on the prediction results, outputting the final face positioning frame and face region feature point coordinates. 4.The method of claim 3, wherein, The steps of S24 include: S241, preset a plurality of anchor boxes for each pixel point of the enhanced feature map; a single-stage detector is designed, and a convolution layer is used to predict a face classification confidence, a face bounding box regression offset, and a face key point regression offset for each anchor box in parallel on the enhanced feature map; S242, the predicted face bounding box regression offset and face key point regression offset are applied to the corresponding preset anchor box to obtain a preliminary prediction result by decoding, the preliminary prediction result including a preliminary face positioning box, a corresponding classification confidence, and a preliminary face region feature point coordinate; S243, a non-maximum suppression algorithm is applied to all preliminary prediction results, redundant preliminary prediction results are filtered out based on the overlap degree of the preliminary face positioning box and the classification confidence, and a final face positioning box, a classification confidence, and a face region feature point coordinate are output. 5.The method of claim 4, wherein the method further comprises: The multi-dimensional quality detection in S3 specifically includes: S31, a corresponding face region image is cropped from the image I based on the face positioning box; S32, the width and height of the cropped face image and the width and height proportions of the original image are calculated; if the width and height proportions are both within a preset first interval and the width-to-height ratio is within a second interval, it is determined that the head portrait size is compliant; S33, the cropped face image is input into a pre-trained CNN network to obtain left eye and right eye closed probability values; if the closed probability values of the two eyes are both less than a set threshold, it is determined that the eyes are open, i.e., compliant; S34, the face region feature point coordinates are input into a pre-trained 3D key point regression network to predict the yaw angle, pitch angle, and roll angle of the head; if each angle is within a corresponding preset compliant range, it is determined that the head posture is compliant; S35, if the detection results of S32, S33, and S34 are all compliant, the quality detection is passed. 6.The method of claim 5, wherein the method further comprises: The steps of S4 include: S41, the face image is divided into grid blocks with a set number of pixels, the gray mean value of each grid block is calculated, the standard deviation of the mean values of all grid blocks is calculated, if the standard deviation is greater than a first threshold, it is determined that the light is uneven, and a Retinex algorithm is used for illumination compensation; S42, the whole face region gray mean value μ is calculated, if μ is less than the lower limit value of the preset range, gamma correction is used to enhance brightness; if μ is greater than the upper limit value of the preset range, gamma correction is used to reduce brightness; S43, the Laplacian gradient variance of the face image is calculated, if the gradient variance is less than a second threshold, it is determined that the image is blurred, and a non-local mean denoising algorithm or a blind deblurring algorithm is used for clear processing; S44, the enhanced face image is obtained after the processing of S41-S43. 7.The method of claim 6, wherein the method further comprises: The steps of S5 include: S51, five feature point coordinates of a pre-defined standard face template are obtained to obtain standard coordinates P_std; S52, five corresponding feature point coordinates in the face image obtained by S2 are read to obtain input coordinates P_input; S53, an affine transformation matrix M is calculated according to the standard coordinates and the input coordinates; S54, using the affine transformation matrix M, the geometric transformation is carried out on each pixel in the face image after quality enhancement, and the bilinear interpolation is used to calculate the pixel value of the transformed image, and finally the standard size face image after alignment is obtained. 8.The method of claim 7, wherein the method further comprises: In S6, the improvement of MobileNetV3 network includes: Using h-swish activation function instead of original ReLU activation function; Introducing SE attention mechanism in bottleneck module; Adding depth separable convolution layer after the last three convolution layers; Using global depth separable convolution layer instead of global average pooling layer. 9.The method of claim 8, wherein, The face feature library in S7 is constructed by the following methods: a) The data source is the standard face photo submitted by the examinee when registering; b) The standard face photo is processed through the steps of S1 to S6 in turn, and the corresponding face feature vector is extracted, the similarity between the face feature vector and the existing face feature vector in the face feature library is calculated, if the similarity with any existing face feature vector exceeds the preset threshold, it is judged as repeated or low quality data and filtered, not stored in the library; Otherwise, execute c); c) The extracted face feature vector is stored in the face feature library after being associated with the examinee ID and the storage timestamp, and an index is established for all face feature vectors in the face feature library.

10. A mobile terminal face recognition and photo quality detection system based on deep learning, characterized in that, It includes: An image acquisition and preprocessing module for real-time acquisition of video images of examinees entering the examination room, and Gaussian filter smoothing and histogram equalization processing of the acquired images in turn to suppress noise and obtain preprocessed face image I containing face image; A face detection module including an improved RetinaFace network for receiving the image I and performing face detection, if the detection is successful, the face positioning frame and the face region feature point coordinates including left and right eye, nose, left and right mouth corner five feature point coordinates are output; if the detection fails, the image acquisition and preprocessing module is triggered to work again; An image quality detection module for cropping a face image from the image I based on the face positioning frame, and performing multi-dimensional quality detection on the cropped face image; the multi-dimensional quality detection at least includes face size compliance detection, closed eye detection and head posture detection; if any detection fails, the image acquisition and preprocessing module is triggered to work again; An image enhancement module for performing image quality enhancement processing on the face image after quality detection to obtain an enhanced face image; A face alignment module for aligning the enhanced face image to a standard template based on the face region feature point coordinates to obtain an aligned face image; A feature extraction module including an improved MobileNetV3 network for performing deep feature extraction on the aligned face image to obtain a face feature vector; A feature comparison module for calculating the cosine similarity between the face feature vector and the registered feature vector in the face feature library; An identity authentication module for determining that the two persons are the same when the cosine similarity is greater than or equal to the set threshold, otherwise determining that the two persons are different and outputting prompt information.

Images