Low-quality face image recognition method

Abstract

The invention provides a low-quality face image recognition method, which can be used for effectively recognizing shielded face images according to the characteristics of shielded and unshielded face images with different proportions, and specifically comprises the following steps: extracting the characteristics of the unshielded face images, and according to different shielding proportion values of faces and the characteristics of the unshielded face images, extracting the characteristics of the unshielded face images; different occlusion erasing networks are trained, so that the product of the occlusion erasing networks and the characteristics of the non-occlusion face image is basically consistent with the characteristics of the occlusion face image. When the target face is recognized, if the target face is an unshielded face image, the target face is directly compared with the original face image features for recognition; and if the face image is a shielded face image, according to the face shielding proportion, obtaining a corresponding shielding erasing network, calculating a product value of the shielding erasing network and the non-shielded face image features, and comparing the product value with the feature value extracted after the special processing of the shielded face for recognition, thereby improving the recognition accuracy of the face under the shielding condition.

Description

Technical Field

[0001] The invention relates to the field of computer vision, and in particular to a low-quality face image recognition method. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Face recognition technology plays an important role in many fields such as public security, commerce, and medicine. However, in practical applications, due to various uncontrollable factors, including changes in lighting, posture, occlusions, and low device resolution, the quality of the collected face images is often low. In particular, the face may be blocked by hair, glasses, masks, hats, scarves, etc. These occlusions will block some facial features, resulting in incomplete collected image information, thus affecting the accuracy and reliability of face recognition.

[0004] At present, the main idea of ​​occluded face recognition is to first detect the occluded object, detect the occluded area, and make the features of this area participate in the subsequent recognition process as little as possible or not at all. The main research methods include principal component analysis (PCA), local feature method (such as local binary pattern LBP), etc. These methods judge the occluded object area by comparing the absolute error in the subspace of the occluded face and the non-occluded face, or extract local texture features for recognition. However, these traditional methods cannot effectively improve the recognition effect under occlusion. The position, size, shape and type of the occlusion may be different, which increases the difficulty of recognition. How to effectively extract key features for matching when some features are occluded is a major challenge. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a low-quality face image recognition method in view of the shortcomings of the prior art.

[0006] A low-quality face image recognition method, the method comprising:

[0007] Step 1: Face detection, obtain face location information, and obtain the position coordinate information of each face in the image through the face detection model.

[0008] Step 2: Ratio calculation: Based on the face position information obtained in step 1, perform face key point occlusion detection and obtain the face occlusion area, and calculate the face image occlusion ratio based on the occlusion area.

[0009] Step 3: Preprocess the occluded image and extract the occluded face features.

[0010] Step 4: Train the occlusion erasure network, based on the face occlusion ratio obtained in step 2, the occluded face features obtained in step 3, and the feature data of the unoccluded face image. The product of the output of the occlusion erasure network and the features of the unoccluded face image has a large similarity with the features of the occluded face image.

[0011] Step 5: Use the occlusion erasure network obtained in steps 1-4 to recognize the face image. When recognizing the target face, if it is an unobstructed face image, it is directly compared with the original face image features; if it is an occluded face image, the corresponding occlusion erasure network is obtained according to the face occlusion ratio, and the product of the occlusion erasure network output and the unobstructed face image features is calculated, and compared with the feature value extracted after special processing of the occluded face for recognition.

[0012] Furthermore, the face detection in step 1 includes:

[0013] Step 1-1: Detect faces in the image based on the face detection model retinaface and obtain the corresponding positions of all faces in the image.

[0014] Step 1-2: Get the coordinates of the face circumscribed rectangle in the image and the location information of the face key points.

[0015] Furthermore, the facial key point occlusion detection in step 2 includes:

[0016] Step 2-1: Construct a 2D heat map for each facial key point and train the facial key point occlusion detection model.

[0017] Step 2-2: Use the facial key point occlusion detection model to detect each key point to obtain occlusion information of each key point, where the occlusion information includes the key point position and confidence level.

[0018] Step 2-3: Determine whether the key points of the face are blocked based on the confidence threshold.

[0019] Furthermore, the image preprocessing in step 3 includes:

[0020] Step 3-1: Determine whether the eye area of ​​the face image is blocked.

[0021] Step 3-2: If the eye area is within the occluded area, the RGB value of the occluded eye area color is set to (120, 120, 120); the color of the non-eye occluded area is set to the average of the colors of other non-occluded areas, that is, the RGB value is set to avg(sum(R,G,B)), where sum() is the calculated sum and avg() is the calculated average.

[0022] Step 3-3: If the eye area is not in the occluded area, the color of the occluded area is set to the average of the colors of other non-occluded areas, that is, the RGB value is set to avg(sum(R,G,B)), where sum() is the calculated sum and avg() is the calculated average.

[0023] Furthermore, the training of the occlusion erasure network in step 4 includes, according to the unobstructed face feature data, the product of the occluded face ratio and the occlusion erasure network, and the occluded face feature data, using arcface-loss to perform model training, and obtaining the occlusion erasure network parameters, wherein the unobstructed and occluded face feature extraction networks are implemented using the arcface network model, and two fully connected layers using PReLU as the activation function are constructed as the occlusion erasure network, the product of the occlusion ratio, the unobstructed face feature and the occlusion erasure network, and the occluded face feature are used to calculate the loss, and the erasure network parameters are updated by back propagation.

[0024] Furthermore, the unobstructed facial feature data described in step 4 is extracted from multiple different unobstructed facial images of the same person manually selected through the arcface network model, and then stored in the face library; the occluded face image obtained by multiplying the occluded face ratio and the occlusion erasure network includes different occlusion ratios.

[0025] Furthermore, the facial image recognition in step 5 specifically includes:

[0026] Step 5-1: Obtain face occlusion erasure features based on the occlusion erasure network and the occlusion ratio.

[0027] Step 5-2: Calculate the similarity between the face erasure feature data and the face occlusion feature data.

[0028] Step 5-3: The facial information corresponding to the facial feature data greater than the similarity threshold is taken as the result of the current face recognition.

[0029] Furthermore, the facial key points in step 2 include 68.

[0030] Furthermore, the occlusion ratio in step 2 includes a horizontal occlusion ratio and a vertical occlusion ratio, and the occlusion ratio is the ratio of the occlusion area to the overall size of the face. Horizontal occlusion here refers to occlusions such as masks and sunglasses; vertical occlusion here refers to occlusions caused by the shooting angle of the face and strong side light, where only half of the face is seen. The face occlusion ratio of the data used to train the occlusion erasure network in step 2 is not greater than 0.7.

[0031] Furthermore, in step 1, if the target face is recognized as an unobstructed face image, it is directly compared and recognized with the features of the unobstructed face images in the face library; if it is an occluded face image, the occluded area of ​​the face is specially processed, and the product value of the occlusion erasure network and the unobstructed face image features is calculated according to the face occlusion ratio and the corresponding occlusion erasure network, and compared and recognized with the feature value extracted after the special processing of the occluded face.

[0032] The present invention aims at the problem of face occlusion, trains a face key point occlusion detection model, obtains the face occlusion ratio, and jointly trains a face occlusion erasure network based on the face occlusion ratio and the feature data of the unobstructed face. For occluded face recognition, a face occlusion erasure network is introduced to improve the recognition effect of occluded face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0034] Figure 1 The figure is the overall flow chart of the low-quality face image recognition method.

[0035] Figure 2 Flowchart for training the face occlusion erasure network.

[0036] Figure 3 Flowchart of data erasure and network training for face occlusion.

[0037] Figure 4 This is the distribution map of 68 key points of the face used in this method.

[0038] Figure 5 This is a diagram showing the increase in the occlusion ratio of the face from top to bottom according to this method.

[0039] Figure 6 This is the figure showing the increase in face occlusion ratio from bottom to top according to this method.

[0040] Figure 7 This is the unobstructed facial feature map extracted by person A using this method.

[0041] Figure 8 This is the facial feature map after occlusion processing extracted by person A in this method.

[0042] Fig. 9 This is the product graph of the unobstructed face features extracted by person A in this method and the occlusion erasure network.

[0043] Fig.10 This is a diagram of the specific training process of the face occlusion erasure network.

[0044] Fig.11 Figure 2. Network structure diagram for face erasing. DETAILED DESCRIPTION

[0045] The present invention proposes a low-quality face image recognition method. First, face key point occlusion detection is performed on the face image obtained by face detection to obtain the face occlusion area and face occlusion ratio; secondly, according to the face occlusion ratio, an occlusion erasure network is trained to obtain occluded face features, which are compared with the unoccluded face features processed by the erasure network, thereby improving the recognition effect of occluded faces.

[0046] The overall process of the present invention is as follows Figure 1 As shown, it includes processes such as face detection, face key point occlusion detection, face occlusion ratio calculation, obtaining occluded face erasure network, face occlusion erasure feature calculation, and face feature comparison; the face detection module refers to the use of the face detection model retinaface to detect faces in the image, find out the corresponding positions of all faces in the image, and the output of the algorithm is the coordinates of the face circumscribed rectangle in the image, as well as the position information of the face key points.

[0047] The facial key point occlusion detection refers to using a facial key point detection model to detect whether there is occlusion on the 68 key points of the face; the model input is the face area after face detection, where the 68 key points of the face are distributed as follows Figure 4 68 key points are currently the most common face key point scheme, which is adopted by the Dlib algorithm in OpenCV. The face key points are divided into internal key points and contour key points. The internal key points include eyebrows, eyes, nose, and mouth, totaling 51 key points, and the contour key points include 17 key points.

[0048] The facial key point detection model needs to construct a 2D heat map for each key point as training data. The value in the heat map represents the probability of each key point's position in the image. A deep convolutional neural network is used as the facial key point detection model for model training, and then each key point is detected, and the occlusion information of each key point is output, including the key point position and confidence. According to the confidence threshold, the information on whether the key point is occluded is obtained.

[0049] According to the occlusion detection results of facial key points, the location information of the occlusion area is obtained; the ratio of the area of ​​the face occlusion area to the area of ​​the entire face is calculated to obtain the face occlusion ratio.

[0050] The occluded area is specially processed, where: if the eyes are occluded, the color of the occluded area is processed to gray, and the RGB value is set to (120,120,120); if the non-eyes are occluded, the color of the occluded area is processed to the average of the colors of other non-occluded areas, that is, the RGB setting value is avg(sum(R,G,B)).

[0051] The facial features extracted after occlusion processing and the unoccluded facial features are used to train the corresponding occlusion erasure network according to the facial occlusion ratio value, the facial features extracted after occlusion processing and the unoccluded facial features, so that the product of the occlusion erasure network and the unoccluded facial image features has a relatively large similarity with the features of the occluded facial image. Figure 7 As shown in the figure, the unobstructed facial image features of person A are extracted. Figure 8 The facial features of person A after occlusion processing are extracted. Fig. 9 The extracted features of the unobstructed face image of person A are multiplied by the occlusion erasure network. It can be seen that there are many dissimilar features between the unobstructed face image features and the face features after occlusion processing, and the similarity value calculated by cosine similarity is 0.45; the features after the product of the unobstructed face image features and the occlusion erasure network are more similar to the face features after occlusion processing, and the similarity value calculated by cosine similarity is 0.92.

[0052] In the specific implementation, when training the face erasure network, the face occlusion ratio can be calculated in advance to form a face occlusion ratio label file for each data to accelerate the model training process. The case where the face occlusion ratio is greater than 0.7 will not be processed.

[0053] In the specific implementation, the face occlusion ratio is statistically processed according to two situations: in the first situation, the occlusion ratio gradually increases from top to bottom, that is, from 0 to 1; Figure 5 As shown in the figure, the occlusion ratio gradually increases from top to bottom, from 0.3 to 0.6, to 0.9. The second situation is in the opposite direction, the occlusion ratio gradually increases from bottom to top, also from 0 to 1; Figure 6 As shown in the figure, the occlusion ratio increases from bottom to top, from 0.3 to 0.6, to 0.9. Statistical processing is performed during the training of the occlusion erasure network and subsequent recognition. It is mainly used to count the face occlusion ratio and process the occlusion area after face occlusion detection; different occlusion erasure networks are trained according to the occlusion position.

[0054] Based on the face recognition method provided by the above example of the present invention, by extracting the facial feature data and the face occlusion ratio in the current image, and obtaining the corresponding erased data based on the face occlusion ratio, the feature data is erased based on the current erased data to generate corresponding occluded feature data; the extracted occluded face feature data is compared with the facial feature data after erasure, so as to improve the accuracy of occluded face recognition.

[0055] Figure 2 FIG. 1 is a flowchart of a network training for erasing occluded facial features according to an embodiment of the present invention. Figure 2As shown, the method of this embodiment includes: extracting unobstructed face feature data; calculating the occlusion ratio of the occluded face and performing special processing on the occluded area; extracting the occluded face feature data after special processing; performing comparative learning on the occluded face feature data based on the product of the unobstructed face feature data, the occluded face ratio and the occlusion erasure network, using arcface-loss for model training, and obtaining the occlusion erasure network parameters. Among them, the unobstructed and occluded face feature extraction networks are implemented using the arcface network model, and the occlusion erasure network is two fully connected layers that use PReLU as the activation function, and its network structure is as follows: Fig.11 As shown in the figure, the network structure with two layers of nonlinear mapping can better map and process the feature parameters.

[0056] Figure 3 is another flow chart of the face recognition method according to an embodiment of the present invention. Figure 2 When operating the embodiment, Figure 3 As shown, the method of this embodiment also includes: obtaining multiple sample images respectively labeled with face categories. In a specific implementation, in order to ensure the effect of the overall erasing network, the occluded face image should cover different occlusion ratios and include multiple different unobstructed face images of the same person. In this example, data from the Webface-OCC public data set is used, and face images with consistent length and width are obtained through face detection and face alignment. By using the random gradient descent method to train the parameters of the erasing network model, the neural network can learn the mapping from the unobstructed face image feature space to the occluded face feature space through training of sample images.

[0057] The specific training process is as follows Fig.10 As shown in the figure, the overall input includes unobstructed and occluded face images of the same person. The facial features are directly extracted from the unobstructed face images; the facial key point occlusion detection is performed on the occluded face images to obtain the occlusion ratio and occlusion area. After processing the occlusion area, the facial features are extracted using the same model as the unobstructed face. The product of the occlusion ratio, unobstructed face features and the occlusion erasure network is used as an output, and the loss is calculated with the occluded face features, and the erasure network parameters are updated through back propagation.

[0058] After the model training is completed, it mainly plays a role in the face occlusion recognition stage. When establishing a face database, you only need to extract unobstructed faces into the database, and occluded faces do not need to be stored in the database. When identifying the target face, for unobstructed faces, directly extract features and compare and identify with the features in the database; for occluded faces, perform occlusion detection on facial key points, obtain the face occlusion ratio, perform special processing on the occluded area, and extract face occlusion features; obtain face occlusion erasure features based on the occlusion ratio and occlusion erasure network, and calculate the similarity between face erasure features and face occlusion feature data; take the face information corresponding to the face feature data greater than the similarity threshold as the result of the current face recognition.

[0059] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the set of network platform abnormal head portrait intelligent monitoring method and system based on face recognition and the system provided by the present invention and some or all steps in each embodiment. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0060] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0061] The present invention provides a method and idea for low-quality face recognition. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A low-quality face image recognition method, characterized in that: Methods include: Step 1: Face detection, obtaining face location information; Step 2: Ratio calculation: Based on the face position information obtained in step 1, perform face key point occlusion detection and obtain the face occlusion area, and calculate the face image occlusion ratio based on the occlusion area; Step 3: preprocess the occluded image and extract the occluded face features; Step 4: training the occlusion erasure network, according to the face occlusion ratio obtained in step 2, the occluded face features obtained in step 3, and the unoccluded face image feature data, training the occlusion erasure network; Step 5: Use the occlusion erasure network obtained in steps 1-4 to recognize the face image.

2. The low-quality face image recognition method according to claim 1, characterized in that: Step 1 of face detection includes: Step 1-1: Detect faces in the image using the face detection model retinaface to obtain the positions of all faces in the image; Step 1-2: Get the coordinates of the face circumscribed rectangle in the image and the location information of the face key points.

3. The low-quality face image recognition method according to claim 1, characterized in that: The facial key point occlusion detection in step 2 includes: Step 2-1: Construct a 2D heat map for each facial key point and train the facial key point occlusion detection model; Step 2-2: using a facial key point occlusion detection model to detect each key point, and obtaining occlusion information of each key point, wherein the occlusion information includes a key point position and a confidence level; Step 2-3: Determine whether the key points of the face are blocked based on the confidence threshold.

4. The low-quality face image recognition method according to claim 1, characterized in that: The image preprocessing in step 3 includes: Step 3-1: Obtain the face occlusion area based on the occlusion position information of the face key points; Step 3-2: If the eye area is within the occluded area, the RGB value of the occluded eye area color is set to (120, 120, 120); the color of the non-eye occluded area is set to the average of the colors of other non-occluded areas, that is, the RGB value is set to avg(sum(R,G,B)), where sum() is the calculated sum and avg() is the calculated average. Step 3-3: If the eye area is not in the occluded area, the color of the occluded area is set to the average of the colors of other non-occluded areas, that is, the RGB value is set to avg(sum(R,G,B)), where sum() is the calculated sum and avg() is the calculated average.

5. The low-quality face image recognition method according to claim 1, characterized in that: The training of the occlusion erasure network in step 4 includes: According to the unoccluded face feature data, the product of the occluded face ratio and the occlusion erasure network, and the occluded face feature data, arcface-loss is used for model training to obtain the occlusion erasure network parameters. The unoccluded and occluded face feature extraction networks are implemented using the arcface network model, and two fully connected layers using PReLU as the activation function are constructed as the occlusion erasure network. The product of the occlusion ratio, the unoccluded face feature and the occlusion erasure network is used to calculate the loss with the occluded face feature, and the erasure network parameters are updated by backpropagation.

6. The low-quality face image recognition method according to claim 5, characterized in that: The unobstructed face feature data described in step 4 is extracted by using the arcface network model from multiple different unobstructed face images of the same person selected manually, and then stored in the face library; the occluded face image obtained by multiplying the occluded face ratio and the occlusion erasure network includes different occlusion ratios.

7. The low-quality face image recognition method according to claim 1, characterized in that: The facial image recognition described in step 5 specifically includes: Step 5-1: Obtain face occlusion erasure features according to the occlusion erasure network and the occlusion ratio; Step 5-2: Calculate the similarity between the face erasure feature and the face occlusion feature data; Step 5-3: The facial information corresponding to the facial feature data greater than the similarity threshold is taken as the result of the current face recognition.

8. The low-quality face image recognition method according to claim 1, characterized in that: The key points described in step 2 include 68.

9. The low-quality face image recognition method according to claim 1, characterized in that: The occlusion ratio in step 2 includes a horizontal occlusion ratio and a vertical occlusion ratio, and the occlusion ratio is the ratio of the occlusion area to the overall size of the face.

10. The low-quality face image recognition method according to claim 9, characterized in that: The face occlusion ratio of the data used to train the occlusion erasure network in step 2 is no more than 0.7.

Images