An occluded face recognition method based on occlusion-feature mapping relationship

Abstract

The application provides an occlusion face recognition method based on an occlusion-feature mapping relationship, and the method comprises the following steps: establishing a face analysis and face key point detection module, obtaining a face semantic region division picture through the face analysis module; performing sub-region division on the face semantic region through the face key point detection module; inputting a pair of original face images and sub-region occlusion face images into a differential twin network to learn the mapping relationship between the occlusion and the deep feature, thereby establishing a mask dictionary of different sub-regions of the face; segmenting a binary image of the occlusion region through a segmentation network FCN-8s; merging the corresponding occlusion sub-region mask dictionary through the binary image query; multiplying the face picture to be recognized with the corresponding mask dictionary in the face recognition network to eliminate the damaged feature elements caused by the occlusion, so that the recognition rate of the occlusion face is improved; and the feasibility of the method is verified through experiments on a public data set.

Description

Technical Field

[0001] This invention relates to image processing technology, and in particular to an occluded face recognition method based on occlusion-feature mapping relationship. Background Technology

[0002] In recent years, due to the development of deep learning, research on facial recognition technology has made new progress. Facial recognition is a method of identity verification by extracting and comparing facial visual feature information. Faces possess uniqueness, consistency, and a high degree of non-replicability. Compared with fingerprint and iris recognition technologies, they are reliable, easy to operate, and contactless, providing stable conditions for identity verification.

[0003] However, unconstrained face recognition still faces many challenges, such as large pose changes, lighting variations, different facial expressions, and occlusion. Face occlusion is a particularly challenging problem because the lack of information about the occluded portion disrupts the facial structure, increases intra-class distances, and decreases inter-class distances, thus affecting recognition accuracy to some extent. How to eliminate the feature elements damaged by occlusion deserves our attention.

[0004] Currently, there are two main approaches to dealing with occluded faces: image inpainting and feature removal. Image inpainting involves pixel-level generative restoration of the occluded area to compensate for missing information. For this type of method, preserving identity information while restoring the occluded area remains a significant challenge. Feature removal methods, based on the visual experience that the human eye typically ignores occluded areas, primarily focus on recognizing the unoccluded regions. In deep learning methods, a common approach is to use attention or similar mechanisms to suppress the features corresponding to the occluded areas. The key to this strategy lies in accurately identifying the features corresponding to the occluded areas. The first attempt to establish a mapping between facial regions and deep features was made by Song Lingxue et al. They divided the face into 3×3 blocks and learned the correspondence between each occluded region and damaged feature elements by using the feature differences between occluded and clean faces. Then, they input the difference into a mask generator to obtain a binary mask corresponding to each occluded region, constructing a mask dictionary. While the interpretability of facial depth features remains an unsolved problem, we believe that facial features represent the structural and facial features of the face in a complex way, thus a more precise mapping relationship should exist between facial features and these features. Song Lingxue et al. confirmed that a mapping relationship does exist between partial occlusion and depth features, but they simply divided the face image into several rectangular regions. This division method may assign the same facial feature to different regions or different facial features to the same region. In reality, any object covering or in front of a face can be defined as an occlusion. Therefore, the type, shape, size, and location of occlusions cannot be fully predicted or completely covered by a database. Generating a realistically occluded face dataset is labor-intensive, inaccurate in coverage, and lacks controllability. Therefore, this solution redesigns the network framework, focusing more on semantic consistency in face sub-region division and feature selection, and refining the mask dictionary. Summary of the Invention

[0005] The technical problem this invention aims to solve is to address the current state of existing technologies by proposing an occlusion-feature mapping-based face recognition method. First, a face parsing network is used to divide the face image into different semantic regions, including facial features such as eyes, nose, mouth, eyebrows, skin, hair, and background. Then, facial key points are combined to further subdivide each semantic region. This approach refines the mask dictionary and makes the network focus more on the semantic information corresponding to features when learning the mask dictionary.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] An occlusion-feature mapping method for face recognition includes the following steps:

[0008] Step S1: Pre-train the ResNet50 model using the face training dataset;

[0009] Step S2: Establish a face parsing module and a face key point detection module. Input the original face images in the face training dataset into the face parsing module to obtain the corresponding face semantic region segmentation dataset.

[0010] Step S3: Divide the face semantic region segmentation dataset from Step S2 into sub-regions using the face key point detection module to generate a face dataset with sub-region occlusion.

[0011] Step S4: Use the ResNet50 model trained in step S1 as the pre-trained model of the differential Siamese network. Input the original face images of the face training dataset and the occluded face images of the sub-region occluded face dataset into the differential Siamese network in pairs. Utilize the difference between the last convolutional features of the original face images and the occluded face images to learn the mapping relationship between occlusion and features.

[0012] Step S5: Based on the mapping relationship in step S4, obtain the corresponding mask from the mask generator in the trained differential Siamese network and construct the mask dictionary for the occluded sub-regions.

[0013] Step S6: The face image to be identified is segmented into occluded and unoccluded regions using the occlusion segmentation network FCN-8s model, and the occlusion segmentation map is divided into several sub-regions according to the segmentation method in step S3; based on the overlap between the occluded region and the several sub-regions, the associated items are obtained from the mask dictionary and merged to obtain the merged mask dictionary;

[0014] Step S7: The face recognition network uses the ResNet50 model trained in step S1 as a pre-trained model to extract the top-level convolutional features of the occluded face to be recognized and multiply them with the merged mask dictionary in step S6 to eliminate the influence of occlusion on the features and improve the recognition rate of occluded faces.

[0015] A further improvement to the technical solution of the present invention is that, after step S7, 1:1 face verification and 1:N face recognition are performed on the public datasets LFW and MegaFace respectively, and the feasibility and effectiveness of the method are tested by comparing face similarity and face recognition accuracy.

[0016] A further improvement to the technical solution of the present invention is that the specific steps of step S2 are as follows:

[0017] Step S21: First, the face is semantically parsed using the BiSeNetV2 real-time face parsing module. Since the distribution of facial features on the face has a strong structural nature, it can be used as prior information. The eyes, nose, mouth, face, and eyebrows of the face are classified and supervised in blocks to achieve accurate perception of the semantic information of the skin and facial features of the face image.

[0018] A further improvement to the technical solution of this invention lies in that step S3 includes locating facial key points using a facial key point detection module, generating 98 key points in the facial region using a facial key point detection network (PFLD), and dividing the facial features into sub-regions by arranging the points into irregular shapes. The number of sub-regions is determined based on the importance of the facial features to recognition. Since the eyes contain important recognition information, the number of sub-regions is relatively large, while the facial skin provides less semantic information, resulting in a reduction in the number of facial regions but an increase in the area of ​​the sub-regions.

[0019] A further improvement to the technical solution of this invention is that step S3 further includes: dividing the facial features and skin into sub-regions by connecting lines using key points captured by the facial key point detection module, and occluding and filling different sub-regions to obtain a batch of simulated occluded face datasets that lose different semantic information. The face is re-divided according to the face parsing and key point detection module, changing from a number of regular squares to 26 sub-regions that better fit the facial structure and semantic information. The mask dictionary region no longer covers the entire image, but is reduced to the facial region that affects recognition.

[0020] The technological advancements achieved by this invention due to the adoption of the above technical solutions are as follows:

[0021] Since any object covering or in front of a face can be considered an occluder, its category cannot be completely covered by the database, resulting in limited generalization ability of traditional occlusion perception networks. The improvement lies in our view that occlusion essentially damages the semantic information of the occluded region. We perform semantic information mining on the face, analyze the visible semantic information within the face, simulate the lost semantic information regions, and achieve occlusion perception independent of the occluder category, thus improving occlusion robustness and face recognition performance. Song Lingxue's research on the correspondence between face segmentation and convolutional features at Tsinghua University has demonstrated a certain mapping relationship between face regions and feature channels. However, her research only divides the face region into equal rectangular areas. Since the uncertainty of the occluded region size affects the final feature having an inconsistent number of effective elements, traditional metrics based on fixed vector design are not entirely applicable and cannot be effectively evaluated. Furthermore, it does not consider the semantic information provided by different facial structures. Through face parsing and keypoint detection modules, we can not only accurately perceive the semantic information of the facial region but also finely segment irregular facial sub-regions. This invention places greater emphasis on the semantic nature of face sub-region segmentation and feature selection, and the mask dictionary only covers the face region, making the segmentation more reasonable and detailed, thereby maximizing the number of effective elements and the accuracy of removing occluded features. Attached Figure Description

[0022] Figure 1 This is a detailed flowchart of the present invention;

[0023] Figure 2 This is a schematic diagram of the structure guiding the semantic information perception of face parsing in this invention;

[0024] Figure 3 This is a schematic diagram of the structure guiding the semantic information perception of facial key point detection in this invention;

[0025] Figure 4 This invention describes the method of segmenting the facial region of a person.

[0026] Figure 5 Examples of feature dictionaries for the two occlusion types of this invention (selecting 5 channels). Detailed Implementation

[0027] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0028] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0029] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] An occlusion-feature mapping method for face recognition, the specific process of which is as follows: Figure 1 As shown, it includes the following steps:

[0031] Step S1: After detecting 5 facial keypoints using the MTCNN face detection model on the CASIA-Webface training dataset and the LFW face verification dataset, face alignment is performed using a similarity transformation to obtain aligned facial images, which are then adjusted to 112*112 pixels. The aligned and cropped CASIA-Webface face dataset is then input into the ResNet50 model as our backbone CNN model for pre-training and validation using the LFW face verification dataset.

[0032] Step S2: The aligned and cropped CASIA-Webface face dataset is processed through the BiSeNetV2 network of the face parsing module to segment the facial features and skin regions. This establishes a semantic information relationship between the parsed labels and the corresponding positions of the face regions, resulting in the corresponding face semantic region segmentation dataset. Figure 2 As shown.

[0033] Step S3: Using the PFLD face landmark detection module, accurately predict 98 facial landmarks from the face semantic region segmentation dataset from Step S2. Figure 3 As shown. Based on the proposed face segmentation method, the facial features and facial skin areas are divided into sub-regions, such as... Figure 4 As shown. Then, the pixels in different sub-regions are set to 0 to artificially occlude and cover or delete semantic information. This results in a batch of simulated sub-region occlusion datasets with lost semantic information.

[0034] Step S4: Input the paired original face images and sub-region occluded face images into the Siamese differential network (SCD). The SCD extracts features from the face image pairs. The model parameters are updated using classification loss and contrastive loss. The classification loss function improves the recognition rate of the pre-trained model, ensuring that the new features resulting from multiplying the top-level convolutional features of the occluded face with the mask can be correctly classified. The contrastive loss function aims to minimize the difference between the new features of the occluded face and the corresponding convolutional features of the original face. Optimizing both loss functions together encourages the mask generation module to assign low mask values ​​to feature elements that differ significantly between the convolutional features of the occluded face and the original face and affect recognition. These elements are the occluded and damaged elements that this scheme focuses on. The paired SCD model is then updated based on the determined mask generation model parameters.

[0035] A mask dictionary is built using the differential Siamese network trained in step S4. The mask generator consists of a simple Conv-PReLU-BN structure, which calculates the absolute value of the feature differences between the top convolutional layers of the Siamese networks. As input, a mask with the same size as the top-level convolutional features is output by the mask generation module, and the output values ​​are mapped to the range [0,1] by the Sigmoid activation function. Each element in this mask is multiplied by the corresponding element of the original top-level convolutional features to obtain new convolutional features. Therefore, the loss function is constructed as follows:

[0036]

[0037] in:

[0038] l cls Represents classification loss, l diff Indicates comparative loss;

[0039] θ represents the vector space formed by the current network weights;

[0040] This represents the output of the mask generator;

[0041] x i Let i represent the original, unobstructed face in the i-th pair of face images;

[0042] This represents the face whose j-th sub-region is occluded in the i-th pair of face images;

[0043] f(·) represents the output of the convolutional layer of the convolutional neural network;

[0044] This represents the new feature resulting from multiplying the top-level convolutional features that obscure the face with the mask;

[0045] The convolutional features represent the original, unobstructed human face;

[0046] F represents a fully connected layer or an average pooling layer following the top convolutional layer of the backbone network;

[0047] γ represents the penalty term coefficient, which is a parameter that is manually set to be greater than 0;

[0048] This represents the probability that the backbone network correctly classifies the data.

[0049] The face occlusion image is input into a pre-trained differential Siamese network model, and a mask generation module is used to obtain a series of mask sets {m1, m2, ... m} for different occlusion sub-regions. n The mask set is then normalized, and the mean of the normalized mask is calculated to obtain the average mask of the occluded sub-region. To further refine the removal of occlusion features, the average mask is binarized to obtain a binary mask dictionary.

[0050] In this invention, the face is divided into 26 non-overlapping sub-regions based on semantic information. A total of 26 mask generation modules need to be trained to establish a corresponding occlusion sub-region dictionary. When a sub-region of the face image output by the mask generation module is occluded, the element in the top-level convolutional feature of the face image that is occluded and thus should have its response value reduced. The reduced element corresponds to a lower value in the output of the mask generation module. The index of the binary mask dictionary is the face sub-region bj, and the index entry is a binary mask Mj. The mask Mj has the same size as the top-level convolutional feature of the backbone network. A value of 0 in Mj represents a feature element that should be removed from the recognition when the face sub-region bj is occluded.

[0051] Step S5: Segment the occluded and unoccluded regions using the FCN-8s occlusion segmentation network model. Calculate the overlap area between the occluded region and each sub-region of the face. When the IOU is greater than 0.6, select the corresponding region dictionary to synthesize a binary mask for the face to be recognized. Figure 4 Taking the face image shown as an example, when the face image is divided into 26 blocks, the occluded face regions are 1, 2, and 3. According to the occlusion sub-region mask dictionary established in step S4, the binary mask dictionary of the face to be identified is: M = M1 & M2 & M3. Figure 5 Five channels of the dictionary-synthesized binary mask are shown under both synthetic occlusion and scarf occlusion. The binary mask M is multiplied by the features of the image to be recognized to obtain the corresponding features to be recognized in the image.

[0052] To verify the feasibility and effectiveness of this invention, the test face dataset was first cropped to a resolution of 112*112 using MTCNN for face alignment, and then randomly occluded using a face parsing or keypoint detection module. A 1:1 face verification experiment was then conducted on the LFW dataset. Face recognition experiments were performed using MefaFace Challenge 1. Two baseline models were compared during verification: baseline 1 was a CNN model trained with CASIA-Webface, and baseline 2 was a fine-tuned version of baseline 1 using a randomly occluded dataset.

[0053] The invention can accurately perceive the semantic information of the face region, mark different semantic information categories, and perform detailed division of the face region, thereby improving the robustness of occlusion and the recognition accuracy of occluded face recognition.

[0054] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0055] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. An occlusion-feature mapping method for face recognition, comprising: Step S1: Pre-train the ResNet50 model using the face training dataset; Step S2: Establish a face parsing module and a face landmark detection module. Input the original face images from the face training dataset into the face parsing module to obtain the corresponding face semantic region segmentation dataset; Step S2 includes: First, the face is semantically analyzed through the face analysis module. As prior information, the eyes, nose, mouth, face, eyebrows and other facial features are classified and supervised in blocks to achieve accurate perception of the semantic information of the skin and facial features in the non-occluded areas. Step S3: The facial semantic region segmentation dataset from step S2 is divided into sub-regions using the facial landmark detection module to generate a facial dataset with sub-region occlusion; Step S3 includes: For facial features that cannot be classified and supervised, the PFLD facial landmark detection module locates facial landmarks. This module generates several landmarks in the facial image, and the area enclosed by these landmarks is used to divide the image into sub-regions. Since the eyes and nose contain important recognition information, their sub-regions are more finely defined. Facial skin provides less semantic information, so the number of facial sub-regions is reduced, but the area is increased. Step S3 also includes: The clean face training dataset is segmented into facial features using the face parsing module; the facial landmark detection module uses key points to connect and divide the facial features and skin into regions, and then fills in the occlusion of different sub-regions to obtain a batch of simulated sub-region occlusion face datasets that lose different semantic information; the face is re-divided according to the face parsing and landmark detection modules, changing from equal rectangular blocks to several sub-regions that better fit the facial structure and semantic information, and the mask dictionary region no longer covers the entire image, but covers the facial regions that affect recognition; Step S4: Use the ResNet50 model trained in step S1 as the pre-trained model of the differential Siamese network. Input the original face images of the face training dataset and the occluded face images of the sub-region occluded face dataset into the differential Siamese network in pairs. Utilize the difference between the last convolutional features of the original face images and the occluded face images to learn the mapping relationship between occlusion and features. Step S5: Based on the mapping relationship in step S4, obtain the corresponding mask from the mask generator in the trained differential Siamese network and construct the mask dictionary for the occluded sub-regions. Step S6: The face image to be identified is segmented into occluded and unoccluded regions using the occlusion segmentation network FCN-8s model, and the occlusion segmentation map is divided into several sub-regions according to the segmentation method in step S3; based on the overlap between the occluded region and the several sub-regions, the associated items are obtained from the mask dictionary and merged to obtain the merged mask dictionary; Step S7: The face recognition network uses the ResNet50 model trained in step S1 as a pre-trained model to extract the top-level convolutional features of the occluded face to be recognized and multiply them with the merged mask dictionary in step S6 to eliminate the influence of occlusion on the features and improve the recognition rate of occluded faces.

2. The occlusion face recognition method based on occlusion-feature mapping relationship according to claim 1, characterized in that: Step S7 is followed by performing 1:1 face verification and 1:N face recognition on the public datasets LFW and MegaFace respectively, and testing the feasibility and effectiveness of the method through face similarity comparison and face recognition accuracy.

Images