A method and system for supporting facial recognition of people wearing masks

Abstract

This invention discloses a method and system for supporting facial recognition of people wearing masks. Based on a cross-modal, cross-domain recognition framework, it performs pose estimation and feature extraction on the face image to be recognized, and compares it with registered features in a registered 3D face database using multiple poses to obtain a recognition score and complete the face recognition. The cross-modal, cross-domain recognition framework includes a face image completion network to synthesize a 3D masked face image from the acquired 3D face model, a face image generation network to render and augment registered images in the registered 3D face database, and a feature extraction network to extract facial features. This application solves the problem of difficult dynamic face recognition in video surveillance. Unlike 2D-2D recognition, it reduces the impact of mask-wearing behavior on recognition features and improves the accuracy of facial recognition under mask conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for supporting facial recognition of people wearing masks. Background Technology

[0002] Facial recognition, as an important branch of biometric identification, has become a hot topic and highlight in the research and development of next-generation artificial intelligence. Benefiting from new algorithms, powerful computing capabilities, big data, and other landmark technologies of next-generation AI, facial recognition has achieved significant applications in multiple fields, changing people's lifestyles and generating substantial socio-economic benefits. Currently, facial recognition can meet the practical application needs of most static recognition in constrained scenarios, such as "face-scanning" payments, unlocking, access control, attendance, and financial transactions. In these scenarios: 1. Facial posture is fixed, generally without significant posture deviation; 2. Ambient lighting is stable, mostly indoors with no significant changes in lighting; 3. Facial information is sufficient, without strong obstructions such as masks or sunglasses. The environment is controllable, and the user cooperates. However, dynamic facial recognition under unconstrained conditions, which has a wider range of applications and greater significance, remains a major challenge. Existing two-dimensional facial registration databases are all unobstructed frontal photos taken under specific lighting conditions, such as ID card photos, passport photos, and work permit photos. In unconstrained scenarios, wearing masks leads to the loss of some facial information and causes a significant difference between the captured facial image and the registered images in the face database. As a result, the current mainstream 2D-to-2D recognition system is unable to achieve accurate facial recognition in the aforementioned unconstrained scenarios.

[0003] CN111639617A discloses a high-precision face recognition technology for people wearing masks, including training a face recognition model for people wearing masks, training facial key points for people wearing masks, and constructing a face recognition service process. This invention uses cropped images of eyebrows, eyes, and forehead from photos to create a training set for face recognition with masks. Through algorithm training, it obtains an algorithmic recognition model that extracts features from the eyes and forehead. Based on RetinaFace, it performs key point annotation and training on face photos with masks, enabling the model to more accurately extract facial key points from photos with masks, facilitating alignment operations in the face preprocessing stage, thereby improving the recognition accuracy of faces wearing masks.

[0004] In this invention, the face image used to train the algorithm model is a two-dimensional planar image. Feature extraction and key point annotation are all based on this two-dimensional planar image, which is similar to the unobstructed frontal photograph under a specific lighting condition mentioned above. In unconstrained scenarios, there will be a significant difference between the face image acquired by this invention and the image used for training, which is the 2D-2D recognition method used in this invention.

[0005] The industrialization of 3D face sensors has greatly promoted the development of 3D modeling and registration combined with real-time 3D acquisition and recognition technology (referred to as 3D-3D recognition). It has become an industry consensus to attempt to overcome the bottleneck of 2D face recognition using 3D technology. However, for dynamic face recognition in video surveillance, replacing the existing video surveillance network composed of 2D cameras entirely with 3D sensors is not feasible from both a technical and economic perspective. Summary of the Invention

[0006] The purpose of this invention is to overcome the problem of difficulty in dynamic face recognition in video surveillance in the prior art, and to provide a method and system that supports face recognition for people wearing masks, unlike 2D-2D recognition.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A method for supporting facial recognition while wearing a mask includes the following steps:

[0009] Receive the captured facial image to be identified;

[0010] Based on the cross-modal and cross-domain recognition framework, pose estimation and feature extraction are performed on the face image to be identified, and multi-pose feature comparison is performed with the registered features in the registered 3D face database.

[0011] Based on the results of the feature comparison, a recognition score is obtained to complete face recognition;

[0012] The cross-modal and cross-domain recognition framework includes a face image completion network, a face image generation network, and a feature extraction network.

[0013] The face image completion network is used to receive a 3D face model and generate a 3D masked face model, which is then placed into the registered 3D face database; the face image generation network is used to render and augment the images stored in the registered 3D face database to obtain 2D face samples; the feature extraction network is used to extract features from the 2D face samples and generate registration features, which are then placed into the registered 3D face database.

[0014] Preferably, the pose estimation is used to assign variable weights to the corresponding sub-items of the similarity metric function for in-plane or out-of-plane pose changes of the face image to be identified in the recognition environment, during multi-pose feature comparison.

[0015] Preferably, the feature comparison is performed by calculating the cosine distance between the features of the face image to be identified and the registered features to obtain a recognition score.

[0016] Preferably, the face image generation network includes a differentiable rendering module and a feature domain transfer module. The registered images in the registered 3D face database are first input into the differentiable rendering module to form images of different poses, and then input into the feature domain transfer module. The feature domain transfer module augments the two-dimensional face samples in the dimensions of illumination and material.

[0017] Preferably, the differentiable rendering module consists of an encoder, a converter, and a decoder, wherein the encoder is used to extract feature vectors, the converter is used to retain the original image features during conversion, and the decoder is used to reconstruct a two-dimensional image from the feature vectors.

[0018] Preferably, the feature domain transfer module consists of an encoder, a converter, and a decoder, wherein the encoder is used to extract feature vectors, the converter is used to retain the original image features during conversion, and the decoder is used to restore the low-level features through the feature vectors.

[0019] Preferably, the face image completion network performs feature point localization on the 3D face model and fits the 3D face model with a 3D mask according to the 3D information of the mask to synthesize a 3D face image with a mask on.

[0020] Preferably, the feature extraction network is based on a homotopy sparse dictionary learning method, and the feature sparse coding function under information loss is as follows:

[0021]

[0022] Among them, dictionary and Lagrange multipliers As a learnable parameter, To reconstruct errors, For sparse coding, Lagrange multipliers constraint Regular terms Participation weights in optimization.

[0023] Preferably, the global optimization objective function of the feature extraction network is as follows:

[0024]

[0025] in, This is the representation obtained by linear projection of the original generated data points. For each identity corresponding to the sparse encoded channel in this book at different iterations, the parameters are... To balance the differences in external formation among sample modes, parameters To balance the differences in self-deformation between identities across sample modalities, For a learnable sparse dictionary, Representative for constraining feature networks Regular terms The Lagrange multipliers of the participating weights in the optimization.

[0026] To achieve the above-mentioned objectives, the present invention also provides the following technical solutions:

[0027] A system supporting facial recognition while wearing a mask includes a processor, a network interface, and a memory, wherein the processor, the network interface, and the memory are interconnected, and the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute any of the above-described methods for supporting facial recognition while wearing a mask.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] To address the challenge of dynamic face recognition in video surveillance, this invention, unlike 2D-2D recognition, employs a method that uses registered 3D face images as a database and real-time 2D face acquisition for recognition. Through a cross-modal, cross-domain recognition framework, it performs multi-pose feature comparison on the acquired faces, reducing the impact of mask-wearing on recognition features and improving the accuracy of face identification under masks. Simultaneously, it utilizes a face image generation network to render and augment the face data, and a face completion network to complete the identity information of masked faces, effectively compensating for the lack of such effective samples in existing 2D face recognition methods. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention;

[0031] Figure 2 This is a flowchart illustrating the training process of the recognition framework of this invention.

[0032] Figure 3 This is a schematic diagram of the face image completion network training of the present invention;

[0033] Figure 4 This is a schematic diagram of the face image generation network training of the present invention;

[0034] Figure 5 This is a schematic diagram of the operation of the identification framework of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0036] Example 1

[0037] See attached document Figure 1 To address the challenge of dynamic facial recognition in video surveillance, this invention provides a method for facial identification of individuals wearing masks, comprising the following steps:

[0038] Receive the captured facial image to be identified;

[0039] Based on the cross-modal and cross-domain recognition framework, pose estimation and feature extraction are performed on the face image to be identified, and multi-pose feature comparison is performed with the target image in the registered 3D face database.

[0040] Based on the results of the feature comparison, a recognition score is obtained to complete face recognition;

[0041] The cross-modal and cross-domain recognition framework includes a face image completion network, a face image generation network, and a feature extraction network.

[0042] The face image completion network is used to receive a 3D face model and generate a 3D masked face model, which is then placed into the registered 3D face database; the face image generation network is used to render and augment the images stored in the registered 3D face database to obtain 2D face samples; the feature extraction network is used to extract features from the 2D face samples and generate registration features, which are then placed into the registered 3D face database.

[0043] See attached document Figure 2 The diagram below shows the training flowchart of the cross-modal and cross-domain recognition framework of the present invention. The face image generation network renders face images from the augmented and registered 3D face database to generate 2D face samples for training the feature extraction network. The face image completion network completes the 3D face model with a mask on for training the feature extraction network. The feature extraction network is trained based on the 2D face samples and the 3D face model to perform feature extraction, which serves as the support for the cross-modal and cross-domain recognition framework.

[0044] Further, refer to the appendix Figure 3This is a schematic diagram of the face image completion network training for the present invention. First, 3D object simulation is used to simulate missing facial information. 3D simulation technology or a high-precision 3D measuring instrument is used to obtain the 3D information of the mask used as an occlusion. Then, based on the method of locating 3D facial feature points, and according to the mask and facial semantic information, the 3D mask is fitted. Second, based on the inverse mapping relationship between 2D and 3D feature points, the 3D facial feature points are located, and a high-fidelity 3D masked face image is synthesized. Finally, the synthesized masked face image is added to the registered 3D face database. The 3D masked face image is mainly used to suppress the influence of occlusion on feature comparison during the training of the feature extraction network. It should be noted that the 3D feature point localization module is a pre-trained deep neural network model. 3D facial feature points can be obtained through forward propagation. The 3D feature point localization module is based on a common convolutional neural network structure, and the specific operation steps are as follows:

[0045] 1. Precise 3D facial models and 3D information of occluders obtained using sophisticated instruments;

[0046] 2. Input a precise 3D face model obtained from a precision instrument to obtain the coordinates of 68 facial feature points in 3D space. Based on the obtained feature point coordinates, select the coordinate range of the area obscured by the mask. The 3D positioning module, using an Hourglass structure, extracts and fuses features from the 3D face at four different scales to achieve feature point localization of the 3D face.

[0047] Specifically, the feature point localization network sequentially extracts the number of channels from the original information. They are respectively Features at different scales, then sequentially... Layer and First The features of the layers are fused and deconvolved to obtain the location of feature points based on cross-size features.

[0048] 3. The occlusion area selected in step 2 is matched with the high-precision 3D information of the occlusion obtained in step 1 to form a 3D occlusion fit.

[0049] Further, refer to the appendix Figure 4The diagram illustrates the training of the face image generation network of this invention. First, using differentiable rendering and feature domain transfer techniques with high-precision 3D model projection onto the image plane, a large number of complete face images are generated. Simultaneously, image style transfer, horizontal mirroring, saturation adjustment, brightness adjustment, and image blurring operations are used to augment the original face images, reducing the modal difference between training samples and actual samples, and decreasing the feature domain difference between augmented data and real data. Second, using identity features and image quality evaluation criteria as constraints, the differentiable rendering model is trained to simulate the addition of missing information and noise to the high-precision 3D faces, forming high-fidelity cross-feature domain augmented data samples. Finally, the augmented data is added to the registered 3D face database.

[0050] Specifically, the face image generation network consists of a differentiable rendering module and a face feature domain transfer network. First, a high-precision 3D face model is obtained using 3D simulation technology or a high-precision 3D measuring instrument. Then through the differentiable rendering module This process renders 2D face images in different poses. Then, a feature domain transfer network is used... The acquired 2D face images are augmented in terms of lighting and material dimensions to construct highly realistic augmented 2D face samples. The process can be simply described as follows:

[0051]

[0052] in, For high-precision 3D face models, It is a differentiable rendering module. For the feature domain transfer module, It is a two-dimensional face sample.

[0053] The differentiable rendering module consists of an encoder, a converter, and a decoder. The encoder uses three layers of 3D convolution to extract 256 64*64 feature vectors from the 3D model. The converter uses a 5-layer ResNet module, with each Reset module being a neural network layer consisting of two convolutional layers. The decoder layer uses deconvolution to restore the features to a 2D image, resulting in a rendered 2D image of the 3D model. The feature domain transfer module also consists of an encoder, a converter, and a decoder. The encoder uses four layers of 2D convolution to extract 256 64*64 feature vectors. The converter uses a 5-layer ResNet module, with each Reset module being a neural network layer consisting of two convolutional layers, achieving the goal of preserving the original image features during conversion. The decoder layer uses deconvolution to restore low-level features from the feature vectors, finally obtaining the feature domain-transferred image, i.e., a 224*224 2D augmented face sample.

[0054] After the face image completion network synthesizes the 3D face image, the missing locations on the 3D sample are represented by a face spatial occlusion mask, and the mask occlusion feature code represents the mask portion calculated based on the missing locations. During the operation of the face image generation network, multi-dimensional features are extracted from the preprocessed face image, and a synthesis confidence score is calculated based on these features. A preset adjustable threshold parameter is used to determine whether the face is complete or masked. If it is a masked face, the face image is further processed, and the face spatial occlusion mask is used to describe the mask-occluded area. The mask occlusion feature code and corresponding sample are stored offline for later use.

[0055] Simultaneously, after the 2D face sample data is generated, it is detected by the face image completion network to identify mask-occluded areas and calculate the corresponding location masks. Based on the location mask information and augmented data, a spatial attention mechanism is used to train the face missing mask detection module. Masked face images are completed using 3D faces and a visual Transformer. Various loss functions, both existing and newly proposed, are used and combined with 3D face models to improve the visual effect and identity feature preservation capability of the multimodal face completion network.

[0056] Furthermore, the feature extraction network of this invention is composed of an MLP, using a registered 3D face database as the training set. Its key feature is the use of a model training method based on homotopy sparse dictionary constraints. Real samples and generated samples of the same identity are combined into joint data points {x, x', ..., y}, and the global representation space of a single identity is further expanded using a linear projection matrix. Subsequently, a low-rank approximation algorithm is used to obtain the principal components of the above representations as input to an iterative sparse thresholding layer, utilizing a specific sparse dictionary. Learning to acquire robust feature representations The empirical classification loss is calculated using a simple shallow multilayer perceptron. On the other hand, the representation obtained by linear projection of the original generated data points Using a Siamese architecture, we can obtain an accompanying empirical loss without encoding or decoding. For each iteration, each identity in the sparsely encoded channel corresponds to a specific sparse dictionary. The coefficients of its column elements and the variables within each element are represented by a loss term. The relevant prediction function constrains the intra-class representation distance between samples of different modalities, as shown in the following formula:

[0057]

[0058] The model is optimized by solving for the model parameters that minimize the feature vector support (i.e. the sparsest expression), and the model is used as a feature selector to adaptively select highly relevant features from the feature variables, thereby further improving the identity preservation properties and feature discriminative properties of the original 3D model and the augmented samples generated by the model.

[0059] Based on this, a representation model was constructed. To achieve identity preservation constraints, and to optimize the global objective function using this model:

[0060]

[0061] in, This is the representation obtained by linear projection of the original generated data points. For each identity corresponding to the sparse encoded channel in this book at different iterations, the parameters are... To balance the differences in external formation among sample modes, parameters To balance the differences in self-deformation between identities across sample modalities, For a learnable sparse dictionary, Representative for constraining feature networks Regular terms The Lagrange multipliers of the participating weights in the optimization.

[0062] Using the above formula and two jointly learnable parameters and To balance the differences in external formation between sample modalities and the differences in intrinsic deformation between identities, thereby improving the model's overall tolerance to identity-independent deformation, and utilizing non-empirical loss. To globally guide and optimize the robustness of the entire feature extraction network.

[0063] Specifically, the feature extraction network is a model based on a homotopy sparse dictionary learning method, and its feature sparse encoding function under information loss is as follows:

[0064]

[0065] Among them, dictionary and Lagrange multipliers As a learnable parameter, To reconstruct errors, For sparse coding, Lagrange multipliers constraint Regular terms Participation weights in optimization.

[0066] It should be noted that for any input synthetic or real sample, a parameter-shared feature extraction module is used to extract equal-dimensional feature codes. By solving the sparse coding problem that minimizes the code support, a non-linear feature-dimensional thresholding module is iteratively trained and optimized as a pre-feature selection operation for the recognition training module. After training, the features of each pose of the registered images extracted from the registered 3D face database are put into the registered 3D face database to generate registration features.

[0067] Further, refer to the appendix Figure 5 This is a schematic diagram illustrating the operation of the cross-modal, cross-domain recognition framework of the present invention. First, pose estimation is performed to account for in-plane or out-of-plane pose changes of the face image to be recognized in the recognition environment. During multi-pose feature comparison, variable weights are assigned to corresponding sub-items of the similarity metric function. In some recognition environments, certain poses occur significantly more frequently; therefore, pose estimation is performed first to improve overall recognition accuracy, serving as a constraint for model parameter optimization. Subsequently, feature alignment is performed based on registered features, including 3D-to-2D projection transformation and in-plane and out-of-plane alignment. Multiple face depth features are extracted and used for comparison. Finally, the matching score for 3D-2D recognition is calculated.

[0068] The specific steps are as follows:

[0069] A certain pose of the image to be identified The following 2D face images are aligned both in-plane and out-of-plane: Indicates the yaw angle. Indicates the roll angle. The pitch angle is represented, and the alignment results are input into the feature extraction network to obtain 4*512 face features. ;

[0070] Pose estimation is performed on the face image to be recognized to obtain... ;

[0071] Specifically, the feature extraction network uses a face image completion network training method, and is trained for all poses and specific poses (such as a 0-degree frontal face, a 30-degree micro-profile face, a 60-degree full profile face, and a 90-degree full profile face). The model structure is ResNet50;

[0072] The target image is input into a feature extraction network to obtain 4*512 features. ;

[0073] Will with posture Below Feature comparison is performed, and the cosine distance between the two sets of vectors is calculated to obtain the recognition score and complete the face recognition.

[0074] It should be noted that the model for the target image is a set of parameterized models trained by fine-tuning for each specific angle (range). Each model performs feature extraction on the input sample once, and the output results include features of multiple poses. In the comparison stage, the features of each pose are compared with the registered features in the registered 3D face database to calculate a multi-pose similarity, and the identity is identified based on the similarity.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for supporting facial recognition of people wearing masks, characterized in that, Includes the following steps: Receive the captured facial image to be identified; Based on the cross-modal and cross-domain recognition framework, pose estimation and feature extraction are performed on the face image to be identified, and multi-pose feature comparison is performed with the registered features in the registered 3D face database. Based on the results of the feature comparison, a recognition score is obtained to complete face recognition; The cross-modal and cross-domain recognition framework includes a face image completion network, a face image generation network, and a feature extraction network. The face image completion network is used to generate a 3D masked face model from the received 3D face model and put it into the registered 3D face database; the face image generation network is used to perform rendering and augmentation operations on the images stored in the registered 3D face database to generate 2D face samples; the feature extraction network is used to generate registration features from the extracted features of the 2D face samples and put the registration features into the registered 3D face database. The face image generation network includes a differentiable rendering module and a feature domain transfer module. The registered images in the registered 3D face database are first input into the differentiable rendering module to form images of different poses, and then input into the feature domain transfer module. The feature domain transfer module augments the two-dimensional face samples in the dimensions of illumination and material. The feature extraction network is based on a homotopy sparse dictionary learning method, and its function for sparse feature encoding under information loss is as follows: Among them, dictionary and Lagrange multipliers As a learnable parameter, To reconstruct errors, For sparse coding, For sparse coding of registration according to identity classification, the Lagrange multipliers are adjusted. Constrained Feature Extraction Network Regular terms Participation weights in optimization; The global optimization objective function of the feature extraction network is as follows: in, The representation obtained by linear projection of the original generated data points in different iterations. For each identity in the sparsely encoded channel at different iteration numbers, the parameters are: To balance the differences in external formation among sample modes, parameters To balance the differences in self-deformation between identities across sample modalities, This represents the constrained feature extraction network in the i-th iteration. Regular terms The Lagrange multipliers of the participating weights in the optimization.

2. The method for supporting facial recognition while wearing a mask as described in claim 1, characterized in that, The pose estimation is used to assign preset weights to the corresponding sub-items of the similarity measurement function when performing multi-pose feature comparison for in-plane or out-of-plane pose changes of the face image to be identified in the recognition environment.

3. The method for supporting facial recognition while wearing a mask as described in claim 1, characterized in that, The feature comparison calculates a recognition score by performing cosine distance calculation on the features of the face image to be identified and the registered features.

4. The method for supporting facial recognition while wearing a mask as described in claim 1, characterized in that, The differentiable rendering module consists of an encoder, a converter, and a decoder. The encoder is used to extract feature vectors, the converter is used to retain the original image features during conversion, and the decoder is used to reconstruct a two-dimensional image using the feature vectors.

5. The method for supporting facial recognition while wearing a mask as described in claim 1, characterized in that, The feature domain transfer module consists of an encoder, a converter, and a decoder. The encoder is used to extract feature vectors, the converter is used to retain the original image features during conversion, and the decoder is used to restore low-level features through the feature vectors.

6. The method for supporting facial recognition while wearing a mask as described in claim 1, characterized in that, The face image completion network performs feature point localization on a 3D face model and fits a 3D mask onto the 3D face model based on the 3D information of the mask, in order to synthesize a 3D face image with a mask on.

7. A system that supports facial recognition for people wearing masks, characterized in that, The device includes a processor, a network interface, and a memory, which are interconnected. The memory stores a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute a method for supporting facial recognition of a person wearing a mask, as described in any one of claims 1-6.

Images