A face recognition method and device based on fast mask generation

Abstract

This invention discloses a face recognition method and apparatus based on fast mask generation. The method includes: training an initial face recognition model using an initial training face image set; adding occlusion masks to the initial training face image set with at least two mask occlusion probabilities, random transparency, and random color to obtain at least two sets of masked training face image sets, and training at least two fine-tuned face recognition models; performing self-supervised training on each fine-tuned face recognition model; fusing the at least two fine-tuned face recognition models after self-supervised training using a distillation method to obtain a target face recognition model; and performing face recognition using the target face recognition model. The face recognition method provided by this invention is applicable to various scenarios; due to the use of self-supervised training and multi-model fusion, the generalization of the target face recognition model and its recognition performance under various occlusion scenarios are significantly improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a face recognition method and apparatus based on fast mask generation. Background Technology

[0002] With the development of computer information technology and the rise of artificial intelligence, many algorithms for deep learning have emerged and are continuously being improved and optimized. Among them, face recognition, as a fundamental task in computer vision, has a wide range of research significance and applications. Face recognition is currently an effective, convenient, and fast technology for quickly extracting biometric features and identifying individuals based on these features, and it has great development potential and application prospects.

[0003] In daily life, there are many situations where faces are obscured or people are wearing masks during facial recognition. Obscured faces lose a lot of facial information, especially when wearing masks, which can cover up to 65% of the information. This causes a sharp drop in the performance of facial recognition models.

[0004] Currently, many methods have been proposed to address the issue of occlusion in face recognition. For example, the face recognition model training method and face recognition method disclosed in Chinese patent document CN113486701A include a sample generation step and a training step. The sample generation step includes: acquiring a first face image and first annotation data for the first face image, wherein the first face image is a face image without a mask; extracting multiple facial feature points from the first face image; determining the area enclosed by multiple preset feature points among the multiple facial feature points as the mask area; filling the mask area with a preset pattern to obtain a second face image; and annotating the second face image according to the first annotation data to obtain second annotation data. The training step includes: training the model to be trained based on the second face image and the second annotation data; and then using the trained face recognition model to perform face recognition.

[0005] The inventors recognized that the method disclosed in the document simply uses a second face image containing the mask area to train a single face recognition model, which is then directly used for face recognition. This makes the resulting face recognition model not only applicable to face recognition scenarios with masks on, but also has very limited performance. Summary of the Invention

[0006] Based on this, and in response to the aforementioned technical problems, a face recognition method and apparatus based on fast mask generation are provided to solve the technical problem that existing face recognition methods simply use face images containing mask areas for one training session to obtain a single face recognition model, which is then directly used for face recognition, resulting in very limited performance of the face recognition model.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] Firstly, a face recognition method based on fast mask generation includes:

[0009] S1, preprocess the publicly available dataset to obtain the initial training face image set;

[0010] S2, Use the initial training face image set to train a convolutional neural network to obtain an initial face recognition model;

[0011] S3, in the initial training face image set, add occlusion masks with at least two occlusion probabilities to form occlusion areas for some faces in the initial training face image set, and obtain at least two sets of training face images with added masks; set random transparency when adding occlusion masks, and use multiple colors to render randomly;

[0012] S4, the initial face recognition model is trained using the at least two sets of masked training face image sets to obtain at least two fine-tuned face recognition models.

[0013] S5, add an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set, and use the initial training face image set and the self-supervised training mask image set to perform self-supervised training on each fine-tuned face recognition model respectively.

[0014] S6. Use the distillation method to fuse the at least two fine-tuned face recognition models after self-supervised training to obtain the target face recognition model.

[0015] S7, perform face recognition using the target face recognition model.

[0016] Optionally, step S1 specifically includes:

[0017] The system detects whether each image in the publicly available dataset contains a human face, and detects the confidence level and size of the face. Images that do not contain a human face are filtered out, as are images that contain a human face with a confidence level lower than a preset confidence threshold or a face with a size smaller than a preset size threshold.

[0018] Facial landmark detection is performed on each image in the filtered dataset to locate the coordinates of the facial landmarks, and facial alignment is performed on each image in the filtered dataset.

[0019] Each image in the face-aligned dataset is cropped to a preset fixed size that includes the face, resulting in the initial training face image set.

[0020] Further optionally, the preset size threshold is 60 pixels * 60 pixels, and the preset fixed size is 112 pixels * 112 pixels.

[0021] Further, optionally, face alignment is performed on each image in the filtered dataset, specifically including:

[0022] For each image in the filtered dataset, select the coordinates of multiple key points for the eyes, nose, and corners of the mouth;

[0023] For each image in the filtered dataset, an affine transformation is performed according to the selected key point coordinates to achieve face alignment.

[0024] Further optionally, in step S3, the face is specifically formed by connecting multiple key points of the face to create occlusion areas with different organ occlusion ratios; the occlusion areas include simultaneously occluding the nose, mouth and chin, exposing the nose but occluding the mouth and chin, and exposing the nose and mouth but occluding the chin.

[0025] The transparency is randomly selected from 10%, 30%, 50%, and 70%, and the multiple colors include black, white, light blue, and dark green.

[0026] Optionally, in step S2, when training the convolutional neural network using the initial training face image set, the classification loss function uses cosface or arcface.

[0027] Optionally, in step S5, the step of performing self-supervised training on each fine-tuned face recognition model using the initial training face image set and the self-supervised training mask image set specifically includes:

[0028] S51, Construct a feature template queue, and randomly initialize the feature template queue during training initialization;

[0029] S52, use the fine-tuned face recognition model to perform feature inference on an original image in the initial training face image set to obtain the corresponding feature Embedding1;

[0030] S53, using the fine-tuned face recognition model, feature reasoning is performed on the corresponding mask image of the original image in the self-supervised training mask image set to obtain the corresponding feature Embedding2;

[0031] S54, the result of multiplying the corresponding feature Embedding1 and the corresponding feature Embedding2 after normalization is output as the corresponding positive sample, and the label is recorded as 1; the result of multiplying the corresponding feature Embedding1 and the feature template queue is output as the corresponding negative sample, and the label is recorded as 0.

[0032] S55, after connecting the corresponding positive samples and negative samples, a binary classification loss function is output to the loss layer, and the corresponding feature Embedding1 is used to cover the feature template queue in the queue order. After the feature template queue is completely covered, the covering is performed again from the beginning.

[0033] S56, S52-S55 are executed iteratively and repeatedly to traverse all original images in the initial training face image set and complete the self-supervised training of the fine-tuned face recognition model;

[0034] S57. For each fine-tuned face recognition model, execute steps S51-S56 to complete the self-supervised training of each fine-tuned face recognition model.

[0035] Optionally, step S6 specifically includes:

[0036] The at least two finely tuned face recognition models, which have been trained under self-supervised supervision, are used to extract feature vectors from the input images. The extracted feature vectors are then used as teacher feature vectors to guide the training of the student neural network.

[0037] The student neural network is used to extract feature vectors from the input image to obtain student feature vectors;

[0038] Fit the student feature vector to the teacher feature vector;

[0039] The student neural network is used as the target face recognition model.

[0040] Further, optionally, when fitting the student feature vector to the teacher feature vector, the loss function uses mean squared error.

[0041] Secondly, a face recognition device based on fast mask generation includes:

[0042] The preprocessing module is used to preprocess the publicly available dataset to obtain an initial training face image set;

[0043] The initial face recognition model training module is used to train a convolutional neural network using the initial training face image set to obtain an initial face recognition model;

[0044] The occlusion mask adding module is used to add occlusion masks to the initial training face image set with at least two occlusion probabilities to form occlusion regions for some faces in the initial training face image set, thereby obtaining at least two sets of training face image sets with added masks; when adding occlusion masks, random transparency is set and multiple colors are used for random rendering;

[0045] The fine-tuning face recognition model training module is used to train the initial face recognition model using the at least two sets of masked training face image sets respectively, so as to obtain at least two fine-tuning face recognition models.

[0046] The self-supervised training module is used to add an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set. The initial training face image set and the self-supervised training mask image set are used to perform self-supervised training on each fine-tuned face recognition model.

[0047] The model fusion module is used to perform model fusion on the at least two fine-tuned face recognition models after self-supervised training using a distillation method to obtain the target face recognition model.

[0048] A face recognition module is used to perform face recognition using the target face recognition model.

[0049] The present invention has at least the following beneficial effects:

[0050] This invention provides an end-to-end fast training method for a face recognition model. First, an initial face recognition model is trained using an initial set of training face images. Then, occlusion masks are added to the initial set of training face images with at least two mask occlusion probabilities, random transparency, and random color, resulting in at least two sets of masked training face images. At least two fine-tuned face recognition models are then trained using these masks. After self-supervised training of these at least two fine-tuned face recognition models, a knowledge distillation method is used to select these at least two fine-tuned face recognition models as target face recognition models for face recognition. The method provided by this invention provides a fast end-to-end training method for face recognition. When training the face recognition model, the training sample set includes samples with and without occlusion masks. Furthermore, the training sample set takes into account different mask occlusion conditions in real-world scenarios using varying transparency and color. This not only enables the face recognition model to be applicable to both masked and unmasked faces, but also makes it suitable for various scenarios. Simultaneously, this invention uses self-supervised training and multi-model fusion to obtain the final target face recognition model, significantly improving its generalization ability and further ensuring a marked improvement in recognition performance under various occlusion scenarios. Attached Figure Description

[0051] Figure 1 A flowchart illustrating a face recognition method based on fast mask generation, provided as an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the preprocessing steps in one embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the self-supervised training steps in one embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the model fusion step in one embodiment of the present invention;

[0055] Figure 5 A block diagram of a face recognition device based on fast mask generation is provided as an embodiment of the present invention;

[0056] Figure 6 This is an internal structural diagram of a computer device provided in one embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In one embodiment, such as Figure 1 As shown, a face recognition method based on fast mask generation is provided, including the following steps:

[0059] S1: Preprocess the publicly available dataset to obtain the initial training face image set.

[0060] Furthermore, step S1 specifically includes:

[0061] The system detects whether each image in the publicly available dataset contains a human face, and detects the confidence level and size of the face. Images that do not contain a human face are filtered out, as are images that contain a human face with a confidence level lower than a preset confidence threshold or a face with a size smaller than a preset size threshold.

[0062] Facial landmark detection is performed on each image in the filtered dataset to locate the coordinates of the facial landmarks, and facial alignment is performed on each image in the filtered dataset.

[0063] Each image in the face-aligned dataset is cropped to a preset fixed size that includes the face, resulting in the initial training face image set.

[0064] The preset size threshold is 60 pixels * 60 pixels, and the preset fixed size is 112 pixels * 112 pixels.

[0065] Furthermore, face alignment is performed on each image in the filtered dataset, specifically including:

[0066] For each image in the filtered dataset, select the coordinates of multiple key points for the eyes, nose, and corners of the mouth;

[0067] For each image in the filtered dataset, an affine transformation is performed according to the selected key point coordinates to achieve face alignment.

[0068] In other words, such as Figure 2 As shown, step S1 first detects the presence of faces and obtains the confidence score and size of each face. Faces with a confidence score below a threshold or a size smaller than 60*60 are filtered out. Then, 106 key points are obtained through facial landmark detection. Using five key points on each side of the eyes, nose, and mouth, an affine transformation is applied to align the face, which is then cropped to a fixed size of 112*112.

[0069] S2, using the initial training face image set to train a convolutional neural network to obtain the initial face recognition model.

[0070] Furthermore, in step S2, when training the convolutional neural network using the initial training face image set, the classification loss function uses cosface or arcface.

[0071] In other words, step S2 involves pre-learning, using a publicly available face dataset to train an initial convolutional neural network model. The initial model has a certain recognition capability in various scenarios, but its performance is very poor in occluded scenarios, failing to meet the high-performance requirements of occluded scenarios.

[0072] S3, in the initial training face image set, add occlusion masks with at least two occlusion probabilities to form occlusion areas for some faces in the initial training face image set, and obtain at least two sets of training face images with added masks; set random transparency when adding occlusion masks, and use multiple colors for random rendering.

[0073] Furthermore, in step S3, the face is specifically formed by connecting multiple key points of the face to create occlusion areas with different organ occlusion ratios; the occlusion areas include simultaneously occluding the nose, mouth and chin, exposing the nose but occluding the mouth and chin, and exposing the nose and mouth but occluding the chin.

[0074] The transparency is randomly selected from 10%, 30%, 50%, and 70%, and the colors include black, white, light blue, and dark green. In other words, in each set of masked training face images, the transparency and color of the mask in each image with an added mask are randomized; therefore, each set of masked training face images contains masking masks with different transparency and color settings.

[0075] In other words, after performing the preprocessing process to obtain the pairing of 106 facial key points with images, in the image enhancement module, the connections between facial key points are used to form occlusion regions on the face in the initial training face image set. Different connection methods can form different occlusion regions, thereby simulating the occlusion of the face when the face is wearing a mask or in special situations in real-world scenarios.

[0076] There are three main ways to use key points to cover a face: covering the nose, mouth, and chin; showing the nose but covering the mouth and chin; and showing the nose and mouth but covering the chin. When adding a mask, in addition to these three methods, the coverage ratio of the nose, mouth, and chin can also be adjusted, covering almost all occlusion situations in real-world scenarios.

[0077] Different colors are used to render the occluded areas. Faces wearing masks or being occluded will have different colors and transparency. The main colors are black, white, light blue, and dark green. The transparency is set to adapt to the mapping of faces under different lighting conditions in the actual scene. The transparency can also simulate the different materials of the mask, so it is set to 10%, 30%, 50%, and 70%.

[0078] To improve face recognition performance under occlusion conditions while maintaining compatibility with face recognition performance under unoccluded conditions, the probability of setting an occlusion mask for faces can be adjusted. The probability of adding a mask during training can be set to 20%, 30%, etc.

[0079] S4. Use at least two sets of masked training face image sets to train the initial face recognition model, and obtain at least two fine-tuned face recognition models.

[0080] The face is trained based on facial landmarks, and a mask is generated. The generated mask images are then used to train a fine-tuned face recognition model. In other words, the initial face recognition model is fine-tuned by adding a mask to train a set of face images. This fine-tuned face recognition model has good recognition capabilities in occluded or masked scenarios.

[0081] Step S4 involves fine-tuning the model using the obtained initial face recognition model parameters.

[0082] S5 adds an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set. Using the initial training face image set and the self-supervised training mask image set, self-supervised training is performed on each fine-tuned face recognition model.

[0083] When adding an occlusion mask to each image in the initial training face image set, you can also set random transparency and use multiple colors for random rendering.

[0084] Further, in step S5, using the initial training face image set and the self-supervised training mask image set, self-supervised training is performed on each fine-tuned face recognition model, specifically including:

[0085] S51, Construct a feature template queue, and randomly initialize the feature template queue during training initialization;

[0086] S52, use the fine-tuned face recognition model to perform feature inference on an original image in the initial training face image set to obtain the corresponding feature Embedding1;

[0087] S53, using a fine-tuned face recognition model to perform feature inference on the corresponding mask image of the original image in the self-supervised training mask image set, to obtain the corresponding feature Embedding2;

[0088] S54, the result of multiplying the corresponding feature Embedding1 and the corresponding feature Embedding2 after normalization is used as the corresponding positive sample output, and the label is recorded as 1; the result of multiplying the corresponding feature Embedding1 and the feature template queue is used as the corresponding negative sample output, and the label is recorded as 0;

[0089] S55, after connecting the corresponding positive samples and negative samples, form a binary classification loss function and output it to the loss layer, and use the corresponding feature Embedding1 to cover the feature template queue in the queue order. After the feature template queue is completely covered, the covering is performed again from the beginning.

[0090] S56, continuously iterate and repeat S52-S55, traversing all original images in the initial training face image set to complete the self-supervised training of the fine-tuned face recognition model;

[0091] S57. For each fine-tuned face recognition model, execute steps S51-S56 to complete the self-supervised training of each fine-tuned face recognition model.

[0092] In other words, such as Figure 3As shown, the original image is used to obtain feature Embedding1 through fine-tuning of the face recognition model. After face masking, the original image is used to obtain feature Embedding2 through fine-tuning of the face recognition model. Feature Embedding1 and feature Embedding2 are normalized and multiplied to obtain positive sample output. Feature Embedding2 is multiplied with the feature template queue to obtain negative sample output. The positive and negative samples are concatenated to form a binary classification output to the loss layer. Then, feature Embedding1 is used to cover the feature template queue.

[0093] Repeat the above steps in each iteration, and after the feature template queue is completely covered, start covering it again from the beginning.

[0094] In other words, in step S5, all original images in the initial training face image set are first occluded and rendered. The rendered data is processed by a neural network to obtain feature embedding. The modulus of the feature embedding is calculated and normalized to [0,1]. The modulus is used as a pseudo-label and multiplied with the output of the softmax layer in the training neural network and then input into the loss layer.

[0095] Then, each fine-tuned face recognition model is fine-tuned again using the model parameters of each fine-tuned face recognition model, and trained in a self-supervised manner.

[0096] Self-supervised training first constructs a queue whose length is an integer multiple of the training input mini-batch. At the beginning of training, the queue is randomly initialized. After each iteration, the queue is filled with the embeddings output by the model. After filling, the queue is overwritten from the beginning.

[0097] The unmasked image input to each fine-tuned face recognition model is called the original image. The original image, after occlusion and rendering, is called the masked image. The feature embeddings output by both the masked image and the original image after passing through the neural network are normalized and multiplied together as positive samples with a label of 1. The feature embeddings of the masked image are multiplied by the queue as negative samples with a label of 0. The positive and negative samples are concatenated and then input into the classification layer, which has two classes.

[0098] Self-supervised training uses a small learning rate for training and an adaptive optimizer for tuning. This self-supervised training method using masked images and original images can greatly improve the generalization of the model and significantly improve the performance of face recognition authentication scenarios.

[0099] S6. Use the distillation method to fuse at least two fine-tuned face recognition models that have been trained under self-supervised supervision to obtain the target face recognition model.

[0100] Furthermore, step S6 specifically includes:

[0101] At least two finely tuned face recognition models trained under self-supervised supervision are used to extract feature vectors from the input image. The extracted feature vectors are then used as teacher feature vectors to guide the training of the student neural network.

[0102] The student neural network is used to extract feature vectors from the input image to obtain the student feature vector;

[0103] Make the student eigenvectors and the teacher eigenvectors;

[0104] The student neural network was used as the target face recognition model.

[0105] There are no restrictions on the input images; they can be any original image selected from the initial training face image set, or an image with an added occlusion mask selected from either the training face image set or the self-supervised training mask image set. When fitting the student's feature vector to the teacher's feature vector, the mean squared error loss function is used.

[0106] In other words, model fusion involves the following steps:

[0107] Step (1): Train two or more models with different mask probabilities. These models have good recognition performance in different occlusion scenarios.

[0108] Step (2): Use the model to extract the feature vector of the input image as the teacher feature vector, which is used to guide the training and learning of the student network.

[0109] Step (3): The input image is processed through a student convolutional neural network to obtain the student feature vector.

[0110] Step (4): Use the mean squared error (MSE) loss function to fit the student feature vector to the teacher feature vector.

[0111] Overall, such as Figure 4 As shown, in the model fusion step, multiple fine-tuned face recognition models are trained. Each fine-tuned face recognition model is trained on a set of face images with added masks, obtained by adding occlusion masks with different probabilities. Then, feature vectors are extracted from the images using these multiple fine-tuned face recognition models. Subsequently, using the extracted feature vectors and knowledge distillation methods, with MSE (mean squared error) as the loss function, a student model is trained. This student model shows significant performance improvements in face recognition under multiple specific scenarios.

[0112] S7 uses a target face recognition model for face recognition.

[0113] This target face recognition model can be used to obtain feature recognition results for captured images of faces with occlusions, and then determine whether the captured image and a certain face information in the database belong to the same person. It can be applied to many fields such as identity verification in financial systems, check-in systems, and monitoring.

[0114] Currently, many methods have been proposed to address occlusion issues in face recognition. These include using face segmentation and 3D rendering to add occlusion information to faces, simulating real-world instances of occluded faces, and then adding these simulated occluded faces to the dataset for training. However, this approach presents two problems: first, face segmentation or 3D rendering can destroy the original image information, adding noise and causing the model to fail to converge during training; second, rendering is time-consuming, requiring significant time for data preprocessing.

[0115] The method provided in this embodiment can train the model end-to-end, and a quality evaluation unit is added during the training process to enhance the robustness of the model. Furthermore, the self-supervised approach can further improve the face recognition performance when the face is occluded.

[0116] This embodiment addresses the issue that face recognition performance drops sharply when faces are obscured or people are wearing masks. Existing methods result in noisy images and long rendering times. Therefore, a face recognition method based on fast mask generation is proposed.

[0117] This embodiment addresses the issues of a lack of real masks in the training set and the long rendering time required for masks. Through an end-to-end rapid training method, it can obtain a face recognition model with the same or even better recognition performance as the model trained using real mask samples or mask rendering samples. By using a distillation method to fuse multiple models, the performance of the recognition model is significantly improved in various scenarios.

[0118] It should be understood that, although Figure 1-4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0119] In one embodiment, such as Figure 5As shown, a face recognition device based on fast mask generation is provided, including the following program modules:

[0120] The preprocessing module 501 is used to preprocess the publicly available dataset to obtain an initial training face image set;

[0121] The initial face recognition model training module 502 is used to train a convolutional neural network using the initial training face image set to obtain the initial face recognition model;

[0122] The occlusion mask adding module 503 is used to add occlusion masks to the initial training face image set with at least two mask occlusion probabilities, forming occlusion areas for some faces in the initial training face image set, and obtaining at least two sets of training face image sets with added masks; when adding occlusion masks, random transparency is set and multiple colors are used for random rendering.

[0123] The fine-tuning face recognition model training module 504 is used to train the initial face recognition model using at least two sets of masked training face image sets to obtain at least two fine-tuning face recognition models.

[0124] The self-supervised training module 505 is used to add an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set. Using the initial training face image set and the self-supervised training mask image set, self-supervised training is performed on each fine-tuned face recognition model.

[0125] The model fusion module 506 is used to fuse at least two fine-tuned face recognition models that have been trained under self-supervised supervision using a distillation method to obtain the target face recognition model.

[0126] The face recognition module 507 is used to perform face recognition using a target face recognition model.

[0127] For specific limitations regarding a face recognition device based on fast mask generation, please refer to the limitations of a face recognition method based on fast mask generation described above, which will not be repeated here. Each module in the aforementioned face recognition device based on fast mask generation can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0128] In one embodiment, a face recognition system based on fast mask generation is provided. This system comprises three units: a preprocessing unit, a model training unit, and a model fusion unit.

[0129] The preprocessing unit is used to process the input image through face detection, facial landmark detection, face alignment, and cropping to a fixed size as a training image.

[0130] The model training unit is used to train the face based on facial key points, render and generate a mask, and train on the generated mask image. This method is used as an enhancement module for model training, enabling the face recognition model to have good recognition capabilities in occluded or masked scenarios.

[0131] The model fusion unit is used to extract features from pre-trained face recognition models with different probabilities and masked faces. The features are then obtained through distillation. The student model performs significantly better than the individual model in scenarios with occlusion and masked faces.

[0132] Furthermore, the preprocessing unit includes:

[0133] The face detection unit is used to detect whether there are faces in the image and whether the face size meets the face recognition size (60 pixels * 60 pixels). Images that do not meet the requirements are filtered out.

[0134] The facial landmark detection unit is used to locate 106 key points of the face;

[0135] The face alignment unit aligns the face into a frontal view by translating, rotating, scaling, and other operations according to key points, which facilitates face recognition tasks.

[0136] The cropping unit crops the image to a fixed size (112 pixels * 112 pixels) that includes the face.

[0137] In other words, the preprocessing unit includes the following steps:

[0138] Step (1): The input image is processed by face detection. If there are no faces or the faces are smaller than the threshold and size requirements, they are filtered out and the next image is preprocessed.

[0139] Step (2): Facial landmark detection is performed on the qualified face to obtain the coordinates of 106 key points of the face.

[0140] Step (3): Select the coordinates of 5 key points of the eyes, nose and mouth and perform affine transformation to align them.

[0141] Step (4): Crop the face image to a fixed size (112*112).

[0142] In summary, the preprocessing unit first detects the presence of faces and obtains their confidence level and size. Faces with confidence levels below a threshold or sizes smaller than 60*60 pixels are filtered out. Next, 106 facial landmarks are detected. Using five landmarks on each eye, nose, and corner of the mouth, the face is aligned using affine transformation and then cropped to a fixed size of 112*112 pixels. All input images for subsequent model training units are images preprocessed using this method.

[0143] Furthermore, the model training unit includes:

[0144] (1) Data processing unit: The model data input adopts the method of matching 106 facial key points with images obtained by the preprocessing unit. After the input data, in the image enhancement module, the connection between facial key points is used to form an occlusion area of ​​the face. Different connection methods can form different occlusion areas, thereby simulating the occlusion of the face when the face is wearing a mask or in special situations in the actual scene.

[0145] There are three main ways to use key points to cover a face: covering the nose, mouth, and chin; showing the nose but covering the mouth and chin; and showing the nose and mouth but covering the chin. In the data augmentation module, not only are there the above three ways of covering, but the covering ratio of the nose, mouth, and chin can also be adjusted, covering almost all covering situations in real-world scenarios.

[0146] Different colors are used to render the occluded areas. Faces wearing masks or being occluded will have different colors and transparency. The colors are mainly black, white, light blue, and dark green. The transparency is set to adapt to the mapping of faces under different lighting conditions in the actual scene. The transparency can also simulate different materials of the mask, so it is set to 10%, 30%, 50%, and 70%.

[0147] To improve face recognition performance under occlusion conditions while maintaining compatibility with face recognition performance under unoccluded conditions, the probability of setting an occlusion mask for faces in the data augmentation module can be adjusted, with a default setting of 20%.

[0148] (2) Model training subunit, used for training face recognition models with masks or occlusions in real-world scenarios. Quality evaluation scores are added to enhance the robustness of model training. Finally, self-supervised training is used to improve the performance of face recognition with masks and occlusions in real-world scenarios.

[0149] The model training subunit includes the following steps:

[0150] Step a: Train an initial face model using a convolutional neural network on a publicly available face dataset. This face model is trained using a classification loss function of cosface or arcface and has a certain recognition ability in various scenarios, but it still cannot meet the high performance requirements in occluded scenarios.

[0151] Step b: Fine-tune the model using the model parameters obtained in step a, and process the input data using the data processing unit. The rendered data passes through the neural network to obtain the feature embedding. Calculate the magnitude of the feature embedding and normalize it to [0,1]. Multiply the magnitude as a pseudo-label with the output of the softmax layer in the training neural network and input it into the loss layer.

[0152] Step c: Use the model parameters obtained in step b to fine-tune the model again, and train it in a self-supervised manner.

[0153] Self-supervised training first constructs a queue, the length of which is an integer multiple of the training input mini-batch. At the beginning of training, the queue is randomly initialized. After each iteration, the queue is filled with the model's output embeddings, and then the process is repeated from the beginning. The unmasked image input to the model is called the original image. After processing by the data processing unit, the original image is called the masked image. The product of the normalized feature embeddings output from both the masked image and the original image is used as a positive sample with a label of 1. The product of the feature embeddings of the masked image and the queue is used as a negative sample with a label of 0. The positive and negative samples are concatenated and then input into the classification layer, which has two classes.

[0154] Training is performed using a small learning rate and optimized using an adaptive optimizer. This self-supervised training method using masked images and original images greatly improves the generalization of the model and significantly enhances performance in face recognition authentication scenarios.

[0155] In other words, the model training unit involves the following steps:

[0156] Step (1): Pre-learning uses a public dataset to train an initial convolutional neural network model. The classification loss function uses cosface. The initial model has a certain recognition ability, but its performance is very poor in occluded scenes.

[0157] Step (2): Use the data augmentation module to add a mask to the input image, and use the initial model in step (1) to fine-tune the masked dataset to train a fine-tuned model. During the training process, the probability of adding a mask is set to 0.2.

[0158] Step (3): Perform self-supervised training on the model from step (2) again. First, construct a feature template queue, which is randomly initialized during training initialization.

[0159] Step (4): The original image is inferred by the model to obtain feature Embedding1. After the original image is masked, it is inferred by the model to obtain feature Embedding2. Feature Embedding1 and feature Embedding2 are normalized and multiplied to obtain positive sample output. Feature Embedding2 is multiplied by the feature template queue in step (3) to obtain negative sample output. The positive and negative samples are connected to form a binary classification output to the loss layer. Then feature Embedding1 covers the feature template queue.

[0160] Step (5): Repeat step (4) for each iteration. After the feature template queue is completely covered, start covering again from the beginning.

[0161] In summary, in the model training unit, an initial model is first trained using a public dataset. Then, the data augmentation module is used to add face masking for fine-tuning training, and the model robustness is enhanced by adding modulus to judge image quality. Through self-supervised learning training, the ability to recognize occluded faces in real-world scenarios is further improved.

[0162] Furthermore, the model fusion unit includes the following steps:

[0163] Step 1): Train multiple models. The models are trained using the data processing unit method under different occlusion probabilities and feature vectors are extracted from the images.

[0164] Step 2): Using the feature vectors and knowledge distillation method from Step 1), and the loss function is MSE (mean squared error), train a student model. This model shows significant performance improvements in multiple specific scenarios in Step 1).

[0165] Specifically, model fusion involves the following steps:

[0166] Step (1): Use the model training unit to train two or more models with different mask probabilities. These models have good recognition performance in different occlusion scenarios.

[0167] Step (2): Use the model to extract the feature vector of the input image as the teacher feature vector, which is used to guide the training and learning of the student network.

[0168] Step (3): The input image is processed through a student convolutional neural network to obtain the student feature vector.

[0169] Step (4): Use the mean squared error (MSE) loss function to fit the student feature vector to the teacher feature vector.

[0170] In summary, in the model fusion unit, the input image is cropped to a fixed size by the preprocessing unit, and then trained by the model training unit to obtain recognition models under different occlusion probabilities. The feature vectors of the corresponding training images are extracted, and a student model is trained using the feature vectors and knowledge distillation. The student model has good recognition performance in occluded scenarios.

[0171] In summary, in the face recognition system based on fast mask generation provided in this embodiment, the preprocessing unit mainly performs image preprocessing operations such as face detection, facial landmark detection, face alignment, and image cropping on the input image; the model training unit trains the face based on the facial landmarks, renders and generates a mask, and trains on the generated mask image, using this method as an enhancement module for model training, enabling the face recognition model to have good recognition capabilities in occluded or masked scenarios; the model fusion unit extracts feature vectors from the images obtained under different mask occlusion probabilities, and then uses a distillation method and these feature vectors to distill a student model, which has good recognition capabilities in various scenarios.

[0172] This embodiment addresses the issues of a lack of real masks in the training set and the long rendering time required for masks. Through an end-to-end rapid training method, it can obtain a face recognition model with the same or even better recognition performance as the model trained using real mask samples or mask rendering samples. By using a distillation method to fuse multiple models, the performance of the recognition model is significantly improved in various scenarios.

[0173] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the face recognition method based on fast mask generation provided in the above embodiment. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0174] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program relating to all or part of the processes in the methods of the above embodiments.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon relating to all or part of the processes in the methods of the above embodiments.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A face recognition method based on fast mask generation, characterized in that, include: S1, preprocess the publicly available dataset to obtain the initial training face image set; S2, Use the initial training face image set to train a convolutional neural network to obtain an initial face recognition model; S3, in the initial training face image set, add occlusion masks with at least two occlusion probabilities to form occlusion areas for some faces in the initial training face image set, and obtain at least two sets of training face images with added masks; set random transparency when adding occlusion masks, and use multiple colors to render randomly; S4, the initial face recognition model is trained using the at least two sets of masked training face image sets to obtain at least two fine-tuned face recognition models. S5, add an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set, and use the initial training face image set and the self-supervised training mask image set to perform self-supervised training on each fine-tuned face recognition model respectively. Using the initial training face image set and the self-supervised training mask image set, self-supervised training is performed on each fine-tuned face recognition model, specifically including: S51, Construct a feature template queue, and randomly initialize the feature template queue during training initialization; S52, use the fine-tuned face recognition model to perform feature inference on an original image in the initial training face image set to obtain the corresponding feature Embedding1; S53, using a fine-tuned face recognition model, feature reasoning is performed on the corresponding mask image of the original image in the self-supervised training mask image set to obtain the corresponding feature Embedding2; S54, the result of multiplying the corresponding feature Embedding1 and the corresponding feature Embedding2 after normalization is output as the corresponding positive sample, and the label is recorded as 1; the result of multiplying the corresponding feature Embedding1 and the feature template queue is output as the corresponding negative sample, and the label is recorded as 0. S55, after connecting the corresponding positive samples and negative samples, a binary classification loss function is output to the loss layer, and the corresponding feature Embedding1 is used to cover the feature template queue in the queue order. After the feature template queue is completely covered, the covering is performed again from the beginning. S56, S52-S55 are executed iteratively and repeatedly to traverse all original images in the initial training face image set and complete the self-supervised training of the fine-tuned face recognition model; S57. For each fine-tuned face recognition model, execute steps S51-S56 to complete the self-supervised training of each fine-tuned face recognition model. S6. Use the distillation method to fuse the at least two fine-tuned face recognition models after self-supervised training to obtain the target face recognition model. S7, perform face recognition using the target face recognition model.

2. The face recognition method based on fast mask generation according to claim 1, characterized in that, Step S1 specifically includes: The system detects whether each image in the publicly available dataset contains a human face, and detects the confidence level and size of the face. Images that do not contain a human face are filtered out, as are images that contain a human face with a confidence level lower than a preset confidence threshold or a face with a size smaller than a preset size threshold. Facial landmark detection is performed on each image in the filtered dataset to locate the coordinates of the facial landmarks, and facial alignment is performed on each image in the filtered dataset. Each image in the face-aligned dataset is cropped to a preset fixed size that includes the face, resulting in the initial training face image set.

3. The face recognition method based on fast mask generation according to claim 2, characterized in that, The preset size threshold is 60 pixels * 60 pixels, and the preset fixed size is 112 pixels * 112 pixels.

4. The face recognition method based on fast mask generation according to claim 2, characterized in that, Face alignment is performed on each image in the filtered dataset, specifically including: For each image in the filtered dataset, select the coordinates of multiple key points for the eyes, nose, and corners of the mouth; For each image in the filtered dataset, an affine transformation is performed according to the selected key point coordinates to achieve face alignment.

5. The face recognition method based on fast mask generation according to claim 2, characterized in that, In step S3, specifically, the face is formed by connecting multiple key points of the face to create occlusion areas with different organ occlusion ratios; the occlusion areas include simultaneously occluding the nose, mouth and chin, exposing the nose but occluding the mouth and chin, and exposing the nose and mouth but occluding the chin. The transparency is randomly selected from 10%, 30%, 50%, and 70%, and the multiple colors include black, white, light blue, and dark green.

6. The face recognition method based on fast mask generation according to claim 1, characterized in that, In step S2, when training the convolutional neural network using the initial training face image set, the classification loss function uses cosface or arcface.

7. The face recognition method based on fast mask generation according to claim 1, characterized in that, Step S6 specifically includes: The at least two finely tuned face recognition models, which have been trained under self-supervised supervision, are used to extract feature vectors from the input images. The extracted feature vectors are then used as teacher feature vectors to guide the training of the student neural network. The student neural network is used to extract feature vectors from the input image to obtain student feature vectors; Fit the student feature vector to the teacher feature vector; The student neural network is used as the target face recognition model.

8. The face recognition method based on fast mask generation according to claim 7, characterized in that, When fitting the student feature vector to the teacher feature vector, the loss function uses mean squared error.

9. A face recognition device based on fast mask generation, characterized in that, include: The preprocessing module is used to preprocess the publicly available dataset to obtain an initial training face image set; The initial face recognition model training module is used to train a convolutional neural network using the initial training face image set to obtain an initial face recognition model; The occlusion mask adding module is used to add occlusion masks to the initial training face image set with at least two occlusion probabilities to form occlusion regions for some faces in the initial training face image set, thereby obtaining at least two sets of training face image sets with added masks; when adding occlusion masks, random transparency is set and multiple colors are used for random rendering; The fine-tuning face recognition model training module is used to train the initial face recognition model using the at least two sets of masked training face image sets respectively, so as to obtain at least two fine-tuning face recognition models. The self-supervised training module is used to add an occlusion mask to each image in the initial training face image set to obtain a self-supervised training mask image set. The initial training face image set and the self-supervised training mask image set are used to perform self-supervised training on each fine-tuned face recognition model. Using the initial training face image set and the self-supervised training mask image set, self-supervised training is performed on each fine-tuned face recognition model, specifically including: S51, Construct a feature template queue, and randomly initialize the feature template queue during training initialization; S52, use the fine-tuned face recognition model to perform feature inference on an original image in the initial training face image set to obtain the corresponding feature Embedding1; S53, using a fine-tuned face recognition model, feature reasoning is performed on the corresponding mask image of the original image in the self-supervised training mask image set to obtain the corresponding feature Embedding2; S54, the result of multiplying the corresponding feature Embedding1 and the corresponding feature Embedding2 after normalization is output as the corresponding positive sample, and the label is recorded as 1; the result of multiplying the corresponding feature Embedding1 and the feature template queue is output as the corresponding negative sample, and the label is recorded as 0. S55, after connecting the corresponding positive samples and negative samples, a binary classification loss function is output to the loss layer, and the corresponding feature Embedding1 is used to cover the feature template queue in the queue order. After the feature template queue is completely covered, the covering is performed again from the beginning. S56, S52-S55 are executed iteratively and repeatedly to traverse all original images in the initial training face image set and complete the self-supervised training of the fine-tuned face recognition model; S57. For each fine-tuned face recognition model, execute steps S51-S56 to complete the self-supervised training of each fine-tuned face recognition model. The model fusion module is used to perform model fusion on the at least two fine-tuned face recognition models after self-supervised training using a distillation method to obtain the target face recognition model. A face recognition module is used to perform face recognition using the target face recognition model.

Images