Face recognition method and system, storage medium and electronic equipment

Abstract

The invention discloses a face recognition method and system, a storage medium and electronic equipment, and relates to the technical field of face recognition, shielded mask features are automatically extracted through a mask prediction subnet, the mask prediction subnet adopts a U-Net architecture, and an upper sampling layer of the U-Net architecture is suitable for feature value transmission. Through the design, the mask prediction subnet can effectively extract the features of the mask area, and shield mask information is provided for the subsequent face shield removal process, so that automation of the shield removal-face recognition process is completed, and dependence of a traditional shield removal mode on manual mask labeling is eliminated. The structure guiding subnet predicts a reasonable face structure, the face recognition subnet recognizes an output shielding-removed picture, the subnets cooperate together to finally recover a shielding face and perform identity recognition, and if an illegal sample exists in a recognition result, an alarm is given on an inner platform of a bank outlet, so that the safety of the bank outlet is improved. The purpose of improving security control of bank outlets is achieved.

Description

Technical Field

[0001] This application relates to the field of facial recognition technology, and more specifically, to a facial recognition method, system, storage medium, and electronic device. Background Technology

[0002] Because bank branches have high security requirements, they typically use facial detection and recognition technology to analyze the flow of people in order to achieve security monitoring of the bank branches.

[0003] However, with the widespread use of masks, when people enter bank branches wearing masks, their facial features are obscured, making it difficult for banks to recognize suspicious individuals and thus hindering the security management of bank branches.

[0004] Therefore, how to use facial recognition to identify people wearing masks when entering bank branches in order to improve the security management of bank branches is an urgent problem that this application needs to solve. Summary of the Invention

[0005] In view of this, this application discloses a face recognition method, system, storage medium, and electronic device, which aims to restore the occluded face and perform identity recognition through the joint cooperation of a mask prediction subnet, a structure guidance subnet, a face de-occlusion subnet, and a face recognition subnet. If the recognition result is an illegal sample recognition result, an alarm will be issued on the internal platform of the bank branch, so as to achieve the purpose of security control of the bank branch.

[0006] To achieve the above objectives, the disclosed technical solution is as follows:

[0007] The first aspect of this application discloses a face recognition method, the method comprising:

[0008] Acquire facial images of masked individuals entering a preset area;

[0009] The mask prediction subnet is used to predict the occlusion area of ​​the masked face image to obtain a binary mask code.

[0010] Facial structure guidance is performed on the masked face image using a structure-guided subnet to obtain a sequence of facial key points.

[0011] The binary mask mask and facial key point sequence are feature-concatenated through the upsampling layer of the mask prediction subnet;

[0012] The occlusion perception layer in the face de-occlusion subnet performs soft activation masking on the boundaries of the stitched features to obtain a de-occluded face image.

[0013] The face feature recognition is performed on the de-occluded face image through a face recognition subnet to obtain the face feature value to be compared;

[0014] The facial feature values ​​of the person to be compared are compared and identified using a preset comparison method to obtain the recognition result.

[0015] If the identification result indicates the presence of an illegal sample, an alarm will be triggered on the bank branch's internal platform.

[0016] Preferably, the training process for the mask prediction subnet includes:

[0017] The acquired masked face data was labeled using an open-source image annotation tool to construct a real mask-covered dataset.

[0018] A face masking tool based on computer vision and the acquired raw face dataset are used to construct a mask dataset; wherein, the raw face dataset is an unmasked face dataset without masks.

[0019] The mask prediction subnet built on the U-Net architecture is trained using a preset loss function, the real mask occlusion dataset, and the mask dataset.

[0020] Preferably, the training process of the structure-guided subnet includes:

[0021] The obtained raw face dataset is used as input and fed into a key point prediction model using a deep face analysis tool to generate real key points;

[0022] An occlusion image is generated using a computer vision-based facial masking tool and the real key points.

[0023] The MobileNet-V2 architecture is adopted as the baseline model for the structured bootstrap subnet;

[0024] The occluded image is used as input to a baseline model, which outputs a key point sequence.

[0025] The structured guided subnet built on the MobileNet-V2 architecture is trained using a preset loss function.

[0026] Preferably, the training process for the face recognition subnet includes:

[0027] The original face dataset is used as input, and the identity tags of different people are used as output, along with angular additive margin loss, to train the face recognition subnet.

[0028] Preferably, the step of applying a soft activation mask to the boundaries of the stitched features through the occlusion perception layer in the face de-occlusion sub-network to obtain the de-occluded face image includes:

[0029] The occlusion perception layer in the face de-occlusion subnet performs mask generation, soft mask activation, and mask convolution on the stitched features to complete the soft activation of the mask boundary, thus obtaining the de-occluded face image.

[0030] Preferably, the step of comparing and recognizing the facial feature values ​​of the person to be compared using a preset comparison method to obtain the recognition result includes:

[0031] The facial feature values ​​to be compared are compared and identified with the preset suspicious person feature values ​​in the database;

[0032] If the facial feature value to be compared matches the preset suspicious person feature value, an identification result indicating the presence of an illegal sample is obtained;

[0033] If the facial feature value to be compared is inconsistent with the preset suspicious person feature value, the identification result is that there is a legitimate sample.

[0034] A second aspect of this application discloses a face recognition system, the system comprising:

[0035] The first acquisition unit is used to acquire images of masked faces entering the preset area;

[0036] The prediction unit is used to predict the occlusion area of ​​the masked face image through the mask prediction subnet to obtain a binary mask code.

[0037] The guidance unit is used to guide the facial structure of the masked face image through the structure guidance subnet to obtain a sequence of facial key points;

[0038] The feature splicing unit is used to splice the binary mask mask and facial key point sequence through the upsampling layer of the mask prediction subnet;

[0039] The second acquisition unit is used to perform soft activation masking on the spliced ​​features through the occlusion perception layer in the face de-occlusion subnet to obtain a de-occluded face image.

[0040] A face feature recognition unit is used to perform face feature recognition on the de-occluded face image through a face recognition subnet to obtain the face feature value to be compared.

[0041] The comparison and recognition unit is used to compare and recognize the facial feature values ​​of the person to be compared using a preset comparison method, and to obtain the recognition result.

[0042] An alarm unit is used to issue an alarm on the bank branch's internal platform if the identification result indicates the presence of an illegal sample.

[0043] Preferably, the prediction unit in the training process of the mask prediction subnet includes:

[0044] The first building module is used to label the acquired masked face data using open-source image annotation tools in order to build a real mask occlusion dataset.

[0045] The second construction module is used to construct a mask dataset based on a computer vision-based face masking tool and the acquired original face dataset; wherein, the original face dataset is an unmasked face dataset without masks;

[0046] The training module is used to train the mask prediction subnet built on the U-Net architecture using a preset loss function, the real mask occlusion dataset, and the mask dataset.

[0047] A third aspect of this application discloses a storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium is located is controlled to perform a face recognition method as described in any one of the first aspects.

[0048] The fourth aspect of this application discloses an electronic device including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any of the first aspects.

[0049] As can be seen from the above technical solution, this application discloses a face recognition method, system, storage medium, and electronic device. The method involves acquiring a masked face image within a preset area, predicting the occlusion area of ​​the masked face image using a mask prediction subnet to obtain a binary mask, guiding the facial structure of the masked face image using a structure guidance subnet to obtain a facial key point sequence, concatenating the binary mask and facial key point sequence using an upsampling layer in the mask prediction subnet, applying soft activation mask boundaries to the concatenated features using an occlusion perception layer in the face de-occlusion subnet to obtain a de-occluded face image, performing facial feature recognition on the de-occluded face image using a face recognition subnet to obtain the face feature value to be compared, comparing and recognizing the face feature value to be compared using a preset comparison method to obtain the recognition result, and issuing an alarm on the bank branch's internal platform if the recognition result indicates the presence of an illegal sample.

[0050] The beneficial effects of this application are as follows: This solution eliminates the need for manually labeled occlusion mask data for face recognition in bank branches, reducing data preparation costs and time. It automatically extracts occlusion mask features by training a mask prediction subnet using a self-made dataset. The mask prediction subnet adopts a U-Net architecture, and the upsampling layer of U-Net is suitable for feature value transfer. The input of this mask prediction subnet is the original image with the mask on, and the output is a binary mask mask. This design enables the mask prediction subnet to effectively extract features of the mask region, providing occlusion mask information for the subsequent face de-occlusion process, thereby automating the de-occlusion and face recognition process in bank branches and eliminating the reliance on manually labeled masks in traditional de-occlusion methods. Furthermore, the structure guidance subnet predicts reasonable facial structures, and finally the face recognition subnet identifies the output de-occluded image. Through the joint collaboration of the mask prediction subnet, structure guidance subnet, face de-occlusion subnet, and face recognition subnet, the occluded face is finally restored and identity is recognized. If the recognition result is that there is an illegal sample, an alarm is triggered on the internal platform of the bank branch to improve the security management of the bank branch. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating a face recognition method disclosed in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of a portion of the convolutional and occlusion sensing layers disclosed in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of the structure of a face recognition device disclosed in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the structure of a face recognition system disclosed in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the structure of the electronic device disclosed in the embodiments of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] As can be seen from the background technology, when people entering bank branches wear masks, their facial features are obscured, which is not conducive to the bank branch's facial recognition of suspicious persons, and thus is detrimental to the security management of the bank branch.

[0060] To address the aforementioned issues, this application discloses a face recognition method, system, storage medium, and electronic device. This solution eliminates the need for manually labeled occlusion mask data for face recognition at bank branches, reducing data preparation costs and time. It automatically extracts occlusion mask features by training a mask prediction subnet using a self-made dataset. The mask prediction subnet employs a U-Net architecture, and the upsampling layer of U-Net is suitable for feature value transfer. The input to this subnet is the original image with the mask on, and the output is a binary mask mask. This design enables the mask prediction subnet to effectively extract features from the mask region, providing occlusion mask information for the subsequent face de-occlusion process. This automates the de-occlusion and face recognition process at bank branches, eliminating the reliance on manually labeled masks in traditional de-occlusion methods. Furthermore, a structure-guided subnet predicts reasonable facial structures, and a face recognition subnet identifies the output occluded image. Through the collaborative efforts of the mask prediction subnet, structure-guided subnet, face occlusion subnet, and face recognition subnet, the occluded face is finally restored and identity is recognized. If the recognition result indicates the presence of an illegal sample, an alarm is triggered on the bank branch's internal platform to enhance the bank branch's security management. The specific implementation method is described in detail in the following embodiments.

[0061] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0062] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0063] refer to Figure 1 The image shows a face recognition method disclosed in an embodiment of this application. The face recognition method mainly includes the following steps:

[0064] S101: Obtain facial images of people wearing masks within a preset area of ​​a bank branch.

[0065] The preset area for bank branches refers to the area covered by monitoring equipment set up according to the actual situation of the bank branches.

[0066] S102: The mask prediction subnet is used to predict the occlusion area of ​​the masked face image to obtain the binary mask code.

[0067] It should be noted that the mask prediction subnet is a key module for automated mask removal in this solution. The mask prediction subnet is built on the U-Net architecture and is used to automatically predict the occlusion area and generate the mask occlusion mask, avoiding the tedious process of manual annotation.

[0068] Due to the irregular shape and color of face masks, a U-Net architecture was adopted to train a face mask semantic segmentation model based on a self-built dataset. To facilitate the transfer of mask information, i.e., the mask occlusion mask, between the face demasking subnet and the mask prediction subnet, the output of the upsampling layer of the mask prediction subnet was layered and connected to the mask demasking layer. This mitigates the blurring caused by unclear recovery boundary predictions during the restoration process. The mask demasking layer is set within the mask prediction subnet.

[0069] The mask prediction subnet and its connection structure with the occlusion sensing layer:

[0070] The mask prediction subnet adopts the U-Net architecture, and the upsampling layer of U-Net is well-suited for feature transfer. The input of this subnet is the original image with the mask on, and the output is a binary mask mask. Through this design, the mask prediction subnet can effectively extract features of the mask region, providing occlusion mask information for the subsequent face de-occlusion process, and automating the de-occlusion-face recognition process.

[0071] Design of the occlusion perception layer:

[0072] The occlusion perception layer is layered and connected to the mask de-occlusion layer based on the output of the upsampling layer of U-Net to mitigate the blurring caused by unclear boundary predictions during the recovery process. Through this design, the mask prediction subnet can specifically adapt to the mask mask with bias in this scenario, overcoming the hard 0-1 encoding problem of traditional de-occlusion methods.

[0073] The training process of the mask prediction subnet is shown in A1-A3:

[0074] A1: The obtained masked face data was labeled using the open-source image annotation tool (LabelMe) to construct a real mask occlusion dataset.

[0075] This includes open-source image annotation tools such as LabelMe. This application does not specifically limit the types of open-source image annotation tools.

[0076] Based on the masked face dataset, LabelMe can be used to construct a real mask-occluded dataset.

[0077] A2: Construct a mask dataset using a computer vision-based face masking tool and the obtained raw face dataset; wherein, the raw face dataset is an unmasked face dataset without masks.

[0078] While using LabelMe to label and construct a real mask occlusion dataset based on the masked face dataset, a synthetic mask dataset is constructed based on open-source computer vision-based face masking tools (such as MaskTheFace) and the original face dataset. The mask dataset includes RGB images in color mode with masks on and binary mask images, etc.

[0079] A3: The mask prediction subnet built on the U-Net architecture is trained using a preset loss function, a real mask occlusion dataset, and a mask dataset.

[0080] The preset loss function can be the L1 Loss function, etc., and this application does not specifically limit the type of preset loss function.

[0081] Model training: The input of the model is the original image with a mask, and the output is a binary mask image, i.e., a binary mask image. L1 Loss is used as the loss function. The model is trained by combining the real dataset and the synthetic dataset. There is a loss value. The expression of the loss value is shown in formula (1).

[0082] (1)

[0083] in, To train the loss value for the mask prediction subnet; For predicted mask codes; This is the actual mask cover.

[0084] S103: Facial structure guidance is performed on the masked face image through the structure guidance subnet to obtain the facial key point sequence.

[0085] The structure-guided subnet utilizes facial key point information for structure-guided de-occlusion, providing facial interface information to the face de-occlusion subnet to ensure a natural restored face. Key point detection in this application is based on the occluded face.

[0086] The structure-guided subnet is built on the MobileNet-V2 architecture.

[0087] Keypoint detection is a crucial task in computer vision, aiming to detect and locate specific important points or regions in an image. Keypoints typically refer to the locations of key facial features, such as the center points of the eyes, nose, and mouth, or key facial contour information. Networks can utilize facial geometric information to guide the image restoration process, ensuring that the restored facial image has reasonable proportions and a natural appearance.

[0088] Structure-guided inpainting is an image restoration method based on prior knowledge. It mainly utilizes structural information in the image, such as edges, shapes, and key points, to guide the restoration process. In this application, structure-guided inpainting uses a facial key point prediction network to guide the restoration of facial images, i.e., facial de-occlusion, ensuring that the restored face has a reasonable geometric structure, thereby improving the naturalness and realism of the de-occluded face.

[0089] Facial de-occlusion refers to restoring the facial image of the occluded area, enabling the image to recover complete facial feature information. This is achieved through image inpainting techniques, with the aim of restoring facial details in the occluded area to their true, unoccluded state as accurately as possible.

[0090] The training process of the structure-guided subnet is shown in B1-B5.

[0091] B1: The acquired native face dataset is used as input and fed into a keypoint prediction model using an open-source deep face analysis tool to generate real keypoints.

[0092] Open-source deep face analysis tools include, but are not limited to, InsightFace.

[0093] In B1, the raw face dataset without masks is used as input during dataset construction, while InsightFace's keypoint prediction model is used to generate ground truth keypoints.

[0094] B2: Generate occluded images using open-source computer vision face masking tools and real key points.

[0095] Computer vision face masking tools include, but are not limited to, the MaskTheFace tool.

[0096] B3: The MobileNet-V2 architecture is adopted as the baseline model for the structured bootstrap subnet.

[0097] B4: Take the occluded image as input and feed it into the baseline model (MobileNet-V2) to output the keypoint sequence through the baseline model (MobileNet-V2).

[0098] During model training, MobileNet-V2 was used as the baseline model for the structure-guided subnet. The model input was the MaskTheFace occlusion image, and the output was the key point sequence.

[0099] B5: Train the structured guided subnet built on the MobileNet-V2 architecture using a preset loss function.

[0100] L1 Loss can be used as the loss function to train the structured guided subnet built on the MobileNet-V2 architecture. During the training process, there is a loss value, the expression of which is shown in Equation (2).

[0101] (2)

[0102] in, To train the loss value for the mask prediction subnet; The coordinates of the key points to be predicted; These are the actual coordinates of the key points.

[0103] S104: The binary mask mask and facial key point sequence are spliced ​​together by upsampling layer of mask prediction subnet.

[0104] The output of the upsampling layer of the mask prediction subnet is layered and connected to the mask demasking layer to mitigate the ambiguity caused by unclear recovery boundary prediction during the recovery process.

[0105] By upsampling the mask prediction subnet, for example, by combining the intermediate results of the mask prediction subnet (32*32, 64*64, 128*128) with the facial key point sequence, the information is gained and superimposed, and the gain is applied to the face de-occlusion subnet.

[0106] S105: The occlusion perception layer in the face de-occlusion subnet performs soft activation masking on the boundaries of the stitched features to obtain the de-occluded face image.

[0107] In S105, the occlusion perception layer in the face de-occlusion subnet performs mask generation, soft 0-1 mask activation, and mask convolution on the stitched features to complete the soft activation of the mask boundary and obtain the de-occluded face image.

[0108] The facial de-occlusion subnet is the core of this application's structure, aiming to output a high-quality image after occlusion removal by combining occlusion masks and facial structure information.

[0109] The face recognition subnet is built on the U-Net architecture, and its implementation involves two steps:

[0110] 1. Dataset construction: Raw faces without masks are used as baseline data. Based on the raw faces, masked face images are generated using the MaskTheFace tool. At the same time, a mask prediction subnet is used to generate mask features for each image, and a structure-guided subnet is used to generate key point features for each image.

[0111] 2. Model training: The model input consists of structural features, mask features, and mask images. The output is the original image of the user without a mask. The total loss function of the network is formed by combining pixel reconstruction loss, style loss, adversarial loss, and smoothness loss.

[0112] Image restoration techniques rely on partial convolution operations to correctly handle occluded areas in an image without affecting unoccluded areas. Partial convolutional layers consider missing regions in the input during convolution calculations, using a weighted average of effective pixel values ​​within local regions to avoid the adverse effects of missing regions on the convolution result. For general image restoration networks, partial convolution can effectively restore images.

[0113] In application scenarios, since some convolutions rely on accurate mask input, the mask results output by the mask prediction subnet will inevitably be inaccurate. This error will penetrate deeper into the network layer by layer in the face demasking subnet, causing distortion in the final recovered image.

[0114] To address the issues associated with hard 0-1 masks and mask update rules, and to improve the robustness of the face de-occlusion subnet, this application proposes an occlusion-aware layer for the boundary of the soft activation mask. The occlusion-aware layer comprises three operations: mask generation, soft 0-1 mask activation, and mask convolution, implemented as follows: Figure 2 and as shown below:

[0115] 1. Mask Generation: Assuming the mask prediction subnet has L layers, its feature map from layer L to layer 1 is represented as follows: The face demasking subnet also contains L layers, and the corresponding feature map is represented as follows: At this point, the mask of the i-th layer of the face demasking subnet is... The value does not depend on the update result of the previous layer mask, but is directly obtained from the corresponding layer of the mask prediction subnet, specifically the mask. The expression for the value is shown in formula (3).

[0116] (3)

[0117] in, The mask value for the i-th layer of the face demasking subnet; It is a one-dimensional convolution with softmax.

[0118] This means that the features of the i-th layer of the occlusion perception layer are directly encapsulated using 1D convolutions and Softmax layers. In this way, there is no need to update the mask information at each layer.

[0119] 2. Soft 0-1 mask activation: The mask is activated using a learnable Gaussian function, and the specific expression is shown in formula (4).

[0120] (4)

[0121] in, For the mask perception layer; , , , All are learnable parameters; A fixed-value convolution kernel is used to smooth the convolution. Horizontal value, The target value of 1*9 is used for local smoothing operations.

[0122] 3. Masked Convolution: Finally, the mask from the mask perception layer is used. Perform partial convolution operations to obtain the feature map of the i-th layer of the face de-occlusion subnet.

[0123] Partial convolution is a type of convolution operation specifically designed for processing images with irregularly occluded regions. In traditional convolution, the kernel operates on the entire image, while partial convolution only convolves the valid (unoccluded) pixels, thus avoiding the influence of occluded parts on the convolution result.

[0124] S106: Perform facial feature recognition on the de-occluded face image through the face recognition subnet to obtain the facial feature value to be compared.

[0125] The original face dataset is used as input, and the identity tags of different people are used as output, along with angular additive margin loss, to train the face recognition subnet.

[0126] The face recognition subnet is the core of this application, and this application uses arcface from the open-source InsightFace architecture as the baseline model for the face recognition subnet.

[0127] The face recognition subnet directly uses an existing unoccluded face recognition dataset as the training dataset. During model training, the original image is used as input, and the identity labels of different people are used as the comparison benchmark for the output. The model is trained using angular additive margin loss. The specific expression of angular additive margin loss is shown in Equation (5).

[0128] (5)

[0129] Where C is the number of categories; N is the number of samples; s is the scaling factor; and m is the angular margin. is the angle between the feature vector of the i-th sample and the corresponding class weight vector; cos() is the cosine similarity between the feature vector and the weight vector.

[0130] By utilizing the cooperation of four modules—mask prediction subnetwork, structure guidance subnetwork, face de-occlusion subnetwork, and face recognition subnetwork—this application restores the input masked face image into a de-occluded face image, preserving the original identity information as much as possible during the process, and finally using a conventional face recognition model for identity determination.

[0131] S107: The facial feature values ​​of the person to be compared are compared and identified using a preset comparison method to obtain the recognition result.

[0132] Specifically, the process of comparing and recognizing the facial feature values ​​of the person to be compared using a preset comparison method to obtain the recognition result is shown in C1-C3.

[0133] C1: Compare and identify the facial feature values ​​of the person to be compared with the preset suspicious person feature values ​​in the database.

[0134] The preset suspicious person characteristic values ​​in the database are the suspicious person characteristic values ​​in the public security alarm list database.

[0135] C2: If the facial feature value to be compared matches the preset suspicious person feature value, the result of identification of illegal sample is obtained.

[0136] C3: If the facial feature value to be compared is inconsistent with the preset suspicious person feature value, the identification result of the existence of a legitimate sample is obtained.

[0137] S108: If the identification result indicates the presence of an illegal sample, an alarm will be triggered on the bank branch's internal platform.

[0138] Bank branches have high security requirements and often use face detection and recognition technology to track and analyze customer flow, including facial recognition of bank employees and suspicious individuals. However, with the widespread use of masks, wearing masks obscures facial features, hindering identity verification, security monitoring, and personnel management in bank branches. To address this, some masked face recognition methods have improved their adaptability to mask-wearing by training on dedicated masked face datasets. However, these datasets typically require both masked and unmasked faces as input, making data collection difficult, and the models may not be robust enough to specific occlusion patterns. Furthermore, in extreme cases, large occlusion areas or complex mask shapes can impede recognition. To address these issues, this solution avoids the use of dedicated masked face datasets. The system employs a convolutional architecture-based de-occlusion method to recover unmasked faces for facial recognition, monitoring customer flow at key locations within the branch.

[0139] To facilitate understanding of the facial recognition process in this application, combined with Figure 3 Please provide an explanation. Figure 3 A schematic diagram of the face recognition device is shown.

[0140] Figure 3 The face recognition device comprises a mask prediction subnetwork, a structure guidance subnetwork, a face de-occlusion subnetwork, and a face recognition subnetwork. The occlusion perception subnetwork is trained using a self-made dataset to automatically extract mask features, eliminating the reliance on manually labeled masks in traditional de-occlusion methods. The structure guidance subnetwork predicts reasonable facial structures, and the face recognition subnetwork identifies the output de-occluded image. These four modules work together to reconstruct the occluded face and determine identity. This application can predict heavily occluded regions in complex scenes, improving the accuracy of de-occlusion and face recognition, and possesses high computational efficiency.

[0141] This application aims to address the problems of reliance on manual annotation and imprecise restoration in existing facial de-occlusion technologies, and ultimately to identify individuals using the de-occluded images. The proposed bank branch mask-wearing facial recognition system based on de-occlusion technology is described as follows: Monitoring images are acquired from a camera port. A face detection model monitors each frame. If a face is detected, it is cropped and placed in a buffer. The mask-wearing facial recognition model iteratively de-occludes (if the mask prediction model can predict the mask) and performs face recognition on the latest images in the buffer. The recognition results are compared with facial features in the database. If the recognition result is an illegal sample, such as if the result is not in the system's configured face database or is on the public security alarm list, it is archived and reported to the police through the bank's internal platform. This application eliminates the dependence of traditional de-occlusion methods on occlusion region masks by training a mask prediction subnet in a multi-subnet structure, simplifying the application process. At the same time, it can guide the face reconstruction process based on the facial geometric information output by the structure prediction subnet, which can retain more identity information for downstream face recognition tasks and improve the accuracy of the face detection process.

[0142] This application reduces the cost of face removal in masked scenarios: It eliminates the need for manually labeled occlusion mask data, significantly reducing data preparation costs and time. Traditional deep learning models often require large amounts of manually labeled data, while this application solves this problem through automated mask generation, making training data acquisition more efficient and reducing the overall cost of data preparation.

[0143] The occlusion removal method of this application preserves more identity features for face recognition: Traditional face recognition models use cropped facial parts (usually the eyes) for model training, which results in the loss of the ability to distinguish other facial features during training. The ability of this invention to preserve facial identity features is reflected in two aspects: 1. It uses a structure-guided architecture to guide the reconstruction work, which makes it easier to preserve the structural features of the face during the reconstruction process; 2. It uses a convolution-based rather than adversarial generative architecture for occlusion removal, which can use pixel reconstruction loss for training during reconstruction, thus preserving more identity features for the model.

[0144] The occlusion perception layer in this application adapts to inaccurate mask prediction results: Traditional partial convolution operations suffer from hard 0-1 convolutional encoding, leading to inaccurate prediction errors propagating and accumulating layer by layer, ultimately causing boundary distortion in the prediction results. This application addresses this by designing an occlusion perception layer that uses soft 0-1 encoding to connect the outputs of each layer of the mask prediction subnet, enabling the model to adapt to fluctuations in the mask prediction subnet's prediction results.

[0145] The beneficial effects of the method embodiments of this application are as follows: This solution eliminates the need for manually labeled occlusion mask data for face recognition in bank branches, reducing the cost and time of data preparation. It automatically extracts occlusion mask features by training a mask prediction subnet using a self-made dataset. The mask prediction subnet adopts a U-Net architecture, and the upsampling layer of U-Net is suitable for feature value transfer. The input of this mask prediction subnet is the original image with the mask on, and the output is a binary mask mask. This design enables the mask prediction subnet to effectively extract features of the mask region, providing occlusion mask information for the subsequent face de-occlusion process, thereby automating the de-occlusion-face recognition process in bank branches and eliminating the reliance on manually labeled masks in traditional de-occlusion methods. Furthermore, the structure guidance subnet predicts reasonable facial structures, and finally the face recognition subnet identifies the output de-occluded image. Through the joint collaboration of the mask prediction subnet, structure guidance subnet, face de-occlusion subnet, and face recognition subnet, the occluded face is finally restored and identity is recognized. If the recognition result is that there is an illegal sample, an alarm is triggered on the internal platform of the bank branch to improve the security management of the bank branch.

[0146] Based on the above embodiments Figure 1 This application discloses a face recognition method and, in its embodiments, also discloses a face recognition system, such as... Figure 4 As shown, the facial recognition system includes:

[0147] The first acquisition unit 401 is used to acquire images of masked faces entering the preset area;

[0148] Prediction unit 402 is used to predict the occlusion area of ​​a masked face image through the mask prediction subnet to obtain a binary mask code.

[0149] The guiding unit 403 is used to guide the facial structure of the masked face image through the structure guiding subnet to obtain the facial key point sequence;

[0150] Feature splicing unit 404 is used to splice the binary mask mask and facial key point sequence through the upsampling layer of the mask prediction subnet;

[0151] The second acquisition unit 405 is used to perform soft activation masking on the spliced ​​features through the occlusion perception layer in the face de-occlusion subnet to obtain a de-occluded face image.

[0152] The face feature recognition unit 406 is used to perform face feature recognition on the de-occluded face image through the face recognition subnet to obtain the face feature value to be compared;

[0153] The comparison and recognition unit 407 is used to compare and recognize the facial feature values ​​of the person to be compared using a preset comparison method, and obtain the recognition result.

[0154] Alarm unit 408 is used to issue an alarm on the internal platform of the bank branch if the identification result is that there is an illegal sample.

[0155] Furthermore, the prediction units in the training process of the mask prediction subnet include:

[0156] The first building module is used to label the acquired masked face data using open-source image annotation tools in order to build a real mask occlusion dataset.

[0157] The second building module is used to construct a mask dataset from a computer vision-based face masking tool and the acquired original face dataset; wherein, the original face dataset is an unmasked face dataset without masks.

[0158] The training module is used to train the mask prediction subnet built on the U-Net architecture using a preset loss function, a real mask occlusion dataset, and a mask dataset.

[0159] Furthermore, the guiding unit 403 for the training process of the structure-guided subnet includes:

[0160] The first generation module is used to take the acquired raw face dataset as input and feed it into the key point prediction model through deep face analysis tools to generate real key points;

[0161] The second generation module is used to generate occluded images using computer vision facial masking tools and real key points;

[0162] The module is used to determine the baseline model for using the MobileNet-V2 architecture as the structure-guided subnet;

[0163] The output module is used to take the occluded image as input and input it into the benchmark model so that the benchmark model can output a key point sequence.

[0164] The training module is used to train the structured training subnet built on the MobileNet-V2 architecture using a preset loss function.

[0165] Furthermore, the face feature recognition unit 406 in the training process of the face recognition subnet is specifically used to train the face recognition subnet by taking the acquired original face dataset as input, using the identity tags of different people as output, and using angular additive margin loss.

[0166] Furthermore, the second acquisition unit 405 is specifically used to perform mask generation, soft mask activation and mask convolution on the stitched features through the occlusion perception layer in the face de-occlusion subnet, so as to complete the process of soft activation mask boundary and obtain the de-occluded face image.

[0167] Furthermore, the comparison and identification unit 407 includes:

[0168] The comparison and recognition module is used to compare and recognize the facial feature values ​​of the person to be compared with the preset suspicious person feature values ​​in the database.

[0169] The first acquisition module is used to obtain the identification result of the presence of illegal samples if the facial feature value to be compared is consistent with the preset suspicious person feature value;

[0170] The second acquisition module is used to obtain the recognition result of the existence of a legitimate sample if the facial feature value to be compared is inconsistent with the preset suspicious person feature value.

[0171] The beneficial effects of the system embodiments in this application are as follows: This solution eliminates the need for manually labeled occlusion mask data for face recognition at bank branches, reducing the cost and time of data preparation. It automatically extracts occlusion mask features by training a mask prediction subnet using a self-made dataset. The mask prediction subnet adopts a U-Net architecture, and the upsampling layer of U-Net is suitable for feature value transfer. The input of this mask prediction subnet is the original image with the mask on, and the output is a binary mask mask. This design enables the mask prediction subnet to effectively extract features of the mask region, providing occlusion mask information for the subsequent face de-occlusion process, thereby automating the de-occlusion-face recognition process at bank branches and eliminating the reliance on manually labeled masks in traditional de-occlusion methods. Furthermore, the structure guidance subnet predicts reasonable facial structures, and finally the face recognition subnet identifies the output de-occluded image. Through the joint collaboration of the mask prediction subnet, structure guidance subnet, face de-occlusion subnet, and face recognition subnet, the occluded face is finally restored and identity is recognized. If the recognition result is that there is an illegal sample, an alarm is triggered on the internal platform of the bank branch to improve the security management of the bank branch.

[0172] This application embodiment also provides a storage medium, the storage medium including stored instructions, wherein, when the instructions are executed, the device where the storage medium is located is controlled to perform the face recognition method described above.

[0173] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein one or more instructions 502 are stored in the memory 501 and configured to be executed by one or more processors 503 to perform the above-mentioned face recognition method.

[0174] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0175] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0176] The steps in the methods of the various embodiments of this application can be adjusted, combined, or deleted according to actual needs.

[0177] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0179] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A face recognition method, characterized in that, The method includes: Acquire facial images of masked individuals entering a preset area; The mask prediction subnet is used to predict the occlusion area of ​​the masked face image to obtain a binary mask code. Facial structure guidance is performed on the masked face image using a structure-guided subnet to obtain a sequence of facial key points. The binary mask mask and facial key point sequence are feature-concatenated through the upsampling layer of the mask prediction subnet; The occlusion perception layer in the face de-occlusion subnet performs soft activation masking on the boundaries of the stitched features to obtain a de-occluded face image. The face feature recognition is performed on the de-occluded face image through a face recognition subnet to obtain the face feature value to be compared; The facial feature values ​​of the person to be compared are compared and identified using a preset comparison method to obtain the recognition result. If the identification result indicates the presence of an illegal sample, an alarm will be triggered on the bank branch's internal platform.

2. The method according to claim 1, characterized in that, The training process of the mask prediction subnet includes: The acquired masked face data was labeled using an open-source image annotation tool to construct a real mask-covered dataset. A face masking tool based on computer vision and the acquired raw face dataset are used to construct a mask dataset; wherein, the raw face dataset is an unmasked face dataset without masks. The mask prediction subnet built on the U-Net architecture is trained using a preset loss function, the real mask occlusion dataset, and the mask dataset.

3. The method according to claim 1, characterized in that, The training process of the structure-guided subnet includes: The obtained raw face dataset is used as input and fed into a key point prediction model using a deep face analysis tool to generate real key points; An occlusion image is generated using a computer vision-based facial masking tool and the real key points. The MobileNet-V2 architecture is adopted as the baseline model for the structured bootstrap subnet; The occluded image is used as input to a baseline model, which outputs a key point sequence. The structured guided subnet built on the MobileNet-V2 architecture is trained using a preset loss function.

4. The method according to claim 1, characterized in that, The training process for the face recognition subnet includes: The original face dataset is used as input, and the identity tags of different people are used as output, along with angular additive margin loss, to train the face recognition subnet.

5. The method according to claim 1, characterized in that, The process of applying a soft activation mask to the boundaries of the stitched features through the occlusion perception layer in the face de-occlusion sub-network to obtain the de-occluded face image includes: The occlusion perception layer in the face de-occlusion subnet performs mask generation, soft mask activation, and mask convolution on the stitched features to complete the soft activation of the mask boundary, thus obtaining the de-occluded face image.

6. The method according to claim 1, characterized in that, The step of comparing and recognizing the facial feature values ​​of the person to be compared using a preset comparison method to obtain the recognition result includes: The facial feature values ​​to be compared are compared and identified with the preset suspicious person feature values ​​in the database; If the facial feature value to be compared matches the preset suspicious person feature value, an identification result indicating the presence of an illegal sample is obtained; If the facial feature value to be compared is inconsistent with the preset suspicious person feature value, the identification result is that there is a legitimate sample.

7. A face recognition system, characterized in that, The system includes: The first acquisition unit is used to acquire images of masked faces entering the preset area; The prediction unit is used to predict the occlusion area of ​​the masked face image through the mask prediction subnet to obtain a binary mask code. The guidance unit is used to guide the facial structure of the masked face image through the structure guidance subnet to obtain a sequence of facial key points; The feature splicing unit is used to splice the binary mask mask and facial key point sequence through the upsampling layer of the mask prediction subnet; The second acquisition unit is used to perform soft activation masking on the spliced ​​features through the occlusion perception layer in the face de-occlusion subnet to obtain a de-occluded face image. A face feature recognition unit is used to perform face feature recognition on the de-occluded face image through a face recognition subnet to obtain the face feature value to be compared. The comparison and recognition unit is used to compare and recognize the facial feature values ​​of the person to be compared using a preset comparison method, and to obtain the recognition result. An alarm unit is used to issue an alarm on the bank branch's internal platform if the identification result indicates the presence of an illegal sample.

8. The system according to claim 7, characterized in that, The prediction units in the training process of the mask prediction subnet include: The first building module is used to label the acquired masked face data using open-source image annotation tools in order to build a real mask occlusion dataset. The second construction module is used to construct a mask dataset based on a computer vision-based face masking tool and the acquired original face dataset; wherein, the original face dataset is an unmasked face dataset without masks; The training module is used to train the mask prediction subnet built on the U-Net architecture using a preset loss function, the real mask occlusion dataset, and the mask dataset.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein when the instructions are executed, the device containing the storage medium is controlled to perform the face recognition method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, It includes a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 6.

Images