Face image gender recognition method suitable for mask wearing state

Abstract

A face image gender recognition method suitable for mask wearing state, comprising a face detection network and a gender classification network, the face detection network adopts feature pyramid technology, fuses multi-scale information to extract features of the person image, adds supervision to the deformable convolution layer, and improves the key point regression loss function to enhance the detection ability of the network for the face wearing a mask, and the gender classification network enhances the generalization ability of the network by difficult sample training, maps the input face image to the Euclidean space to obtain face embedding features, and outputs the gender recognition result through the classification network layer, so that the face image wearing a mask can be detected while ensuring a certain accuracy and meeting the detection speed requirement.

Description

Technical Field

[0001] This invention belongs to the field of facial recognition technology and relates to the classification of facial image attributes, providing a method for gender recognition of facial images suitable for those wearing masks. Background Technology

[0002] Gender recognition is the process of detecting a face in an image, extracting its image features, and identifying its gender attribute. This technology is mainly used in the field of automatic identity authentication, and therefore has high accuracy requirements. Currently, mainstream gender recognition methods are primarily based on cascaded deep learning networks, where the face detection network provides functions such as face localization and face alignment, while the attribute classification network provides functions such as feature extraction and gender recognition. Because face detection and classification network technologies have reached a high level of development, the above-mentioned deep learning-based gender recognition methods have been widely used in the field of identity authentication due to their high accuracy and fast inference speed.

[0003] However, with the significant increase in the number of pedestrians wearing masks in public places, especially in situations requiring rapid identification where removing masks is inconvenient, the accuracy of existing face detection methods is greatly affected. Furthermore, while some networks exist specifically designed to detect faces wearing masks, they can only detect faces wearing ordinary medical masks, which is incompatible with the diverse styles of masks available in everyday life. Simultaneously, mask occlusion also drastically reduces the accuracy of gender classification networks, posing challenges to gender identification. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a face image gender recognition method suitable for people wearing masks, which improves the accuracy of face detection network and gender classification network, and can better identify the gender of people wearing masks.

[0005] The technical solution of the present invention is: a method for gender recognition of facial images suitable for mask-wearing states, comprising the following steps:

[0006] Step 1: Construct a set of human image samples, including a training set and a test set: In the training set, people in the images are not wearing masks, and facial location annotations, i.e., the coordinates of the facial target boxes, are added to the images. In the test set, people in the images are wearing masks, and the number of training samples is ten times or more than that of the test set.

[0007] Step 2: Preprocessing of the training sample set of human images: Based on the face position annotations in the training sample set, the face images are cropped from the human images, and images of different styles of masks are collected. For each face image, one mask image is randomly selected. The 2DASL algorithm for face 3D reconstruction and dense alignment is used to align the key points of the face image and the mask image, thereby putting a simulated mask on the face.

[0008] Step 3: Construct a face detection network model for people wearing masks: Construct a face detection network based on the RetinaFace network, add supervision of deformable convolutional layers, and increase the proportion of binocular keypoint regression loss in all losses to enhance the network's ability to detect faces wearing masks.

[0009] Step 4: The face detection network is trained and tested using the training sample set and the test sample set of human images respectively. The output of the face detection network is the coordinates of the face target box. Based on the coordinates of the face target box, the face image is cropped from the image to construct the face image sample set, which is used for training and testing of the gender classification network.

[0010] Step 5: Construct and train the gender classification network model: Based on the FaceNet network, construct a gender classification network, add a classification layer, design triplet sample generation rules, and train with hard samples to enhance the network's generalization ability. The input to the gender classification network is a face image. The network completes the mapping, obtains the face embedding features, and outputs the person's gender, as detailed below:

[0011] Step 5.1: Add a classification layer at the end of the FaceNet network to build a gender classification network. The classification layer includes one convolutional layer, one pooling layer and one fully connected layer, which converts the 128-dimensional face embedding features output by the FaceNet network into gender classification results.

[0012] Step 5.2: Divide the face image sample set into m batches. For each batch, randomly select one image as a template. Then, select one image from the same sex as a positive example and randomly select one image from the opposite sex as a negative example. The template, positive example, and negative example together form a triplet sample. Repeat the above operation n times to initially build the training set and perform initial training and testing on the FaceNet network.

[0013] Step 5.3: After initial training and testing, for each face image, the FaceNet network obtains its 128-dimensional face embedding features. For each batch, the Euclidean distance between the face embedding features corresponding to all face images is calculated. One image is randomly selected as a template. Then, the same-sex image with the largest Euclidean distance is selected as a positive example, and the opposite-sex image with the smallest Euclidean distance is selected as a negative example. The template, positive example, and negative example together form a triplet hard sample. The above operation is repeated n times to reconstruct the training set and train the gender classification network. The training with hard samples enhances the generalization ability of the network.

[0014] Finally, the face detection network and gender classification network trained by the above steps perform face detection and gender classification on the input face image of a person wearing a mask, thus completing gender recognition.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] First, this invention improves the accuracy of face detection when faces are wearing masks. Current online datasets of faces wearing masks contain only a limited variety of mask styles, which doesn't match the diverse mask styles available in real-life situations. This results in face detection algorithms achieving high accuracy on online datasets but lower accuracy in practical applications. This invention uses the 2DASL algorithm to align key points between face images and mask images, thereby creating a sample set of face images simulating mask wearing. This dataset is then incorporated into the training process of the face detection network, enabling the network to learn most mask features. With the same network structure, the face detection accuracy is increased from 80.75% to 90.11%.

[0017] Secondly, this invention proposes an improved face detection network. Supervision of the deformable convolutional layer is added, allowing pseudo-boundaries to act as anchor points, thus improving the network's detection accuracy. Among all facial key points, due to mask occlusion, only the left and right eye key points are accurately located. Therefore, the loss function is improved to increase the contribution of binocular key point regression loss, improving face detection accuracy by 1.24%.

[0018] Third, this invention redesigned the triplet sample generation rules of the FaceNet network, enhanced the network's generalization ability by training with hard samples, and added a gender classification layer, which improved the gender classification accuracy from 90.15% to 93.5%. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the effect of the method for simulating wearing a mask on a human face according to the present invention.

[0021] Figure 3 This is a network structure diagram of the face detection network in this invention.

[0022] Figure 4 This is a schematic diagram illustrating the principle of training difficult samples in the gender classification network of this invention. Detailed Implementation

[0023] This invention proposes a method for gender recognition of facial images suitable for use while wearing a mask. Addressing the needs of applications requiring identity verification where removing masks is inconvenient, this invention solves the problem of accurately identifying gender when the face is obscured by a mask, effectively improving the accuracy of current gender recognition methods.

[0024] like Figure 1 As shown, the specific implementation process of the present invention is as follows:

[0025] Step 1: Construct a set of human image samples, including a training sample set and a test sample set: In the training sample set, people in the images are not wearing masks, and facial location annotations, i.e., the coordinates of the facial target boxes, are added to the images. In the test sample set, people in the images are wearing masks, and the number of training samples is ten times or more than that of the test sample set.

[0026] Step 2: Preprocessing of the training sample set of human images: Based on the face position annotations in the training sample set, the face images are cropped from the human images, and images of different styles of masks are collected. For each face image, one mask image is randomly selected. The 2DASL algorithm for face 3D reconstruction and dense alignment is used to align the key points of the face image and the mask image, thereby putting a simulated mask on the face, as follows.

[0027] Step 2.1: Use the 3D reconstruction and dense alignment algorithm 2DASL to reconstruct the face image using 3D point cloud to obtain the key point coordinates of the face image. Perform the same operation on the mask image to obtain the key point coordinates.

[0028] Step 2.2: By densely aligning facial landmarks and mask landmarks, the face image and mask image are aligned and superimposed to obtain a face image of a person wearing a simulated mask. After the above processing, the size and format of the face image remain unchanged. The face wearing the simulated mask is then restored to the person image, as shown below. Figure 2 As shown.

[0029] Step 3: Construct a face detection network model for people wearing masks: Based on the RetinaFace network, construct a face detection network, adding supervision to deformable convolutional layers and increasing the proportion of binocular keypoint regression loss in the total loss to enhance the network's ability to detect faces wearing masks. The face detection network is as follows: Figure 3 As shown, the details are as follows.

[0030] Step 3.1: Construct a detection network based on the RetinaFace network. This network uses feature pyramid technology and integrates multi-scale information to extract features from the human image. The image is downsampled to obtain T feature layers. The last three effective feature layers are numbered C1, C2, and C3 respectively. C3 is passed through a 3×3 convolutional layer to obtain P3. C2 is passed through a 3×3 convolutional layer and fused with the upsampled P3 to obtain P2. C1 is passed through a 3×3 convolutional layer and fused with the upsampled P2 to obtain P1. Finally, after passing through 5 SSH context modules, the coordinates of the face target box are output.

[0031] Step 3.2: In the training sample set of human images, the bounding boxes for faces have been labeled, and their coordinates are represented by (x, y, z). t ,y t ,w t ,h t ) indicates that (x t ,y t (w) is the center coordinate. t ,h t The coordinates of the face bounding box are defined by the width and height of the bounding box, respectively. During network training, downsampling is performed on each pair of feature layers, which reduces the size of the face bounding box. By analogy, we can obtain C. i The coordinates of the face bounding box in the feature layer are given by (x... i ,y i ,w i ,h i ) indicates that i = 1, 2, 3.

[0032] Step 3.3: This invention incorporates supervision of deformable convolutional layers, replacing all the aforementioned 3×3 convolutional layers with the improved deformable convolutional layers of this invention. The deformable convolutional layers generate an x-direction offset and a y-direction offset for each sampling point in the convolutional kernel. After the sampling points are offset, they have an irregular positional distribution, enhancing the non-rigid transformation modeling capability of the convolutional network. Secondly, the smallest rectangle that can contain all the offset sampling points is designated as a pseudo-box. By comparing the face target box and the pseudo-box, the localization loss is calculated to supervise the learning of the offset, narrowing the gap between the face target box and the pseudo-box. This allows the pseudo-box to act as an anchor box, improving the network's detection accuracy.

[0033] Step 3.4: In C i Of the nine x-direction offsets in the feature layer, the smallest is represented by w. imin This indicates that the largest use of w imax This indicates that, among the nine y-direction offsets, the smallest is represented by h. imin This indicates that the largest is represented by h. imax The localization loss is represented by the sum of the Euclidean distance between the top-left corner of the face bounding box and the bottom-right corner of the pseudo-bounding box.

[0034]

[0035] Existing deformable convolutional layers only provide offsets without any supervision. This invention improves upon this by using deformable convolutional layers in the RetinaFace network during the feature extraction stage. Furthermore, it supervises the learning of offsets by progressively reducing the target box size and designs a corresponding offset localization loss function, which can effectively improve the accuracy of face detection networks.

[0036] Step 3.5: Remove the facial landmark regression loss, which contributes less to the original RetinaFace loss function when the face is covered by a mask. Simultaneously, split the original facial landmark regression loss into binocular regression loss and other regression losses, increasing the weight of the former. Add these losses to the localization loss to obtain the loss function for the masked face detection network.

[0037]

[0038] Where p i It predicts the probability that the i-th bounding box is a face. It's a real label. and L box These are the face classification loss function and face bounding box regression loss function of RetinaFace, L eye and L else It is the regression loss function for the split eyes and other key points, L p λ1, λ2, λ3, and λ4 are the offset positioning loss, and λ4 are the weights of each loss function, respectively.

[0039] Step 4: The face detection network is trained and tested using both training and testing sample sets of human images. The network outputs the coordinates of the face bounding boxes. Based on these coordinates, face images are cropped from the images to construct a face image sample set, which is used for training and testing the gender classification network, as detailed below:

[0040] Step 4.1: Clean the human image sample data, and remove overly blurry or too small images according to the set threshold;

[0041] Step 4.2: Use the training sample set of human images to train the face detection network model for people wearing masks. Input the human image test sample set into the trained network model and output the predicted coordinates of the face target box. When the detection accuracy of the test sample set reaches 90%, the network is considered to have been successfully trained.

[0042] Step 4.3: Based on the coordinates of the face target box, crop the face image from all the images in the human image sample set to construct a face image sample set. Divide this sample set into two categories, male and female, according to gender.

[0043] Step 4.4: For the classified face image sample set, perform data augmentation operations such as mirroring, rotation, distortion, color transformation, and scale transformation for training and testing of the gender classification network.

[0044] Step 5: Construct and train the gender classification network model: Construct a gender classification network based on the FaceNet network, add a classification layer, design triplet sample generation rules, and train with hard samples to enhance the network's generalization ability. The input to the gender classification network is a face image. The network completes the mapping, obtains the face embedding features, and outputs the person's gender, as detailed below.

[0045] Step 5.1: In this invention, a classification layer is added to the end of the FaceNet network to construct a gender classification network. The classification layer includes one convolutional layer, one pooling layer and one fully connected layer, thereby converting the 128-dimensional face embedding features output by the FaceNet network into gender classification results.

[0046] Step 5.2: Since the original FaceNet network is mainly used in the field of face recognition, its triple loss aims to cluster each person. However, the gender classification network hopes to cluster each gender. Therefore, the triple sample generation rule of the FaceNet network is redesigned: the face image sample set is divided into m batches. For each batch, one image is randomly selected as the template. Then, one image is randomly selected from the same gender image as the positive example and one image is randomly selected from the opposite gender image as the negative example. The template, positive example and negative example together form a triple sample. The above operation is repeated n times to initially build the training set and to initially train and test the FaceNet network.

[0047] Step 5.3: After the initial training of the FaceNet network, the added classification layer also needs to be trained. After initial training and testing, for each face image, the FaceNet network has obtained its 128-dimensional face embedding features. For each batch, the Euclidean distance between the face embedding features corresponding to all face images is calculated. One image is randomly selected as a template, and then the same-sex image with the largest Euclidean distance is selected as a positive example, and the opposite-sex image with the smallest Euclidean distance is selected as a negative example. The template, positive example, and negative example together form a triplet hard sample. The above operation is repeated n times to reconstruct the training set and train the gender classification network. Training with hard samples enhances the network's generalization ability. Figure 4 As shown.

[0048] Finally, the face detection network and gender classification network trained by the above steps perform face detection and gender classification on the input image of a person wearing a mask, thus completing gender recognition.

Claims

1. A method for gender recognition of facial images suitable for use while wearing a mask, characterized by: Includes the following steps: Step 1: Construct a set of human image samples, including a training set and a test set: In the training set, people in the images are not wearing masks, and facial location annotations, i.e., the coordinates of the facial target boxes, are added to the images. In the test set, people in the images are wearing masks, and the number of training samples is ten times or more than that of the test set. Step 2: Preprocessing of the training sample set of human images: Based on the face position annotations in the training sample set, the face images are cropped from the human images, and images of different styles of masks are collected. For each face image, one mask image is randomly selected. The 2DASL algorithm for face 3D reconstruction and dense alignment is used to align the key points of the face image and the mask image, thereby putting a simulated mask on the face. Step 3: Construct a face detection network model for people wearing masks: Based on the RetinaFace network, construct a face detection network, add supervision to deformable convolutional layers, and increase the proportion of binocular keypoint regression loss in all losses to enhance the network's ability to detect faces wearing masks; specifically: Step 3.1: Construct a detection network based on the RetinaFace network. This network uses feature pyramid technology and integrates multi-scale information to extract features from the image of a person. The image is downsampled to obtain T feature layers, and the last three effective feature layers are numbered as follows: , , , go through After the convolutional layer, we get , go through After convolutional layer and after upsampling Fusion , go through After convolutional layer and after upsampling Fusion Finally, after passing through 5 SSH context modules, the coordinates of the face bounding box are output; Step 3.2: In the training sample set of human images, the bounding boxes of faces have been labeled, and their coordinates are... It means that among them With the center coordinates, These are the width and height of the target bounding box, respectively. During network training, the face target bounding box coordinates are reduced with each downsampling operation performed on each pair of feature layers. And so on, we can obtain The coordinates of the face bounding box in the feature layer are used as follows: This means that i = 1, 2, 3; Step 3.3: Combine all the above... All convolutional layers are replaced with deformable convolutional layers, and supervision of these deformable convolutional layers is added. Each deformable convolutional layer generates a value for each sampling point in the convolutional kernel. Direction offset and one The directional offset, with the sampling points having an irregular positional distribution after offset, enhances the non-rigid transformation modeling capability of convolutional networks; Secondly, the smallest rectangle containing all offset sampling points is designated as the pseudo-box. The localization loss is calculated by comparing the face target box and the pseudo-box to supervise the learning of the offset, thereby narrowing the gap between the face target box and the pseudo-box and allowing the pseudo-box to act as an anchor box, thus improving the network detection accuracy. Step 3.4: In 9 feature layers Among the directional offsets, the smallest is used It means that the biggest use It indicates; in 9 Among the directional offsets, the smallest is used It means that the biggest use The localization loss is represented by the sum of the Euclidean distance between the top-left corner of the face bounding box and the bottom-right corner of the pseudo-bounding box. Step 3.5: Remove the facial landmark regression loss from the original loss function of the RetinaFace network when the face is covered by a mask. Simultaneously, split the facial landmark regression loss into binocular regression loss and other regression losses, increasing the weight of the former. Add these losses to the localization loss to obtain the loss function for the masked face detection network. in It predicts the probability that the i-th bounding box is a face. It's a real label. and These are the face classification loss function and face bounding box regression loss function of RetinaFace. and It is the regression loss function for the split eyes and other key points. It is the offset positioning loss. These are the weights of each loss function; Step 4: The face detection network is trained and tested using the training sample set and the test sample set of human images respectively. The output of the face detection network is the coordinates of the face target box. Based on the coordinates of the face bounding box, the face image is cropped from the image to construct a face image sample set for training and testing of the gender classification network; Step 5: Construct and train the gender classification network model: Based on the FaceNet network, construct a gender classification network, add a classification layer, design triplet sample generation rules, and train with hard samples to enhance the network's generalization ability. The input to the gender classification network is a face image. The network completes the mapping, obtains the face embedding features, and outputs the person's gender, as detailed below: Step 5.1: Add a classification layer at the end of the FaceNet network to build a gender classification network. The classification layer includes one convolutional layer, one pooling layer and one fully connected layer, which converts the 128-dimensional face embedding features output by the FaceNet network into gender classification results. Step 5.2: Divide the face image sample set into m batches. For each batch, randomly select one image as a template. Then, select one image from the same sex as a positive example and randomly select one image from the opposite sex as a negative example. The template, positive example, and negative example together form a triplet sample. Repeat the above operation n times to initially build the training set and perform initial training and testing on the FaceNet network. Step 5.3: After initial training and testing, for each face image, the FaceNet network obtains its 128-dimensional face embedding features. For each batch, the Euclidean distance between the face embedding features corresponding to all face images is calculated. One image is randomly selected as a template. Then, the same-sex image with the largest Euclidean distance is selected as a positive example, and the opposite-sex image with the smallest Euclidean distance is selected as a negative example. The template, positive example, and negative example together form a triplet hard sample. The above operation is repeated n times to reconstruct the training set and train the gender classification network. The training with hard samples enhances the generalization ability of the network. Finally, the face detection network and gender classification network trained by the above steps perform face detection and gender classification on the input face image of a person wearing a mask, thus completing gender recognition.

2. The method for gender recognition of facial images suitable for mask-wearing states according to claim 1, characterized in that: Step 2 is as follows: Step 2.1: Use the 3D reconstruction and dense alignment algorithm 2DASL to reconstruct the face image using 3D point cloud to obtain the key point coordinates of the face image, and perform the same operation on the mask image to obtain the key point coordinates of the mask. Step 2.2: By densely aligning the facial key points and mask key points, the face image and mask image are aligned and superimposed to obtain a face image of a person wearing a simulated mask. After the above processing, the size and format of the face image remain unchanged, and the face wearing the simulated mask is restored to the person image.

3. The method for gender recognition of facial images suitable for mask-wearing states according to claim 1, characterized in that: Step 4 specifically involves: Step 4.1: Clean the human image sample data, and remove overly blurry or too small images according to the set threshold; Step 4.2: Use the training sample set of human images to train the face detection network model for people wearing masks. Input the human image test sample set into the trained network model and output the predicted coordinates of the face target box. When the detection accuracy of the test sample set reaches 90%, the network is considered to have been successfully trained. Step 4.3: Based on the coordinates of the face target box, crop the face image from all the images in the human image sample set to construct a face image sample set. Divide this sample set into two categories, male and female, according to gender. Step 4.4: For the classified face image sample set, perform data augmentation operations, including mirroring, rotation, distortion, color transformation, and scale transformation, for the training and testing of the gender classification network.

Images