Robust shielded face recognition method

By defining the occlusion template and deep network model combined with softmax and multi-attribute label loss function, robust features are generated, which solves the problem of poor recognition performance under arbitrary occlusion in the face image in the prior art, and achieves efficient occlusion of the face Identify effects.

CN118982659APending Publication Date: 2024-11-19JIAXING UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410297342.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-12-11
Filing Date
2024-03-03
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The prior art performs poorly when processing facial recognition under any occlusion situations (such as lighting, posture, external objects, etc.) in the face image, especially the training-based method relies on limited and uneven occlusion data. This results in limited recognition effect.

Method used

A robust occlusion face recognition method is adopted to generate occlusion face images by defining occlusion templates, and a deep network model is used to combine softmax loss function and multi-attribute label loss function for training to generate robust features to identify occlusion faces.

Benefits of technology

It realizes efficient face recognition under arbitrary occlusion, reduces dependence on occlusion area detection, provides an easy-to-extend multi-attribute label framework, and improves recognition effect and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BSA0000297815230000021
    Figure BSA0000297815230000021
  • Figure BSA0000297815230000031
    Figure BSA0000297815230000031
  • Figure BSA0000297815230000051
    Figure BSA0000297815230000051
Patent Text Reader

Abstract

The invention provides a robust shielded face recognition method, which comprises the following steps of: 1, generating a shielded face image according to a defined shielding template; 2, marking an identity tag and a multi-attribute tag for the shielded face image; 3, inputting the sheltered face image together with the identity tag and the multi-attribute tag into a deep network model for training; and 4, in a test stage, the identity prediction of the network model on the tested face image can be obtained only by inputting the to-be-tested face image into the trained network. According to the shielded face recognition method provided by the invention, any shielding problem can be solved; the defined multi-attribute tag framework has expansibility, is used for guiding robust feature extraction, and has a good recognition effect. Besides, according to the shielded face generation method provided by the invention, the accurate shielded area and shielded label information can be obtained without cost, the process of detecting the shielded area in the previous method is avoided, and the dependence of the algorithm on the detection of the shielded area is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of image processing and pattern recognition, and more specifically to a robust occluded face recognition method. Background Art

[0002] Face recognition has received great attention in recent years and has become a successful application of pattern recognition and machine learning. However, in unconstrained face recognition applications, i.e., faces captured without user cooperation, usually only a portion of the face is visible. The face may be occluded by glasses, scarves, veils, smartphones, shadows, hands, or self-occlusion caused by posture. Currently, occluded face recognition remains a challenging and unsolved problem.

[0003] Existing occluded face recognition methods can be roughly divided into two categories depending on whether occluded regions need to be detected. The first category is occluded region-based methods, which first detect occluded and unoccluded regions of the input face image and then extract features from the unoccluded regions. These methods have achieved impressive results, especially on faces with limited occlusion. However, their performance depends heavily on the accuracy of face occlusion detection, which is a challenging task in itself.

[0004] Methods in the second category, called training-based methods, learn occlusion-robust features directly from training data. This category of methods extracts user-defined texture features from cropped face images and then uses the extracted features to train a face recognition classifier. Compared to occlusion region-based methods, these training-based methods do not need to explicitly detect occluded and unoccluded regions, but directly extract features from the entire face image. The main limitation of training-based methods is that they rely heavily on training data. Training data affects training-based methods in two ways: first, there are not enough occluded faces in most databases; second, face images in most databases usually contain a limited number of occlusion types. An example is the AR database. This is one of the widely used databases for face recognition under occlusion conditions. Face images in AR contain occlusions of glasses and scarves. However, in real applications, such as face recognition for video surveillance, arbitrary occlusions may occur. In other words, the available training data only provides a limited representation of all possible occlusion situations. The large IJB-C database contains a subset of faces with 18 occlusion types. But its occlusion types are unevenly distributed, sometimes there are only one or two images of one type. This imbalance will affect the performance of this training-based method. Summary of the invention

[0005] 1. Technical issues solved

[0006] In view of the shortcomings of the existing technology, this application proposes a robust occluded face recognition method to solve the problem of facial recognition in face images with arbitrary occlusions such as lighting, posture, and external objects, and proposes a random occluded face generation strategy.

[0007] (II) Technical solution

[0008] To achieve the above objectives, this application provides the following technical solutions:

[0009] 1. A robust occluded face recognition method, characterized in that it comprises:

[0010] 1. Generate an occluded face image according to the defined occlusion template;

[0011] 2. marking identity labels and multi-attribute labels according to the blocked face image;

[0012] 3. Inputting the blocked face image, the marked identity label and the multi-attribute label into the deep network model for training;

[0013] 4. Input the face image to be tested into the trained network model to recognize the face image to be tested.

[0014] 2. According to the occluded face recognition method described in item 1, wherein: the step of defining an occlusion template in step 1 includes: defining a common occlusion template and a random occlusion template according to the type of occlusion.

[0015] 3. The method for occluded face recognition according to item 1, wherein: the step of generating an occluded face image in step 1 comprises:

[0016] Use feature points to pre-align the face image with the standard average face, and divide the face area into 15-18 areas on average;

[0017] Randomly select an occlusion template and cover it on the aligned face image to obtain the occluded face image.

[0018] 4. According to the occluded face recognition method described in item 1, wherein: the multi-attribute label in step 2 includes an occlusion label, a soft biometric label and a semantic attribute label.

[0019] 5. According to the occluded face recognition method described in item 1, wherein: the identity tag in step 2 refers to classifying the face as a specific identifier that symbolizes the identity.

[0020] 6. According to the occluded face recognition method described in item 1, the loss function of the deep network model includes a softmax loss function and a multi-attribute label loss function.

[0021] 7. According to the occluded face recognition method of item 6, the calculation formula of the multi-attribute label attribute loss function is as follows:

[0022]

[0023] is the multi-attribute loss layer parameter matrix, k and H4 are the extracted features and multi-attributes respectively.

[0024] In the formula,

[0025] The dimension of the label vector, A i Represents the multi-attribute label of the face sample, f i is the characteristic representation of the sample.

[0026] 8. The occluded face recognition method according to item 7, wherein the extracted features refer to the output of the second to last fully connected layer in the network model.

[0027] 9. The method for occluded face recognition according to item 6, wherein the calculation formula of the softmax loss is as follows:

[0028]

[0029] f(·) represents the feature map learned by training the deep network model, and k is the feature f(x i ), W = [w1,…,w C ]∈R K×C and b = [b1, ..., b C ]∈R 1×C They represent the weight and bias of the last fully connected layer, N represents the number of samples, C represents the number of sample categories, and x i represents the face image of sample i, y i represents the identity label of sample i.

[0030] 10. According to the occluded face recognition method of item 6, the calculation formula of the loss function of the deep network model is as follows:

[0031] L=L s +λL att r

[0032] L s is the softmax loss, and λ is a hyperparameter used to balance the softmax loss function and the multi-attribute label loss function.

[0033] The present application also provides a method for generating an occluded face, which is characterized by including occlusion template selection and occluded face generation.

[0034] The present application also provides a multi-attribute labeling method, which is characterized by including occlusion attribute labeling, soft biometric attribute labeling and semantic attribute labeling, and the multi-attribute labeling can be expanded to labels with more and richer attributes according to actual use.

[0035] The present application also provides a multi-attribute label loss function, characterized in that the multi-attribute label loss is an implicit loss function, that is, the multi-attribute label loss needs to be calculated only in the training phase, and there is no need to predict the multi-attribute label in the testing phase.

[0036] (III) Beneficial effects

[0037] Compared with the prior art, this application provides a robust occluded face recognition method, which has the following beneficial effects:

[0038] 1. A robust occluded face recognition method of the present application can handle any occlusion problem. At the same time, the defined multi-attribute label framework is easily scalable and is used to guide robust feature extraction, with good recognition effect;

[0039] 2. The present application discloses a robust occluded face recognition method and an occluded face generation method, which can obtain accurate occluded area and label information at no cost, avoiding the need to detect occluded areas in previous methods and reducing the algorithm's dependence on occluded area detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The overall framework diagram of a robust occluded face recognition method for a specific implementation of the present application.

[0041] Figure 2 Schematic diagram of dividing the face area into 18 areas on average.

[0042] Figure 3 Schematic diagram of a random occlusion template for generating an occluded face image. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0044] The present application provides a robust occluded face recognition method. By defining an occlusion template, an arbitrarily occluded face image can be generated, and accurate occlusion area and label information can be obtained at no cost, avoiding the need to detect occlusion areas in previous methods and reducing the algorithm's dependence on occlusion area detection. The defined multi-attribute labels are used to guide robust feature extraction and have good recognition effects. In addition, the multi-attribute label framework is scalable.

[0045] In the present application, the occlusion template according to the definition refers to the possible types of occlusions. The present application divides any occlusion into ordinary occlusion and random occlusion, and lists a small number of occlusion templates respectively. The method's ability to handle occlusions can be further improved by expanding the types and quantities of these occlusion templates; in the present application, the method of generating an occluded face image according to the definition of an occlusion template is conventional in the art, for example, a face image can be obtained by occluding at a fixed or random position of the face image.

[0046] This application marks identity tags and multi-attribute tags based on the occluded face. The identity tags are conventional in the art, for example, different categories can be used to mark faces of different identities; the multi-attribute tags described in this application are used to guide feature extraction and are scalable, for example, they can include more and richer soft biological attributes, semantic attributes, etc. More and richer multi-attribute tags can improve the accuracy of the method to a certain extent, but they will also increase the computational complexity of the method.

[0047] In some embodiments, the step of defining an occlusion template in step 1 includes: defining a common occlusion template and a random occlusion template according to the type of occlusion. In some embodiments, the step of generating an occluded face image in step 1 includes: pre-aligning the face image with a standard face using feature points, and dividing the face area into 15-18 areas on average; randomly selecting an occlusion template and covering it on the aligned face image to obtain an occluded face image.

[0048] In this application, the common occlusion template refers to conventional types of occlusion, such as glasses occlusion, scarf occlusion, etc.;

[0049] The random occlusion template refers to other arbitrary occlusion situations, such as self-occlusion caused by posture, shadow occlusion caused by lighting, occlusion by external objects such as masks and mobile phones, etc.

[0050] The feature points refer to coordinate points on the face with clear semantics, such as the tip of the nose, the corners of the mouth, the corners of the eyes, etc.

[0051] The pre-alignment refers to aligning the face image with a standard average face.

[0052] In some embodiments, the multi-attribute label in step 2 includes an occlusion label, a soft biometric label, and a semantic attribute label. In some embodiments, the identity label in step 2 refers to a specific identifier that classifies a face as a symbol of identity.

[0053] In this application, the occlusion label refers to a label that marks whether the face area is occluded; the soft biometric label refers to the biological attributes representing the face, such as gender, race, age, etc.; the semantic attribute label refers to the semantic attributes of the face, such as moles, scars, marks, etc.

[0054] In some embodiments, the loss function of the deep network model includes a softmax loss function and a multi-attribute label loss function.

[0055] The softmax loss function refers to the conventional use of cross entropy loss to measure the difference between the predicted distribution and the actual distribution; the multi-attribute label loss function refers to the use of mean absolute error loss as the difference between the predicted distribution and the actual multi-attribute label distribution.

[0056] In some implementations, the multi-attribute label attribute loss function is calculated as follows:

[0057]

[0058] is the multi-attribute loss layer parameter matrix, k and H4 are the extracted features and multi-attributes respectively.

[0059] In the formula,

[0060] The dimension of the label vector, A i Represents the multi-attribute label of the face sample, f i is the feature representation of the sample. In some embodiments, the extracted feature refers to the output of the second to last fully connected layer in the network model.

[0061] In some embodiments, the softmax loss is calculated as follows:

[0062]

[0063] f(·) represents the feature map learned by training the deep network model, and k is the feature f(x i ), W = [w1,…,w C ]∈R K×C and b = [b1, ..., b C ]∈R 1×C They represent the weight and bias of the last fully connected layer, respectively. N represents the number of samples, C represents the number of sample categories, and x i represents the face image of sample i, yi represents the identity label of sample i.

[0064] In some embodiments, the calculation formula of the loss function of the deep network model is as follows:

[0065] L=L s +λL attr

[0066] L s is the softmax loss, and λ is a hyperparameter used to balance the softmax loss function and the multi-attribute label loss function.

[0067] The present application can process the recognition of any obstructed face, including self-occlusion, ordinary conventional occlusion, and occlusion by other external objects, through the method of obstructed face recognition described above.

[0068] Figure 1 This is a general framework diagram of a robust face recognition method of a specific implementation method of the present application, comprising the following steps:

[0069] First, for any unobstructed face, Figure 3 The defined occlusion template listed generates an occluded face image. The first line of the occlusion model is a normal occlusion with a relatively fixed position; the second line is a random occlusion, which is used to generate a random face occlusion image. The steps are as follows: 1) Use feature points to pre-align the face image with the standard average face, and divide the face area into equal parts: Figure 2 18 regions are shown, each of which corresponds to an occlusion attribute; 2) Occlusion face generation: including common occlusion generation and random occlusion face generation. For common occlusions such as glasses, scarves, etc., the occlusion template is overlaid on the aligned face; for random occlusion, the generation method is: randomly generate a coordinate in the entire face area, randomly select Figure 3 The occlusion template defined in is overlaid on the face image to generate an occluded face image.

[0070] Secondly, the identity label and multi-attribute label are marked according to the occluded face image, and the multi-attribute label includes an occlusion attribute label, a soft biological attribute label (such as gender, race and age, etc.) and a semantic attribute label (such as moles, scars, marks, etc.). The marking method of the occlusion attribute label is: if the face area is occluded, its occlusion attribute is marked as 1, otherwise, the occlusion attribute is marked as 0. The soft biological attribute label and the semantic attribute label can be marked with the help of conventional attribute detection methods, such as gender and mole detection methods. The marking method of the soft biological attribute label is: for the gender attribute, 0 represents male and 1 represents female; the racial attribute includes Asians, Africans and Caucasians, which are represented by 0, 1 and 2 respectively; the age attribute includes three stages of childhood, youth and old age, which are represented by 0, 1 and 2 respectively. The semantic attributes include moles, scars and marks. The marking method is: 1 represents a mole (scar / mark) on the face, and 0 represents no mole (scar / mark) on the face.

[0071] Then, the above-mentioned occluded face image, the marked identity label and the multi-attribute label are simultaneously input into the deep network model for training until convergence. The loss function of the deep network model includes softmax loss and multi-attribute label loss, where the softmax loss function is a conventional loss function, and L s It means that the calculation formula is:

[0072]

[0073] f(·) represents the feature map learned by training the deep network model, and k is the feature f(x i ), W = [w1, ..., w C ]∈R K×C and b = [b1, ..., b C ]∈R 1×C They represent the weight and bias of the last fully connected layer, N represents the number of samples, C represents the number of sample categories, and x i represents the face image of sample i, y i represents the identity label of sample i.

[0074] The multi-attribute label attribute loss function formula is as follows:

[0075]

[0076] In the formula, is the parameter matrix obtained by training the multi-attribute loss layer, k and H4 are the dimensions of the feature and multi-attribute label vectors, respectively. i Represents the multi-attribute label of the face sample, f i is the characteristic representation of the sample.

[0077] The total loss function of the deep network model is expressed as:

[0078] L=L s +λL attr

[0079] Where, L s is the softmax loss, and λ is a hyperparameter used to balance the softmax loss function and the multi-attribute label loss function.

[0080] Finally, the face image to be tested is input into the trained network to recognize the test face image.

[0081] This application describes a robust face recognition method. In the training phase, first generate occluded faces; then mark multi-attribute labels; finally, input the face image, identity label, and multi-attribute label into the deep network model for training until convergence. In the test phase, any face image is directly input into the trained deep learning model to obtain predicted identity information.

[0082] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A robust occluded face recognition method, characterized in that: It includes:

1. Generate an occluded face image according to the defined occlusion template; 2. marking identity labels and multi-attribute labels according to the blocked face image; 3. Inputting the blocked face image, the marked identity label and the multi-attribute label into the deep network model for training; 4. Input the face image to be tested into the trained network model to recognize the face image to be tested.

2. The method for face recognition under occlusion according to claim 1, characterized in that: The step of defining an occlusion template in step 1 includes: defining a common occlusion template and a random occlusion template according to the type of occlusion.

3. The method for face recognition under occlusion according to claim 1, characterized in that: The step of generating the blocked face image in step 1 includes: Use feature points to pre-align the face image with the standard average face, and divide the face area into 15-18 areas on average; Randomly select an occlusion template and cover it on the aligned face image to obtain the occluded face image.

4. The method for face recognition under occlusion according to claim 1, characterized in that: The multi-attribute labels in step 2 include occlusion labels, soft biological characteristic labels and semantic attribute labels.

5. The method for face recognition under occlusion according to claim 1, characterized in that: The identity tag in step 2 refers to classifying a face as a specific identifier that symbolizes an identity.

6. The method for face recognition with occlusion according to claim 1, characterized in that: The loss function of the deep network model includes a softmax loss function and a multi-attribute label loss function.

7. The method for face recognition with occlusion according to claim 6, characterized in that: The calculation formula of the multi-attribute label attribute loss function is as follows: is the multi-attribute loss layer parameter matrix, k and H4 are the extracted features and multi-attribute formulas respectively, The dimension of the label vector, A1 represents the multi-attribute label of the face sample, and f is the feature representation of the sample.

8. The method for face recognition with occlusion according to claim 7, characterized in that: The extracted features refer to the output of the penultimate fully connected layer in the network model.

9. The method for face recognition with occlusion according to claim 6, characterized in that: The calculation formula of the softmax loss is as follows: f(·) represents the feature map learned by training the deep network model, and k is the feature f(x i ), W = [w1,…,w C ]∈R K×C and b = [b1, ..., b C ]∈R 1×C They represent the weight and bias of the last fully connected layer, N represents the number of samples, C represents the number of sample categories, and x i represents the face image of sample i, y i represents the identity label of sample i.

10. The method for face recognition with occlusion according to claim 6, characterized in that: The calculation formula of the loss function of the deep network model is as follows: L=L s +λL attr L s is the softmax loss, and λ is a hyperparameter used to balance the softmax loss function and the multi-attribute label loss function.

Citation Information

Cited By

  • Local face age estimation method based on multi-channel learning in shielding scene

    CN119888823A