Face quality assessment method and device based on multi-label

Abstract

The present invention discloses a multi-label-based face quality assessment method: multi-label annotation is performed on the blur, quality, and mask condition of the face image, and multi-classification annotation is performed on the posture angle; the annotated face image is subjected to a multi-label and multi-classification combined model to obtain face blur, quality, mask condition labels, and posture angle classification, thereby training the multi-label and multi-classification combined model; the fusion coefficients of face blur, quality, mask condition labels, and posture angle classification are learned to obtain face quality influencing factor fusion weights; the face image to be evaluated is subjected to the trained multi-label and multi-classification combined model to obtain face blur, quality, mask condition labels, and posture angle classification, and then the final face quality assessment score is obtained based on the learned face quality influencing factor fusion weights. The present invention also provides a corresponding multi-label-based face quality assessment device.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and more specifically, relates to a multi-label based face quality assessment method and device. Background Art

[0002] When embedded devices perform face capture or face recognition, due to the influence of motion and environment, face images may appear blurry, reflective, and have large posture angles, resulting in low-quality face images. Low-quality face images will reduce the face capture quality and face recognition rate. In order to reduce the impact of low-quality faces, it is necessary to use face quality assessment methods to screen out high-quality face images.

[0003] Current face quality assessment methods include:

[0004] Face quality assessment is achieved by connecting multiple models in series or parallel (e.g., CN202010730848.5, CN202110730128.3). Multiple models are used to evaluate blur, lighting, occlusion, and other conditions. These models require large parameters and computational complexity, making them impractical to run on embedded edge devices.

[0005] Face quality assessment is performed using a multi-task approach (e.g., CN202110389136.6, CN202110252100.3), where multiple task heads output scores for quality-influencing factors. Alternating training of multiple task heads cannot guarantee optimal results for all tasks.

[0006] Using feature analysis methods to achieve face quality assessment (such as CN202210173160.0, CN202111178942.5), and using the face Gaussian features and variance features obtained by the face recognition network for quality assessment will perform face recognition and feature extraction on all faces, which does not meet the requirement that embedded devices only recognize high-quality face images.

[0007] The current methods for estimating face pose angles are:

[0008] 1. Use regression method to regress the posture angle. The regression angle interval (-100°, 100°) is too large to train, and the actual effect is not good.

[0009] 2. Use a combination of classification and regression (such as CN114821705A). Classify first and then regress to determine the attitude angle. First, classify the three attitude angles into intervals of 3 degrees (-99, 99), with 66 categories for each attitude angle, for a total of 198 categories. Finally, perform regression correction within 3 degrees. The problems with this method are: (1) 198 categories of attitude angles require a large number of training samples, and the pitch and roll angles rarely exceed 45 degrees. (2) The precise classification of this method requires a stronger feature extraction capability of the model, which cannot meet the speed requirements on edge devices. Summary of the Invention

[0010] In response to the above defects or improvement needs of the prior art, the present invention provides a multi-label based face quality assessment scheme, which replaces the angle regression of facial posture angles with interval classification, uses a multi-label and multi-classification combined model for training, and uses learnable parameters to learn the fusion weights of each part, avoiding manual setting of fusion parameters, thereby providing a face quality assessment scheme with better accuracy and faster calculation speed on edge devices.

[0011] To achieve the above object, according to one aspect of the present invention, a multi-label based face quality assessment method is provided, comprising the following steps:

[0012] Data annotation: Multi-label annotation of facial image blur, quality, and mask status, and multi-classification annotation of posture angles;

[0013] Model training: The labeled face images are passed through a multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels, and posture angle classification, thereby training the multi-label and multi-classification combination model;

[0014] Fusion factor learning: Learn the fusion coefficients of face blur, quality, mask condition labels, and posture angle classification to obtain the fusion weights of face quality influencing factors;

[0015] Face quality assessment: The face image to be evaluated is passed through the trained multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels and posture angle classification, and then the final face quality assessment score is obtained based on the fusion weights of the learned face quality influencing factors.

[0016] In one embodiment of the present invention, the multi-label annotation of the blur, quality, and mask status of the facial image and the multi-classification annotation of the posture angle include:

[0017] Multiple labels are used to identify blurry, good, or bad facial images, as well as mask conditions. Each label is independent of the others, and each image can contain one or more of these labels.

[0018] The attitude angle is divided into seven categories, each category is mutually exclusive, and the attitude angle can only be one of them.

[0019] In one embodiment of the present invention, attitude angles are divided into seven categories including:

[0020] The yaw angle is divided into three intervals: (0°, 30°), (30°, 60°), and (60°, 90°). Pitch and roll are combined and only divided into (0°, 30°) and above 30°.

[0021] In one embodiment of the present invention, the seven categories of attitude angles are specifically:

[0022] yaw∈(0°,30°) and pitch and roll∈(0°,30°);

[0023] yaw∈(30°, 60°) and pitch and roll∈(0°, 30°);

[0024] yaw∈(60°,90°) and pitch and roll∈(0°,30°);

[0025] yaw∈(0°,30°) and pitch or roll>30°;

[0026] yaw∈(30°, 60°) and pitch and roll>30°;

[0027] yaw∈(60°,90°) and pitch and roll>30°;

[0028] other.

[0029] In one embodiment of the present invention, the annotation format of the face image is {C1, C2, C3, C4}, where:

[0030] C1 represents the blurriness of the image. The label range is between (0, 1). The clearer the image is, the larger the value is. Conversely, the blurrier the image is, the smaller the value is.

[0031] C2 represents the quality of the face image. The label is 0 or 1, 0 represents a very extreme face, an incomplete face, or a non-face image, and 1 represents a normal face image;

[0032] C3 indicates whether the face is blocked or wearing a mask, and the label is 0 or 1, 0 means the face is blocked or wearing a mask; 1 means no blockage and no mask;

[0033] C4 represents the face pose angle, and its label is one of {1, 2, 3, 4, 5, 6, 7}, corresponding to one of the seven face pose angles.

[0034] In one embodiment of the present invention, the multi-label and multi-classification combination model includes preprocessing, backbone network feature extraction, fully connected feature classification, and loss calculation. After back propagation, iterative training is performed to obtain the final multi-label and multi-classification combination model.

[0035] In one embodiment of the present invention, fully connected FC is used to classify the features extracted by the backbone network. The dimension of FC is (256, 10), where 256 represents the number of channels of the features extracted by the backbone network, and 10 is the number of channels to be classified. The first three channels represent blur, good or bad, and mask conditions. Sigmoid is used to activate each channel to obtain the independent probability of each category; the last seven channels represent posture angles, corresponding to one of the seven posture angles. Softmax is used to merge the seven channels to obtain the probability of each category, and the category with the highest probability is selected as the final posture angle category.

[0036] In one embodiment of the present invention, the loss function uses binary cross entropy, and the formula of binary cross entropy is as follows:

[0037] L={l1,l2,...,l n} T

[0038] In the above formula, the dimension of L is N*10, N is the total number of samples, 10 is the number of categories to be classified, and the loss of each sample is l n Calculated by the following formula:

[0039] l n =-y n log(x n )-(1-y n )log(1-x n )

[0040] where y n Labeled face images with face blur, good or bad, mask condition labels, and pose angle classification, x n The face blur, good or bad, mask condition labels and posture angle classification are obtained for the labeled face images through the multi-label and multi-classification combination model.

[0041] In one embodiment of the present invention, the fusion coefficients of face blur, quality, mask condition labels, and posture angle classification are learned to obtain the fusion weights of face quality influencing factors, including:

[0042] After the image passes through the multi-label and multi-classification combined model, the output is the blur, quality, mask condition, and posture angle classification. Blur, quality, and mask condition are already in the range of (0, 1), denoted as S1, S2, and S3 respectively. The posture angle needs to be converted into specific pitch, yaw, and roll values ​​according to the category definition table. Then, pitch, yaw, and roll are transformed to the range of (0, 1) according to the following formula to obtain the face posture angle score, denoted as S4;

[0043]

[0044] The final face quality score SQ is calculated by the following formula: SQ = αS1 + βS2 + γS3 + δS4;

[0045] α, β, γ, and δ represent the fusion coefficients of the four branches, and the fusion coefficients are learned through learnable parameters.

[0046] According to another aspect of the present invention, a multi-label-based face quality assessment device is also provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions executable by the at least one processor. After being executed by the processor, the instructions are used to complete the above-mentioned multi-label-based face quality assessment method.

[0047] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0048] (1) The actual use results show that using angle intervals to classify facial posture angles is better than direct angle regression, and the accuracy is significantly improved;

[0049] (2) Using a training method that combines multi-label and multi-classification, compared to multi-task training, it can multiply the number of supervised labels with the same number of samples, thus improving the algorithm effect. At the same time, there is no problem that multiple tasks cannot be optimized simultaneously due to multi-task alternating training;

[0050] (3) The prediction scores of multiple tasks are fused using learnable parameters, which avoids manual parameter setting. Manual fusion parameters are greatly affected by the test scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the process of multi-label-based face quality assessment method of the present invention;

[0052] Figure 2 Schematic diagram of the structure of the multi-label and multi-classification combination model in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0054] The prior patent application (CN202110389136.6) has the following problems:

[0055] 1. The posture angle uses a common regression method, and the regression angle range (-100°, 100°) is too large, making it difficult to train and the actual effect is not good.

[0056] 2. Use a multi-task head approach to train the model. This approach has the following problems: a) it cannot guarantee that multiple tasks will achieve optimal results; b) tasks are trained alternately, and data labels are limited; c) multiple task heads and multiple sets of parameters increase the size of the model.

[0057] 3. The prediction scores of multiple tasks are fused and added by weight. The weight coefficients are set manually. The manually set fusion parameters are greatly affected by the test scenario.

[0058] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0059] 1. Replace the angle regression of the face pose angle with the interval classification method. The yaw pose angle is divided into three intervals: (0°, 30°), (30°, 60°), and (60°, 90°). Pitch and roll are combined and only classified into (0°, 30°) and above 30°. The specific categories are as follows:

[0060]

[0061] 2. Use multi-label and multi-classification combination models for training.

[0062] 3. Use learnable parameters to learn the fusion weights of each part to avoid manually setting the fusion parameters.

[0063] Specifically, if Figure 1 As shown, the present invention provides a multi-label based face quality assessment method, the overall process is as follows:

[0064] 1. Data annotation: Multi-label annotation of facial image blur, quality, and mask status, and multi-classification annotation of posture angles;

[0065] 2. Model training: The labeled face images are passed through a multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels, and posture angle classification, thereby training the multi-label and multi-classification combination model;

[0066] 3. Fusion factor learning: Learn the fusion coefficients of face blur, quality, mask condition labels, and posture angle classification to obtain the fusion weights of face quality influencing factors;

[0067] 4. Face quality assessment: The face image to be evaluated is subjected to the trained multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels and posture angle classification. The final face quality assessment score is then obtained based on the fusion weights of the learned face quality influencing factors.

[0068] The following is a detailed description of the four parts of the multi-label based face quality assessment method in conjunction with a specific embodiment:

[0069] 1. Data annotation

[0070] The quality assessment method mainly evaluates image quality from four dimensions: blur, quality, mask condition, and posture angle. Blur, quality, and mask condition use multiple labels, and each label is independent of each other. Each image can contain one or more of these labels. The posture angle is implemented by classification and is divided into seven categories. Each category is mutually exclusive, and the posture angle will only be one of them.

[0071] The face pose angle is classified using a multi-classification method. The yaw pose angle is divided into three intervals: (0°, 30°), (30°, 60°), and (60°, 90°). Pitch and roll are combined and only classified into (0°, 30°) and above 30°. The specific categories are as follows:

[0072] For any face image, it is necessary to annotate four dimensions: blur, quality, mask condition, and posture angle. The posture angle is divided into seven categories, so there are a total of 10 categories. The first three categories correspond to blur, quality, and mask condition, and the last seven categories correspond to posture angle.

[0073] The annotation format is {C1, C2, C3, C4}, where

[0074] C1 represents the blurriness of the image. The label range is between (0, 1). The clearer the image is, the larger the value is. Conversely, the blurrier the image is, the smaller the value is.

[0075] C2 represents the quality of the face image. The label is 0 or 1, 0 represents a very extreme face, an incomplete face, or a non-face image, and 1 represents a normal face image;

[0076] C3 indicates whether the face is blocked or wearing a mask, and the label is 0 or 1, 0 means the face is blocked or wearing a mask; 1 means no blockage and no mask;

[0077] C4 represents the face pose angle, and its label is one of {1, 2, 3, 4, 5, 6, 7}, corresponding to one of the seven face pose angles;

[0078] For example, a face image is labeled {0.75, 1, 0, 5}. The meaning of the label is: the image clarity is 0.75, the face is normal, there is occlusion or wearing a mask, yaw∈ (30°, 60°), and pitch and roll>30°.

[0079] 2. Model training

[0080] The multi-label and multi-classification combination model in the embodiment of the present invention is shown in the attached figure. Figure 2 The image is preprocessed, the backbone network extracts features, the fully connected features are classified, and the loss is calculated. After back propagation, iterative training is performed to obtain the final multi-label and multi-classification combination model. The various parts of the model are described as follows:

[0081] 2.1 Image preprocessing: Perform normalization operations on the image by subtracting the mean and dividing the variance, and scale it to H*W*C of (48, 48, 3).

[0082] 2.2 Backbone network feature extraction. The backbone network can be a lightweight model designed by yourself or the Mobilenet or Shufflenet model commonly used in the industry. The input of the Mobilenet and Shufflenet models is 224. Some modifications need to be made to the network to finally output the features of the image. The number of feature channels is 256.

[0083] 2.3 Fully connected feature classification, use fully connected FC to classify the features extracted by the backbone network. The dimension of FC is (256, 10), where 256 represents the number of channels of the features extracted by the backbone network, and 10 is the number of channels to be classified. The first three channels represent blur, good or bad, and mask conditions. The categories are independent of each other and can be one or more of them. Sigmoid is used to activate each channel to obtain the independent probability of each category. The last 7 channels represent posture angles, corresponding to one of the 7 postures. The categories are mutually exclusive and can only be one of the 7 posture angles. Softmax is used to merge the 7 channels to obtain the probability of each category, and the category with the largest probability is selected as the final posture angle category.

[0084] 2.4 Loss calculation, the loss function uses binary cross entropy, and the formula for binary cross entropy is as follows:

[0085] L={l1,l2,...,ln} T (1)

[0086] In the above formula, the dimension of L is N*10, where N is the total number of samples, 10 is the number of categories to be classified, and the loss of each sample is l n Calculated by the following formula:

[0087] l n =-y n log(x n )-(1-y n )log(1-x n ) (2)

[0088] For example, a face image is labeled {0.75, 1, 0, 5} (the annotation format is {C1, C2, C3, C4}, the image label is {0.75, 1, 0, 5}, that is, the label of C4 is 5). First, the label 5 of C4 is transformed into one-hot encoding, that is, it becomes {0, 0, 0, 0, 1, 0, 0} (C4 represents the posture angle of the face, and the label is one of {1, 2, 3, 4, 5, 6, 7}. One-hot encoding is a coding method in which the corresponding position is set to 1 and the other positions are set to 0. The label of C4 is 5, so the 5th position is 1 and the other positions are 0). It is then merged with the first three labels to become {0.75, 1, 0, 0, 0, 0, 0, 1, 0, 0}, denoted as y n After the fully connected features are classified, each of the first three channels is activated using sigmoid, and the last seven channels are combined using softmax. Finally, the activation probabilities of 10 categories are obtained, which are recorded as x n , y n ,x n Substitute this into Formula 2 to calculate the loss.

[0089] 3. Fusion factor learning. After the image passes through the multi-label and multi-classification combination model, the output is blur, quality, mask condition, and posture angle classification. Blur, quality, and mask condition are already in the range of (0, 1), denoted as S1, S2, and S3 respectively. The posture angle needs to be converted into specific pitch, yaw, and roll values ​​according to the category definition table. Then, pitch, yaw, and roll are transformed to the range of (0, 1) according to the following formula (3) to obtain the score of the face posture angle, denoted as S4.

[0090]

[0091] The final face quality score SQ is calculated by the following formula

[0092] SQ=αS1+βS2+γS3+δS4 (4)

[0093] α, β, γ, and δ represent the fusion coefficients of the four branches. The fusion coefficients are learned through learnable parameters, and the learned quality scores can be generated by stronger face quality assessment networks, such as SDD-FIQA, SER-FIQ, etc.

[0094] 4. Face quality assessment: The face image to be evaluated is subjected to the trained multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels and posture angle classification, and then the final face quality assessment score is obtained based on the fusion weights of the learned face quality influencing factors.

[0095] Furthermore, the present invention also provides a multi-label based face quality assessment device: comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the above-mentioned multi-label based face quality assessment method.

[0096] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-label based face quality assessment method, characterized in that: The steps include: Data annotation: Multi-label annotation of facial image blur, quality, and mask conditions, and multi-classification annotation of posture angles; the multi-label annotation of facial image blur, quality, and mask conditions, and multi-classification annotation of posture angles include: facial image blur, quality, and mask conditions use multiple labels, each label is independent of each other, and each image can contain one or more of these labels; posture angles are divided into seven categories, each category is mutually exclusive, and the posture angle can only be one of them; the seven categories of posture angles are specifically: yaw∈(0°, 30°) and pitch and roll ∈ (0°, 30°); yaw ∈ (30°, 60°) and pitch and roll ∈ (0°, 30°); yaw ∈ (60°, 90°) and pitch and roll ∈ (0°, 30°); yaw ∈ (0°, 30°) and pitch or roll > 30°; yaw ∈ (30°, 60°) and pitch and roll > 30°; yaw ∈ (60°, 90°) and pitch and roll > 30°; others; Model training: The labeled face images are passed through a multi-label and multi-classification combination model to obtain face blur, good or bad, mask condition labels, and posture angle classification, thereby training the multi-label and multi-classification combination model; Fusion factor learning: Learn the fusion coefficients of face blur, quality, mask condition labels, and posture angle classification to obtain the fusion weights of face quality influencing factors; Face quality assessment: The face image to be evaluated is subjected to the multi-label and multi-classification combination model obtained through the training to obtain face blur, good or bad, mask condition labels and posture angle classification, and then the final face quality assessment score is obtained based on the fusion weights of the learned face quality influencing factors.

2. The multi-label based face quality assessment method according to claim 1, wherein: The annotation format of the face image is {C1, C2, C3, C4}, where: C1 represents the blurriness of the image. The label range is between (0, 1). The clearer the image is, the larger the value is. Conversely, the blurrier the image is, the smaller the value is. C2 represents the quality of the face image, with a label of 0 or 1, where 0 represents a very extreme face, an incomplete face, or a non-face image, and 1 represents a normal face image; C3 indicates whether the face is blocked or wearing a mask, and the label is 0 or 1, 0 means the face is blocked or wearing a mask; 1 means no blockage and no mask; C4 represents the face pose angle, and its label is one of {1, 2, 3, 4, 5, 6, 7}, corresponding to one of the seven face pose angles.

3. The multi-label based face quality assessment method according to claim 1, wherein: The multi-label and multi-classification combination model includes preprocessing, backbone network feature extraction, fully connected feature classification, and loss calculation. After back propagation, iterative training is performed to obtain the final multi-label and multi-classification combination model.

4. The multi-label based face quality assessment method according to claim 3, wherein: Use fully connected FC to classify the features extracted by the backbone network. The dimension of FC is (256, 10), where 256 represents the number of channels of the features extracted by the backbone network, and 10 is the number of channels to be classified. The first three channels represent blur, good or bad, and mask conditions. Sigmoid is used to activate each channel to obtain the independent probability of each category; the last seven channels represent posture angles, corresponding to one of the seven posture angles. Softmax is used to combine the seven channels to obtain the probability of each category, and the category with the largest probability is selected as the final posture angle category.

5. The multi-label based face quality assessment method according to claim 3, wherein: The loss function uses binary cross entropy, and the formula for binary cross entropy is as follows: In the above formula, the dimension of L is N*10, N is the total number of samples, 10 is the number of categories to be classified, and the loss of each sample is l n Calculated by the following formula: where y n For the face blur, good or bad, mask condition label and posture angle classification of the labeled face image, x n The face blur, good or bad, mask condition labels and posture angle classification are obtained for the labeled face images through the multi-label and multi-classification combination model.

6. The multi-label based face quality assessment method according to claim 1, wherein: The fusion coefficients of face blur, quality, mask condition labels and posture angle classification are learned to obtain the fusion weights of face quality influencing factors, including: After the image passes through the multi-label and multi-classification combined model, the output is the blur, quality, mask condition, and posture angle classification. Blur, quality, and mask condition are already in the range of (0, 1), denoted as S1, S2, and S3 respectively. The posture angle needs to be converted into specific pitch, yaw, and roll values ​​according to the category definition table. Then, pitch, yaw, and roll are transformed to the range of (0, 1) according to the following formula to obtain the face posture angle score, denoted as S4; The final face quality score SQ is calculated by the following formula: ； α, β, γ, and δ represent the fusion coefficients of the four branches, and the fusion coefficients are learned through learnable parameters.

7. A multi-label based face quality assessment device, characterized by: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the multi-label-based face quality assessment method according to any one of claims 1 to 6.

Images