A method for quantitative analysis of face image quality

Abstract

The present invention relates to a method for quantitative analysis of face image quality. Four factors, namely fine-grained, pose, clarity, and illumination, are introduced for analysis. The face images are classified in a fine-grained manner to distinguish problems such as misdetection, low quality, and occlusion in the face images. The clarity estimation model is trained through ranking learning to achieve accurate estimation of the clarity of face images. The pose angle regression model is used to obtain the specific angles of the face in three-dimensional space. Then, combined with illumination analysis and tested with images in different scenarios, the score mapping parameters suitable for different types and the weights of influencing factors are fitted to give relatively accurate face quality scores under different application scenarios, thereby effectively filtering low-quality faces and providing a subjective face quality evaluation for face images. In addition, the models used are all lightweight models optimized by compression, with fast calculation speed and less resource occupancy.

Description

Technical Field

[0001] The present invention relates to a method for quantitatively analyzing the quality of face images, belonging to the technical field of face image quality assessment. Background Art

[0002] Image quality assessment algorithms aim to give objective quantitative values consistent with human subjective quality judgments using mathematical models. The difference in face image quality assessment is that not only the image quality needs to be considered, but also whether the face image can be used for face recognition. Factors such as the pose, occlusion degree, blur degree, and illumination conditions of the face in the image will all affect the face recognition result, making it difficult to convert the subjective feeling of the face image into a number for measurement. This is a huge challenge faced by face image quality assessment.

[0003] In tasks such as face recognition, face clustering, and face attribute analysis, it is necessary to first filter out face images with poor quality. Moreover, face detection algorithms will inevitably have false detection situations. Directly outputting the face detection results to subsequent links will greatly affect the effects of subsequent algorithms. Therefore, there is an urgent need for a face quality assessment algorithm that can effectively filter out low-quality images and accurately quantitatively analyze the quality of face images.

[0004] Currently, face quality assessment methods can be divided into two categories. One is to subjectively define quality indicators through the human visual system (Human Quality Values, HQV), and the other is that the face recognition effect directly determines the quality score (Machine Quality Values, MQV). In the HQV method, the quality of face images is reflected by calculating influencing factors such as pose and clarity; while in the MQV method, the quality score is usually obtained from the matching similarity of face recognition features.

[0005] In actual application scenarios, simple subjectively defined quality evaluation methods are not sufficient to accurately evaluate the quality of face images, and the quality scores generated based on face recognition similarity completely depend on the face recognition model, which is quite different from the human subjective perception results.

[0006] Currently, face quality assessment methods have the following defects:

[0007] (1) The accuracy of clarity and pose judgment is relatively low

[0008] In the method of subjectively defining face quality, most calculation methods for clarity, pose, etc. adopt traditional methods. However, traditional clarity estimation methods (such as Laplace, Sobel operators, etc.) cannot effectively solve the problems of blur and noise simultaneously; traditional pose estimation methods, for example, by measuring the symmetry of the face area to evaluate the pose result, cannot obtain the specific angle of the face in three-dimensional space, and the quantitative analysis of the pose is not accurate enough.

[0009] (2) Ignored the problem of face occlusion

[0010] Currently, in the method of subjectively defining face quality, there is almost no evaluation of the occlusion influencing factor, and occlusion has a great impact on face recognition. In application scenarios such as monitoring, masks, helmets, and other external occlusion situations are widespread. The face quality assessment algorithm should be able to take into account the occlusion factor and reflect the impact of different occlusion degrees on the quality.

[0011] (3) The method based on face recognition features has poor generalization ability and slow calculation speed

[0012] In order to use face recognition features for knowledge transfer, the quality score regression network has basically the same structure as the face recognition network. In practical applications, the network is not lightweight enough and has poor generalization ability for different face feature extraction models.

[0013] (4) The quality evaluation based on face recognition features does not conform to subjective perception

[0014] In the quality assessment method based on face recognition features, the quality score label is generated by recognition similarity. However, usually in the face recognition dataset, there are few low-quality samples of the same person. The model trained with such generated labels cannot achieve good discrimination for low-quality samples. When calculating similarity, a high-quality sample needs to be selected first. The quality labels generated for different objects are relative and there is no clear standard, which deviates from the subjective perception result. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to provide a method for quantitative analysis of face image quality. Through face quality fine-grained classification, face pose estimation, face sharpness estimation model, and illumination analysis, low-quality face filtering can be effectively carried out to provide a face quality evaluation that conforms to the subjective for face images.

[0016] The present invention adopts the following technical solutions to solve the above technical problems: The present invention designs a method for quantitative analysis of face image quality, which is used to quantify the quality of a local face image to be measured in a to-be-measured image, and performs the following steps A to J to obtain the score corresponding to the local face image to be measured, for quality screening of the local face image to be measured;

[0017] Step A. Based on a quality classification network with a face image as the input and the preset face image fine-grained classification corresponding to the face image as the output, process the local face image to be measured to obtain the fine-grained classification corresponding to the local face image to be measured, and use it as the quality classification corresponding to the local face image to be measured, and then enter step B;

[0018] Step B. Based on the image to be measured, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset first ratio to obtain a first locally expanded face image to be measured. If it exceeds the area of the image to be measured, fill it with all 0 pixel values and proceed to Step C;

[0019] Step C. Based on a pose classification network that takes a face image as input and outputs the probabilities of preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the face image, process the first locally expanded face image to be measured to obtain the prediction results of the preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the first locally expanded face image to be measured, and perform mathematical expectation regression to obtain the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the first locally expanded face image to be measured, that is, obtain the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the local face image to be measured, and then proceed to Step D;

[0020] Step D. Based on the image to be measured, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset second ratio to obtain a second locally expanded face image to be measured, and proceed to Step E;

[0021] Step E. Based on a sharpness estimation network that takes a face image as input and outputs the sharpness data corresponding to the face image, process the second locally expanded face image to be measured to obtain the sharpness data corresponding to the second locally expanded face image to be measured, and use the sigmoid function to map the sharpness data to between 0 and 1 as the sharpness value corresponding to the second locally expanded face image to be measured, that is, obtain the sharpness value corresponding to the local face image to be measured, and then proceed to Step F;

[0022] Step F. Based on the position coordinates of the corner points of the local face image to be measured and the length and width of the local face image to be measured, shrink the selection box corresponding to the local face image to be measured to obtain the local illumination area in the local face image to be measured, and calculate the mean value of the V channel in the HSV color space corresponding to the local illumination area as the face illumination value corresponding to the local face image to be measured, and then proceed to Step G;

[0023] Step G. Calculate the scores corresponding to the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the local face image to be measured, and according to the preset yaw angle direction weight, preset pitch angle direction weight, and roll angle direction weight under each quality classification, apply a weighted method to obtain the pose score corresponding to the local face image to be measured, and then proceed to Step H;

[0024] Step H. Calculate the sharpness score corresponding to the local face image to be measured according to the sharpness value corresponding to the local face image to be measured, and then enter Step I;

[0025] Step I. Calculate the illumination score corresponding to the local face image to be measured according to the face illumination value corresponding to the local face image to be measured, and then enter Step J;

[0026] Step J. According to the pose weight, sharpness weight, and illumination weight corresponding to each quality classification, combined with the quality classification corresponding to the local face image to be measured, according to the following formula:

[0027]

[0028] Obtain the score Score corresponding to the local face image to be measured, where, respectively represent the pose weight, sharpness weight, and illumination weight under the quality classification corresponding to the local face image to be measured, S P 、S C 、S L respectively represent the pose score, sharpness score, and illumination score corresponding to the local face image to be measured, m t represents the preset maximum face image score under the quality classification corresponding to the local face image to be measured.

[0029] As a preferred technical solution of the present invention: in the above-mentioned Step A, based on the fine-grained classification corresponding to the obtained local face image to be measured, combined with the preset mapping relationship between each fine-grained classification and the preset quality classifications, obtain the quality classification corresponding to the fine-grained classification corresponding to the local face image to be measured, that is, obtain the quality classification corresponding to the local face image to be measured, and then enter Step B.

[0030] As a preferred technical solution of the present invention: in the above-mentioned Step C, for the predictions of the preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the obtained first locally expanded face image to be measured, according to the following formula:

[0031]

[0032]

[0033]

[0034] Perform mathematical expectation regression to obtain the continuous predicted angle values yaw p in the yaw angle direction, pitch p in the pitch angle direction, and rollp , that is, obtaining the continuous predicted angle values of the face in the local face image to be measured corresponding to the yaw angle direction, pitch angle direction, and roll angle direction respectively; where i = {0, 1, 2,..., I}, I represents the number of divided angle intervals corresponding to the face in each pose angle direction, and logit yaw represents the output of the pose classification network corresponding to the face in the yaw angle direction, and logit pitch represents the output of the pose classification network corresponding to the face in the pitch angle direction, and logit roll represents the output of the pose classification network corresponding to the face in the roll angle direction, softmax(logit yaw ) i represents logit yaw corresponding to the probability of the i-th angle interval, softmax(logit pitch ) i represents logit pitch corresponding to the probability of the i-th angle interval, softmax(logit roll ) i represents logit roll corresponding to the probability of the i-th angle interval.

[0035] As a preferred technical solution of the present invention: in the step F, according to the following formula:

[0036]

[0037] Calculate to obtain the mean value L of the V channel in the HSV color space corresponding to the local illumination area, as the face illumination value corresponding to the local face image to be measured, where W and H are the width and height of the local illumination area respectively, and V wh is the pixel value of the V channel of the HSV color space corresponding to the coordinate position (w, h) in the local illumination area.

[0038] As a preferred technical solution of the present invention: in the step G, perform the following steps G1 to G4 to obtain the scores corresponding to the continuous predicted angle values of the face in the yaw angle direction, the scores corresponding to the continuous predicted angle values of the face in the pitch angle direction, and the scores corresponding to the continuous predicted angle values of the face in the roll angle direction in the local face image to be measured;

[0039] Step G1. Based on the coordinate system with the pose angle as the abscissa and the score as the ordinate, for the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face: based on the change range of the pose angle of the face in the corresponding direction, with the scores corresponding to the limit pose angles of the face rotating to both sides in this direction being 0, forming the coordinate positions of two corner points, and with the preset maximum score value corresponding to the pose angle of 0 of the face rotating in this direction, forming the coordinate position of the vertex, and then enter step G2;

[0040] Step G2. For the pitch angle direction and roll angle direction corresponding to the face: Based on the same preset first score values corresponding to the same preset first rotation attitude angles when the face rotates the same preset first rotation attitude angle to both sides in the corresponding direction, two first rotation point coordinate positions are formed, where the preset first rotation attitude angle is greater than the 0-degree attitude angle and less than the limit attitude angle, and the preset first score value corresponding to the preset first rotation attitude angle is less than the score value on the straight line connected between the coordinate position of the same-side corner point and the vertex coordinate position corresponding to the preset first rotation attitude angle;

[0041] For the yaw angle direction corresponding to the face: Based on the same preset second score values corresponding to the same preset second rotation attitude angles when the face rotates the same preset second rotation attitude angle to both sides in the corresponding yaw angle direction under each quality classification, two second rotation point coordinate positions corresponding to each quality classification are formed, where the preset second rotation attitude angle under each quality classification is greater than the 0-degree attitude angle and less than the limit attitude angle, and the preset second score value corresponding to the preset second rotation attitude angle under each quality classification is greater than the score value on the straight line connected between the coordinate position of the same-side corner point and the vertex coordinate position corresponding to the preset second rotation attitude angle;

[0042] Then enter step G3;

[0043] Step G3. Connect the first rotation point coordinate positions and the coordinate positions of the same-side corner points in a straight line in sequence from the vertex coordinate position to both sides, to form the corresponding relationship between the attitude angles and scores in the pitch angle direction and roll angle direction corresponding to the face;

[0044] For each quality classification respectively, connect the second rotation point coordinate position corresponding to the quality classification and the coordinate positions of the same-side corner points in a straight line in sequence from the vertex coordinate position to both sides, to form the corresponding relationship between the attitude angles and scores in the yaw angle direction corresponding to the face under this quality classification, and further obtain the corresponding relationship between the attitude angles and scores in the yaw angle direction corresponding to the face under each quality classification;

[0045] Then enter step G4;

[0046] Step G4. Based on the corresponding relationship between the attitude angles and scores in the pitch angle direction and roll angle direction corresponding to the face, and the corresponding relationship between the attitude angles and scores in the yaw angle direction corresponding to the face under each quality classification, combined with the quality classification corresponding to the local face image to be measured, obtain the scores corresponding to the predicted angle continuous values in the yaw angle direction, the scores corresponding to the predicted angle continuous values in the pitch angle direction, and the scores corresponding to the predicted angle continuous values in the roll angle direction of the face in the local face image to be measured.

[0047] As a preferred technical solution of the present invention: Step H includes Steps H1 to H4 as follows, obtaining the clarity score corresponding to the local face image to be measured;

[0048] Step H1. For each preset face sample image, according to the method of Step E, obtain the clarity value corresponding to each face sample image respectively, and then enter Step H2;

[0049] Step H2. According to the image clarity score calculation method, calculate the clarity score corresponding to each face sample image respectively, and then enter Step H3;

[0050] Step H3. Based on the coordinate system with the clarity value as the abscissa and the clarity score as the ordinate, use the clarity value and clarity score corresponding to each face sample image respectively to form the positions of each fitting point, and fit to obtain the corresponding relationship between the clarity value and the clarity score, and then enter Step H4;

[0051] Step H4. According to the corresponding relationship between the clarity value and the clarity score, and the clarity value corresponding to the local face image to be measured, obtain the clarity score corresponding to the local face image to be measured.

[0052] As a preferred technical solution of the present invention: Step I includes Steps I1 to I4, obtaining the illumination score corresponding to the local face image to be measured;

[0053] Step I1. Based on the coordinate system with the illumination value as the abscissa and the illumination score as the ordinate, combined with the preset illumination value range, with the score corresponding to the minimum illumination value as 0, form the starting coordinate position, and with the score corresponding to the maximum illumination value as the preset score, form the ending coordinate position, and then enter Step I2;

[0054] Step I2. Based on the preset maximum illumination score corresponding to the preset high-illumination value between the minimum illumination value and the maximum illumination value under each quality classification, form the high-score coordinate positions corresponding to each quality classification respectively, and then enter Step I3;

[0055] Step I3. For each quality classification respectively, connect the high-score coordinate position and the ending coordinate position corresponding to the quality classification in a downward-opening arc in sequence from the starting coordinate position, to form the corresponding relationship between the illumination value and the illumination score corresponding to the quality classification, and further obtain the corresponding relationship between the illumination value and the illumination score corresponding to each quality classification respectively, and then enter Step I4;

[0056] Step I4. According to the corresponding relationship between the illumination value and the illumination score corresponding to each quality classification respectively, combined with the quality classification corresponding to the local face image to be measured, and the face illumination value corresponding to the local face image to be measured, obtain the illumination score corresponding to the local face image to be measured.

[0057] As a preferred technical solution of the present invention: in the step I2, it further includes, based on the high-score coordinate positions corresponding to each quality classification respectively, using the preset auxiliary light scores corresponding to the preset auxiliary light values between the preset high-score light values and the maximum light value under each quality classification to form the auxiliary coordinate positions corresponding to each quality classification respectively;

[0058] In the step I3, for each quality classification respectively, starting from the starting coordinate position, connecting the high-score coordinate position, the auxiliary coordinate position, and the ending coordinate position corresponding to the quality classification in sequence with a downward-opening arc to form the corresponding relationship between the light value and the light score corresponding to the quality classification, and further obtaining the corresponding relationship between the light value and the light score corresponding to each quality classification respectively.

[0059] As a preferred technical solution of the present invention: in the step A, for the Resnet18 network, remove the last residual module in the network and replace the average pooling layer in the network with an adaptive average pooling layer to obtain an updated network, and the quality classification network is implemented based on this updated network;

[0060] Based on the preset various sample face images and the preset fine-grained classifications of the face images corresponding to the various sample face images respectively, taking the face image as the input and the fine-grained classification corresponding to the face image as the output, train the quality classification network with the above structure, and update to obtain the quality classification network;

[0061] In the step C, for the Resnet18 network, remove the last residual module in the network and update the fully connected layer in the network to include branch fully connected layers corresponding to the yaw angle direction, the pitch angle direction, and the roll angle direction respectively to obtain an updated network, and the pose classification network is implemented based on this updated network;

[0062] Based on the preset various sample face images and the preset angle interval categories corresponding to the yaw angle direction, the pitch angle direction, and the roll angle direction of the faces in the various sample face images respectively, taking the face image as the input and the prediction results of the preset angle interval categories corresponding to the yaw angle direction, the pitch angle direction, and the roll angle direction of the face in the face image as the output, train the pose classification network with the above structure, and update to obtain the pose classification network;

[0063] In the step E, the sharpness estimation network is implemented based on the Resnet10 network; based on the preset various sample face images and the sharpness data corresponding to the various sample face images respectively, taking the face image as the input and the sharpness data corresponding to the face image as the output, train the sharpness estimation network with the above structure, and update to obtain the sharpness estimation network.

[0064] As a preferred technical solution of the present invention: during the training process of the clarity estimation network: for the preset various sample face images, different methods and different degrees of distortion processing are respectively performed to obtain different degrees of low-quality sample face images corresponding to each sample face image under each distortion method, which together constitute each sample face image, and the pairwise method is used to sort each sample face image. Combining with the rankloss loss function, with the face image as the input and the clarity data corresponding to the face image as the output, the clarity estimation network is trained.

[0065] For the method for quantitative analysis of face image quality of the present invention, compared with the prior art by adopting the above technical solution, it has the following technical effects:

[0066] The method for quantitative analysis of face image quality designed by the present invention introduces four aspects of factors, namely fine-grained, pose, clarity, and illumination, for analysis, and classifies the face images in a fine-grained manner to distinguish problems such as misdetection, low quality, and occlusion in the face images; the clarity estimation model is trained through ranking learning to achieve accurate estimation of the clarity of face images; the pose angle regression model is used to obtain the specific angle of the face in three-dimensional space; combined with illumination analysis and tested with different scene images, the score mapping parameters suitable for different types and the weights of influencing factors are fitted to give relatively accurate face quality scores under different application scenarios, thereby effectively filtering low-quality faces and providing a subjective face quality evaluation for face images; in addition, the models used are all lightweight models optimized by compression, with fast calculation speed and less resource occupation. Brief Description of the Drawings

[0067] Figure 1 It is a framework schematic diagram of the method for quantitative analysis of face image quality designed by the present invention;

[0068] Figure 2 It is the corresponding relationship between the pose angle and the score in the implementation designed by the present invention;

[0069] Figure 3 It is the corresponding relationship between the clarity value and the clarity score in the implementation designed by the present invention;

[0070] Figure 4 It is the corresponding relationship between the illumination value and the illumination score in the implementation designed by the present invention. Detailed Description of the Invention

[0071] The following further details the specific implementation manners of the present invention in conjunction with the accompanying drawings of the specification.

[0072] The present invention designs a quantitative analysis method for the quality of face images, which is used to quantify the quality of local face images to be measured in the image to be measured. In practical applications, the construction and training of a quality classification network, a pose classification network, and a sharpness estimation network are first completed.

[0073] Among them, for the quality classification network, for the Resnet18 network, the last residual module in the network is removed, and the average pooling layer in the network is replaced with an adaptive average pooling layer to obtain an updated network. The quality classification network is implemented based on this updated network; and based on the preset face images of various samples and the fine-grained classification of the preset face images corresponding to the face images of various samples respectively, with the face image as the input and the fine-grained classification corresponding to the face image as the output, the quality classification network with the above structure is trained to update and obtain the quality classification network.

[0074] For the pose classification network, for the Resnet18 network, the last residual module in the network is removed, and the fully connected layer in the network is updated to include branch fully connected layers corresponding to the yaw angle direction, pitch angle direction, and roll angle direction respectively to obtain an updated network. The pose classification network is implemented based on this updated network; and based on the preset face images of various samples and the preset angle interval categories corresponding to the yaw angle direction, pitch angle direction, and roll angle direction of the faces in the face images of various samples respectively, with the face image as the input and the prediction results of the preset angle interval categories corresponding to the yaw angle direction, pitch angle direction, and roll angle direction of the face in the face image as the output, the pose classification network with the above structure is trained to update and obtain the pose classification network.

[0075] The sharpness estimation network is implemented based on the Resnet10 network; based on the preset face images of various samples and the sharpness data corresponding to the face images of various samples respectively, with the face image as the input and the sharpness data corresponding to the face image as the output, the sharpness estimation network with the above structure is trained to update and obtain the sharpness estimation network.

[0076] During the training process of the sharpness estimation network in the application: for the preset face images of various samples, different methods and different degrees of distortion processing are respectively performed, including 10 distortion methods such as reducing resolution, Gaussian blur, Gaussian noise, etc. Each distortion method has 4 different distortion degrees to obtain low-quality sample face images with different degrees corresponding to each distortion method for the face images of various samples respectively, which together constitute the face images of each sample. And the face images of each sample are sorted by the pairwise method, combined with the rankloss loss function, with the face image as the input and the sharpness data corresponding to the face image as the output, the sharpness estimation network is trained.

[0077] As shown in Table 1 below, it is a schematic diagram of the structures of the quality classification network, the pose classification network, and the clarity estimation network.

[0078] Table 1

[0079]

[0080] And as shown in Table 2 below, it is a schematic diagram of the specific model sizes of the quality classification network, the pose classification network, and the clarity estimation network in actual applications.

[0081] Table 2

[0082] Model Input Size Number of Parameters FLOPs Quality 3*96*96 2.8M 258.91M Pose 3*112*112 2.8M 352.45M Clarity 3*224*224 4.9M 894.16M

[0083] Based on the acquisition of the above quality classification network, pose classification network, and clarity estimation network, in actual applications, as Figure 1 shown, perform the following steps A to J to obtain the score corresponding to the local face image to be measured, which is used to perform quality screening on the local face image to be measured. Among them, Figure 1 the source of the face pictures that appear is the public dataset information about VGGFACE.

[0084] Step A. Based on the quality classification network with the face image as the input and the preset fine-grained classification of the face image as the output, process the local face image to be measured to obtain the fine-grained classification corresponding to the local face image to be measured; further based on the obtained fine-grained classification corresponding to the local face image to be measured, combined with the preset mapping relationship between each fine-grained classification and the preset quality classifications, obtain the quality classification corresponding to the fine-grained classification corresponding to the local face image to be measured, that is, obtain the quality classification corresponding to the local face image to be measured, and then enter Step B.

[0085] In actual applications, currently 28 types of fine-grained classifications of face quality are supported (including various misdetections, abnormal faces, occlusions, etc.). Then, according to the above design, considering common scenarios and subsequent face feature extraction requirements, map the above fine-grained categories to several common face quality classification outputs, including normal, occluded, pose abnormal, low-quality face, chromaticity abnormal, and non-face. That is, the preset mapping relationship between each fine-grained classification and the preset quality classifications is as shown in Table 3 below, and obtain the quality classification corresponding to the local face image to be measured.

[0086] Table 3

[0087] Quality Output Class Number Quality Classification Output Corresponding Fine-Grained Classification 0 Normal Normal, Slight Occlusion, etc. 1 Facial Occlusion Mask, Brim Occlusion, etc. 2 Abnormal Pose Looking Down, 90-Degree Side Face 3 Low-Quality Human Face Side Face with Mask, Blurred, Truncated, etc. 4 Chromaticity Abnormality Black and White, etc. -1 Non-Human Face Animals, Tires, etc.

[0088] Step B. Based on the image to be measured, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset first ratio to obtain a first locally expanded face image to be measured. If it exceeds the area of the image to be measured, fill it with all 0 pixel values, and adjust the first locally expanded face image to the preset standard size, and perform normalization and standardization processing, and then enter Step C.

[0089] In the application, if there are feature points in the first locally expanded face image to be measured, for the first locally expanded face image to be measured, use the key points to determine whether the input image has been rotated by ±90 or ±180 degrees. If it has been rotated, rotate the first locally expanded face image to the normal direction.

[0090] Step C. Based on a pose classification network with a face image as the input and the probabilities of preset angle intervals corresponding to the yaw angle direction, pitch angle direction, and roll angle direction of the face in the face image as the output, process the first locally expanded face image to be measured to obtain the prediction results of the preset angle intervals corresponding to the yaw angle direction, pitch angle direction, and roll angle direction of the face in the first locally expanded face image to be measured, and calculate according to the following formula:

[0091]

[0092]

[0093]

[0094] Perform mathematical expectation regression to obtain the predicted angle continuous value yaw corresponding to the yaw angle direction of the face in the first locally expanded face image to be measured p , the predicted angle continuous value pitch corresponding to the pitch angle direction p , and the predicted angle continuous value roll corresponding to the roll angle direction p , that is, obtain the predicted angle continuous values corresponding to the yaw angle direction, pitch angle direction, and roll angle direction of the face in the local face image to be measured, and then enter Step D; where i = {0, 1, 2,..., I}, I represents the number of divided angle intervals corresponding to each pose angle direction of the face, logit yaw represents the output of the pose classification network corresponding to the face in the yaw angle direction, logit pitch represents the output of the pose classification network corresponding to the face in the pitch angle direction, logit roll represents the output of the pose classification network corresponding to the face in the roll angle direction, softmax(logit yaw ) i represents the probability corresponding to the i-th angle interval of logit yaw , softmax(logit pitch) i Represents logit pitch The probability corresponding to the i-th angular interval, softmax(logit roll ) i Represents logit roll The probability corresponding to the i-th angular interval.

[0095] Step D. Based on the image to be measured, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset second ratio to obtain a second locally expanded face image to be measured, and adjust the second locally expanded face image to the preset standard size, as well as perform normalization and standardization processing, and then enter Step E.

[0096] Step E. Based on the sharpness estimation network with the face image as the input and the sharpness data corresponding to the face image as the output, process the second locally expanded face image to be measured to obtain the sharpness data corresponding to the second locally expanded face image to be measured, and use the sigmoid function to map the sharpness data to between 0 and 1 as the sharpness value corresponding to the second locally expanded face image to be measured, that is, obtain the sharpness value corresponding to the local face image to be measured, and then enter Step F.

[0097] Step F. Based on the position coordinates of the corner points of the local face image to be measured, as well as the length and width of the local face image to be measured, shrink the selection box corresponding to the local face image to be measured to obtain the local illumination area in the local face image to be measured. According to the following formula:

[0098]

[0099] Calculate the mean value L of the V channel in the HSV color space corresponding to the local illumination area as the face illumination value corresponding to the local face image to be measured. The larger L is, the brighter the face is, and the smaller L is, the darker the face is. Then enter Step G. Where W and H are the width and height of the local illumination area respectively, and V wh Is the pixel value of the V channel in the HSV color space corresponding to the (w, h) coordinate position in the local illumination area.

[0100] Step G. Execute the following Step G1 to Step G4 to calculate the scores corresponding to the predicted angle continuous values in the yaw angle direction, the scores corresponding to the predicted angle continuous values in the pitch angle direction, and the scores corresponding to the predicted angle continuous values in the roll angle direction for the face in the local face image to be measured; and according to the preset yaw angle direction weight, preset pitch angle direction weight, and roll angle direction weight under each quality classification, apply a weighting method to obtain the pose score corresponding to the local face image to be measured, and then enter Step H.

[0101] Step G1. Based on the coordinate system with the attitude angle on the abscissa and the score on the ordinate, for the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the human face: Based on the range of change of the attitude angle of the human face in the corresponding direction, with the scores corresponding to the extreme attitude angles of the human face rotating to both sides in this direction being 0 respectively, forming the coordinate positions of two angular points, and with the preset maximum score value corresponding to the attitude angle of 0 when the human face rotates in this direction, forming the vertex coordinate position, and then proceed to step G2.

[0102] Step G2. For the pitch angle direction and roll angle direction corresponding to the human face: Based on the same preset first score values corresponding to the human face rotating the same preset first rotation attitude angles to both sides in the corresponding direction, forming the coordinate positions of two first rotation points, where the preset first rotation attitude angle is greater than the 0-degree attitude angle and less than the extreme attitude angle, and the preset first score value corresponding to the preset first rotation attitude angle is less than the score value on the straight line connecting the coordinate position of the angular point on the same side and the vertex coordinate position corresponding to the preset first rotation attitude angle;

[0103] For the yaw angle direction corresponding to the human face: Based on the same preset second score values corresponding to the human face rotating the same preset second rotation attitude angles to both sides in the corresponding yaw angle direction under each quality classification, forming the coordinate positions of two second rotation points corresponding to each quality classification respectively, where the preset second rotation attitude angle under each quality classification is greater than the 0-degree attitude angle and less than the extreme attitude angle, and the preset second score value corresponding to the preset second rotation attitude angle under each quality classification is greater than the score value on the straight line connecting the coordinate position of the angular point on the same side and the vertex coordinate position corresponding to the preset second rotation attitude angle; then proceed to step G3.

[0104] Step G3. Connect the coordinate positions of the first rotation points and the coordinate positions of the angular points on the same side in a straight line in turn from the vertex coordinate position to both sides, forming the corresponding relationship between the attitude angle and the score in the pitch angle direction and roll angle direction corresponding to the human face, as Figure 2 shown;

[0105] For each quality classification respectively, connect the coordinate positions of the second rotation points corresponding to the quality classification and the coordinate positions of the angular points on the same side in a straight line in turn from the vertex coordinate position to both sides, forming the corresponding relationship between the attitude angle and the score in the yaw angle direction corresponding to the human face under this quality classification, and further obtaining the corresponding relationship between the attitude angle and the score in the yaw angle direction corresponding to the human face under each quality classification, as Figure 2 shown; then proceed to step G4.

[0106] Step G4. Based on the correspondence between the attitude angles and scores in the pitch angle direction and roll angle direction corresponding to the face, as well as the correspondence between the attitude angles and scores in the yaw angle direction corresponding to the face under each quality classification, and combining the quality classification corresponding to the local face image to be measured, obtain the scores corresponding to the continuous predicted angle values in the yaw angle direction of the face in the local face image to be measured, the scores corresponding to the continuous predicted angle values in the pitch angle direction, and the scores corresponding to the continuous predicted angle values in the roll angle direction.

[0107] Step H. According to the clarity value corresponding to the local face image to be measured, perform the following steps H1 to H4, calculate and obtain the clarity score corresponding to the local face image to be measured, and then enter Step I.

[0108] Step H1. For each preset face sample image, obtain the clarity value corresponding to each face sample image according to the method in Step E, and then enter Step H2.

[0109] Step H2. Calculate and obtain the clarity scores corresponding to each face sample image according to the image clarity score calculation method, and then enter Step H3.

[0110] Step H3. Based on the coordinate system with the clarity value as the abscissa and the clarity score as the ordinate, use the clarity value and clarity score corresponding to each face sample image to form the positions of each fitting point, and fit to obtain the correspondence between the clarity value and the clarity score, as Figure 3 shown, and then enter Step H4.

[0111] Step H4. According to the correspondence between the clarity value and the clarity score, and the clarity value corresponding to the local face image to be measured, obtain the clarity score corresponding to the local face image to be measured.

[0112] Step I. According to the face illumination value corresponding to the local face image to be measured, perform the following steps I1 to I4, calculate and obtain the illumination score corresponding to the local face image to be measured, and then enter Step J.

[0113] Step I1. Based on the coordinate system with the illumination value as the abscissa and the illumination score as the ordinate, in combination with the preset illumination value range, with the score corresponding to the minimum illumination value being 0, form the starting coordinate position, and with the score corresponding to the maximum illumination value being the preset score, form the ending coordinate position, and then enter Step I2.

[0114] Step I2. Based on the preset maximum illumination scores corresponding to the preset high-score illumination values between the minimum illumination value and the maximum illumination value under each quality classification, construct the high-score coordinate positions corresponding to each quality classification; and based on the high-score coordinate positions corresponding to each quality classification, with the preset auxiliary illumination scores corresponding to the preset auxiliary illumination values between the preset high-score illumination value and the maximum illumination value under each quality classification, construct the auxiliary coordinate positions corresponding to each quality classification, and then enter Step I3.

[0115] Step I3. For each quality classification respectively, connect the high-score coordinate position, the auxiliary coordinate position, and the end coordinate position corresponding to the quality classification in a downward-opening arc in sequence from the starting coordinate position, to form the corresponding relationship between the illumination value and the illumination score corresponding to the quality classification, and further obtain the corresponding relationship between the illumination value and the illumination score corresponding to each quality classification respectively, as Figure 4 shown, and then enter Step I4.

[0116] Step I4. According to the corresponding relationship between the illumination value and the illumination score corresponding to each quality classification respectively, combined with the quality classification corresponding to the local face image to be measured, and the face illumination value corresponding to the local face image to be measured, obtain the illumination score corresponding to the local face image to be measured.

[0117] Step J. According to the pose weight, clarity weight, and illumination weight corresponding to each quality classification respectively, combined with the quality classification corresponding to the local face image to be measured, according to the following formula:

[0118]

[0119] Obtain the score Score corresponding to the local face image to be measured, where, respectively represent the pose weight, clarity weight, and illumination weight under the quality classification corresponding to the local face image to be measured, S P 、S C 、S L respectively represent the pose score, clarity score, and illumination score corresponding to the local face image to be measured, m t represents the preset maximum face image score under the quality classification corresponding to the local face image to be measured.

[0120] A quantitative analysis method for face image quality designed by the above technical solution analyzes by introducing four factors: fine-grained, pose, clarity, and illumination. The face images are classified in a fine-grained manner to distinguish problems such as misdetection, low quality, and occlusion in the face images. The clarity estimation model is trained through ranking learning to achieve accurate estimation of the clarity of face images. The pose angle regression model is used to obtain the specific angle of the face in three-dimensional space. Then, combined with illumination analysis and tested with images of different scenarios, the score mapping parameters suitable for different types and the weights of influencing factors are fitted to give a relatively accurate face quality score under different application scenarios, thereby effectively filtering low-quality faces and providing a subjective face quality evaluation for face images. In addition, the models used are all lightweight models optimized after compression, with fast calculation speed and less resource occupancy.

[0121] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those of ordinary skill in the art.

Claims

1. A quantitative analysis method for face image quality, which is used to quantitatively analyze the quality of a local face image to be measured in a to-be-measured image. Characterized in that: Perform the following steps A to J to obtain the score corresponding to the local face image to be measured, which is used to screen the quality of the local face image to be measured; Step A: Based on a quality classification network with a face image as the input and the fine-grained classification of the preset face image corresponding to the face image as the output, process the local face image to be measured to obtain the fine-grained classification corresponding to the local face image to be measured, and use it as the quality classification corresponding to the local face image to be measured, and then enter Step B; Step B: Based on the to-be-measured image, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset first ratio to obtain a first locally expanded face image to be measured. If it exceeds the to-be-measured image area, fill it with all 0 pixel values, and then enter Step C; Step C: Based on a pose classification network with a face image as the input and the probabilities of preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the face image as the output, process the first locally expanded face image to be measured to obtain the prediction results of the preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the first locally expanded face image to be measured, and perform mathematical expectation regression to obtain the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the first locally expanded face image to be measured, that is, obtain the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the local face image to be measured, and then enter Step D; Step D: Based on the to-be-measured image, with the local face image to be measured as the center, expand the selection box corresponding to the local face image to be measured by a preset second ratio to obtain a second locally expanded face image to be measured, and then enter Step E; Step E: Based on a sharpness estimation network with a face image as the input and the sharpness data corresponding to the face image as the output, process the second locally expanded face image to be measured to obtain the sharpness data corresponding to the second locally expanded face image to be measured, and use the sigmoid function to map the sharpness data to between 0 and 1 as the sharpness value corresponding to the second locally expanded face image to be measured, that is, obtain the sharpness value corresponding to the local face image to be measured, and then enter Step F; Step F: Based on the position coordinates of each corner point of the local face image to be measured and the length and width of the local face image to be measured, shrink the selection box corresponding to the local face image to be measured to obtain the local illumination area in the local face image to be measured, and calculate the mean value of the V channel in the HSV color space corresponding to the local illumination area as the face illumination value corresponding to the local face image to be measured, and then enter Step G; Step G. Calculate the scores corresponding to the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction of the face in the local face image to be measured. Then, based on the preset yaw angle direction weight, preset pitch angle direction weight, and roll angle direction weight under each quality classification, use the weighted method to obtain the pose score corresponding to the local face image to be measured. Then, proceed to Step H; Step H. Calculate the sharpness score corresponding to the local face image to be measured based on the sharpness value corresponding to the local face image to be measured. Then, proceed to Step I; Step I. Calculate the illumination score corresponding to the local face image to be measured based on the face illumination value corresponding to the local face image to be measured. Then, proceed to Step J; Step J. Based on the pose weight, sharpness weight, and illumination weight corresponding to each quality classification, combined with the quality classification corresponding to the local face image to be measured, according to the following formula: Obtain the score Score corresponding to the local face image to be measured, where respectively represent the pose weight, sharpness weight, and illumination weight under the quality classification corresponding to the local face image to be measured, S P 、S C 、S L respectively represent the pose score, sharpness score, and illumination score corresponding to the local face image to be measured, m t represents the preset maximum face image score under the quality classification corresponding to the local face image to be measured.

2. The method for quantitatively analyzing the quality of a face image according to claim 1, characterized in that: In Step A, based on the fine-grained classification corresponding to the obtained local face image to be measured, combined with the preset mapping relationship between each fine-grained classification and the preset quality classifications, obtain the quality classification corresponding to the fine-grained classification corresponding to the local face image to be measured, that is, obtain the quality classification corresponding to the local face image to be measured. Then, proceed to Step B.

3. The method for quantitatively analyzing the quality of a face image according to claim 1, characterized in that: In Step C, for the predicted results in the preset angle intervals in the yaw angle direction, pitch angle direction, and roll angle direction of the face in the obtained first locally expanded face image to be measured, according to the following formula: Perform mathematical expectation regression to obtain the continuous predicted angle values yaw of the face in the yaw angle direction corresponding to the first locally expanded face image to be measured p , the continuous predicted angle value pitch in the pitch angle direction p , and the continuous predicted angle value roll in the roll angle direction p , that is, obtain the continuous predicted angle values of the face in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the locally measured face image; where i = {0, 1, 2,..., I}, and I represents the number of angle intervals divided in the direction of each pose angle corresponding to the face, logit yaw represents the output of the pose classification network corresponding to the face in the yaw angle direction, logit pitch represents the output of the pose classification network corresponding to the face in the pitch angle direction, logit roll represents the output of the pose classification network corresponding to the face in the roll angle direction, softmax(logit yaw ) i represents the probability corresponding to the i-th angle interval of logit yaw , softmax(logit pitch ) i represents the probability corresponding to the i-th angle interval of logit pitch , softmax(logit roll ) i represents the probability corresponding to the i-th angle interval of logit roll .

4. The method for quantitatively analyzing the quality of a face image according to claim 1, characterized in that: In Step F, according to the following formula: Calculate the mean value L of the V channel in the HSV color space corresponding to the local illumination area as the face illumination value corresponding to the local face image to be measured, where W and H are the width and height of the local illumination area, respectively, and V wh is the pixel value of the V channel in the HSV color space corresponding to the coordinate position (w, h) in the local illumination area.

5. The method for quantitatively analyzing the quality of a face image according to claim 1, characterized in that: In Step G, perform the following Steps G1 to G4 to obtain the scores corresponding to the continuous predicted angle values in the yaw angle direction, pitch angle direction, and roll angle direction of the face in the local face image to be measured; Step G1. Based on a coordinate system with the pose angle as the abscissa and the score as the ordinate, for the yaw angle direction, pitch angle direction, and roll angle direction of the face: Based on the change range of the pose angle of the face in the corresponding direction, take the scores corresponding to the extreme pose angles of the face rotating to both sides in this direction as 0 to form the coordinate positions of two corner points, and take the preset maximum score value corresponding to the pose angle of 0 of the face rotating in this direction to form the coordinate position of the vertex. Then, proceed to Step G2; Step G2. For the pitch angle direction and roll angle direction corresponding to the face: Based on the same preset first score values corresponding to the face rotating the same preset first rotation posture angle to both sides in the corresponding direction, two first rotation point coordinate positions are formed, where the preset first rotation posture angle is greater than 0 degree posture angle and less than the limit posture angle, and the preset first score value corresponding to the preset first rotation posture angle is less than the score value on the straight line connected between the coordinate position of the same-side corner point and the vertex coordinate position corresponding to the preset first rotation posture angle; For the yaw angle direction corresponding to the face: Based on the same preset second score values corresponding to the face rotating the same preset second rotation posture angle to both sides in the corresponding yaw angle direction under each quality classification, two second rotation point coordinate positions corresponding to each quality classification are formed, where the preset second rotation posture angle under each quality classification is greater than 0 degree posture angle and less than the limit posture angle, and the preset second score value corresponding to the preset second rotation posture angle under each quality classification is greater than the score value on the straight line connected between the coordinate position of the same-side corner point and the vertex coordinate position corresponding to the preset second rotation posture angle; Then enter step G3; Step G3. Connect the first rotation point coordinate positions and the coordinate positions of the same-side corner points in sequence from the vertex coordinate position to both sides, to form the corresponding relationship between the posture angle and the score in the pitch angle direction and roll angle direction corresponding to the face; For each quality classification respectively, connect the second rotation point coordinate positions corresponding to the quality classification and the coordinate positions of the same-side corner points in sequence from the vertex coordinate position to both sides, to form the corresponding relationship between the posture angle and the score in the yaw angle direction corresponding to the face under this quality classification, and further obtain the corresponding relationship between the posture angle and the score in the yaw angle direction corresponding to the face under each quality classification; Then enter step G4; Step G4. Based on the corresponding relationship between the posture angle and the score in the pitch angle direction and roll angle direction corresponding to the face, and the corresponding relationship between the posture angle and the score in the yaw angle direction corresponding to the face under each quality classification, combined with the quality classification corresponding to the local face image to be measured, obtain the scores corresponding to the predicted angle continuous values in the yaw angle direction, pitch angle direction, and roll angle direction corresponding to the face in the local face image to be measured.

6. According to the method for quantitative analysis of face image quality described in claim 1, it is characterized in that: The step H includes steps H1 to H4 as follows, to obtain the sharpness score corresponding to the local face image to be measured; Step H1. For each preset face sample image, obtain the sharpness value corresponding to each face sample image according to the method of step E, and then enter step H2; Step H2. Calculate the sharpness score corresponding to each face sample image according to the image sharpness score calculation method, and then enter step H3; Step H3. Based on a coordinate system with the clarity value on the abscissa and the clarity score on the ordinate, using the clarity values and clarity scores corresponding to each face sample image respectively, form the positions of each fitting point, and fit to obtain the corresponding relationship between the clarity value and the clarity score, then proceed to Step H4; Step H4. According to the corresponding relationship between the clarity value and the clarity score, and the clarity value corresponding to the local face image to be measured, obtain the clarity score corresponding to the local face image to be measured.

7. According to the method for quantitative analysis of face image quality described in claim 1, it is characterized in that: Step I includes Step I1 to Step I4 to obtain the illumination score corresponding to the local face image to be measured; Step I1. Based on a coordinate system with the illumination value on the abscissa and the illumination score on the ordinate, in combination with the preset illumination value range, with the score corresponding to the minimum illumination value being 0, form the starting coordinate position, and with the score corresponding to the maximum illumination value being the preset score, form the ending coordinate position, then proceed to Step I2; Step I2. Based on the preset maximum illumination scores corresponding to the preset high - illumination values between the minimum illumination value and the maximum illumination value under each quality classification, form the high - score coordinate positions corresponding to each quality classification respectively, then proceed to Step I3; Step I3. For each quality classification respectively, connect the high - score coordinate position, the ending coordinate position corresponding to the quality classification in a downward - opening arc in sequence from the starting coordinate position, form the corresponding relationship between the illumination value and the illumination score corresponding to this quality classification, and further obtain the corresponding relationships between the illumination values and the illumination scores corresponding to each quality classification respectively, then proceed to Step I4; Step I4. According to the corresponding relationships between the illumination values and the illumination scores corresponding to each quality classification respectively, in combination with the quality classification corresponding to the local face image to be measured and the face illumination value corresponding to the local face image to be measured, obtain the illumination score corresponding to the local face image to be measured.

8. According to the method for quantitative analysis of face image quality described in claim 7, it is characterized in that: In Step I2, it also includes, based on the high - score coordinate positions corresponding to each quality classification respectively, using the preset auxiliary illumination scores corresponding to the preset auxiliary illumination values between the preset high - illumination value and the maximum illumination value under each quality classification, form the auxiliary coordinate positions corresponding to each quality classification respectively; In Step I3, for each quality classification respectively, connect the high - score coordinate position, the auxiliary coordinate position, the ending coordinate position corresponding to the quality classification in a downward - opening arc in sequence from the starting coordinate position, form the corresponding relationship between the illumination value and the illumination score corresponding to this quality classification, and further obtain the corresponding relationships between the illumination values and the illumination scores corresponding to each quality classification respectively.

9. According to the method for quantitative analysis of face image quality described in claim 1, it is characterized in that: In Step A, for the Resnet18 network, remove the last residual module in the network, and replace the average pooling layer in the network with an adaptive average pooling layer to obtain the updated network, and the quality classification network is implemented based on this updated network; Based on the preset various sample face images and the fine-grained classification of the preset face images corresponding to the various sample face images respectively, training is performed on the quality classification network with the above structure, taking the face image as the input and the fine-grained classification corresponding to the face image as the output, and the quality classification network is updated and obtained; In step C, for the Resnet18 network, the last residual module in the network is removed, and the fully connected layer in the network is updated to include branch fully connected layers corresponding to the yaw angle direction, pitch angle direction, and roll angle direction respectively, and the updated network is obtained. The pose classification network is implemented based on this updated network; Based on the preset various sample face images and the preset angle interval categories corresponding to the faces in the various sample face images in the yaw angle direction, pitch angle direction, and roll angle direction respectively, training is performed on the pose classification network with the above structure, taking the face image as the input and the prediction results of the preset angle interval categories corresponding to the faces in the face image in the yaw angle direction, pitch angle direction, and roll angle direction respectively as the output, and the pose classification network is updated and obtained; In step E, the sharpness estimation network is implemented based on the Resnet10 network; Based on the preset various sample face images and the sharpness data corresponding to the various sample face images respectively, training is performed on the sharpness estimation network with the above structure, taking the face image as the input and the sharpness data corresponding to the face image as the output, and the sharpness estimation network is updated and obtained.

10. The method for quantitative analysis of face image quality according to claim 9, characterized in that: During the training process of the sharpness estimation network: for the preset various sample face images, different methods and different degrees of distortion processing are respectively performed to obtain the low-quality sample face images with different degrees corresponding to each distortion method for the various sample face images, which together constitute each sample face image, and the pairwise method is used to sort each sample face image. Combining the rank loss function, training is performed on the sharpness estimation network with the face image as the input and the sharpness data corresponding to the face image as the output.

Images