Multi-task face quality evaluation method and system based on attitude and ambiguity
By improving the YOLOv8 algorithm, combining face pose and blur information, a multi-task face quality evaluation method was designed, which solved the problem of low image quality evaluation in face recognition of YoloV8 model, and achieved more accurate face quality evaluation and Improve the inference efficiency of the model.
Patent Information
- Application Number
- CN202411136746.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the Yolo V8 model does not evaluate the image quality sufficiently when face recognition, making it difficult to achieve accurate face detection and recognition.
By improving the YOLOv8 algorithm, combining face pose and image blurring information, a multi-task face quality evaluation method is designed, including target box regression, face pose classification and face blurring scoring tasks, and multi-task learning is used to improve evaluation accuracy.
A more accurate face quality evaluation is achieved, the model's inference efficiency and robustness is improved, and higher quality data is provided to support subsequent face recognition models.
Smart Images

Figure CN118982856A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of face recognition, and in particular to a multi-task face quality assessment method and system based on posture and ambiguity. Background Art
[0002] Face recognition is a biometric technology that identifies people based on their facial features. It uses a camera or camcorder to collect images or video streams containing faces, automatically detects and tracks faces in images, and then performs facial recognition on the detected faces. It is also commonly called portrait recognition or facial recognition.
[0003] When performing face recognition (face detection or face tag analysis), first of all, the quality of the picture or video needs to be evaluated to screen out high-quality face images or videos to improve the accuracy and reliability of face recognition or face detection.
[0004] In the related art, there is a technical problem that the image quality assessment quality is not high when performing face recognition through the YoloV8 model. Based on this, based on the improvement of the YoloV8 model, a multi-task face quality assessment method and system based on posture and ambiguity is proposed. Summary of the invention
[0005] The present invention provides a multi-task face quality assessment method and system based on posture and ambiguity, which realizes comprehensive assessment of face image quality by combining face posture and image ambiguity information.
[0006] According to one aspect of the present disclosure, a multi-task face quality assessment method based on posture and ambiguity is provided, the method comprising: Obtaining video data to be evaluated, and decoding the video data to obtain an image to be evaluated; Extract the features of the image to be evaluated through the backbone network backbone in the YOLOv8 algorithm; The features of the image to be evaluated extracted by the backbone network are fused through the neck network of the YOLOv8 algorithm; wherein the neck network in the YOLOv8 algorithm connects the backbone network and the head network; The head network of the YOLOv8 algorithm is used to predict the image after feature fusion, including: The image after feature fusion is processed by improving the head layer of the YOLOv8 algorithm to complete the target frame regression task, the face posture classification task, and the face ambiguity scoring task; the head layer of the improved YOLOv8 algorithm includes a first head branch, a second head branch, and a third head branch; wherein the first head branch includes a target frame regression task layer, the second head branch includes a face posture classification task layer, and the third head branch includes a face ambiguity scoring task layer; Scoring the tasks according to the multi-task face quality scoring strategy to obtain a face quality score and a face target frame coordinate; the multi-task face quality scoring strategy includes: the face quality score is the sum of the face posture score and the face blur score; the face quality score has a value between 0 and 1, and a larger value indicates a higher quality; Among them, the value of the face blur score is between 0-2, and the larger the value, the clearer it is.
[0007] In a possible implementation, the multi-task face quality scoring strategy further includes: the face blur score is directly obtained through the face blur score task layer, and the face posture score is determined through the correspondence between the face posture classification and the face posture score, and the correspondence includes: Neutral - 1 point, slightly left - 0.5 point, severely left - 0 point, slightly right - 0.5 point, severely right - 0 point, slightly above - 0.5, severely above 0 point, slightly below - 0.5 point, severely below - 0 point; Among them, the face posture classification output by the face posture classification task layer is the digital label of the corresponding classification.
[0008] In a possible implementation, the target box regression task layer includes a convolution module A1, a first attention module, a convolution module A2, and a first fully connected module connected in sequence; The convolution module A1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and Convolutional layer; the first attention module adopts a self-attention layer; the structure of convolutional module A2 is the same as that of convolutional module A1; the first fully connected module includes a first fully connected layer, in which the input dimension is the output dimension of convolutional module A2, and the output dimension is The number of candidate boxes. 4 indicates the number of coordinates of each candidate box.
[0009] In a possible implementation, the face posture classification task layer includes a convolution module B1, a second attention module, a convolution module B2, and a second fully connected module connected in sequence; The convolution module B1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and Convolutional layer; the second attention module adopts a self-attention layer; the structure of convolutional module B2 is the same as that of convolutional module B1; the second fully connected module includes a second fully connected layer, in which the input dimension is the output dimension of convolutional module B2, and the output dimension is The number of candidate boxes, 9 represents the number of pose categories for each candidate box.
[0010] In a possible implementation, the face blur scoring task layer includes a convolution module C1, a third attention module, a convolution module C2, and a third fully connected module connected in sequence; The convolution module C1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and convolution layer; the third attention module adopts a self-attention layer; the convolution module C2 has the same structure as the convolution module C1; the third fully connected module includes a third fully connected layer, in which the input dimension is the output dimension of the convolution module C2, and the output dimension is The number of candidate boxes, 1 represents the number of pose categories of each candidate box; wherein, the structures of the first attention module, the second attention module and the third attention module are the same, and the structures of the first fully connected module, the second fully connected module and the third fully connected module are the same.
[0011] Another aspect of the present disclosure is a multi-task face quality assessment system based on posture and ambiguity, the system applies the assessment method, and the system includes: One or more cameras for collecting video data; Video stream decoder, which decodes the video stream generated by the camera to obtain image frames; The multi-task face quality assessment module is used to perform quality assessment on the image frame according to the assessment method to obtain the coordinates of the face target frame and the face quality score.
[0012] Technical effect: A multi-task face quality assessment network is designed based on the YOLOv8 network, which can simultaneously realize face target frame detection, face posture category recognition, and face blur score. It also performs a comprehensive score on each face based on the posture category and blur score, thereby achieving more accurate face quality assessment.
[0013] The disclosed embodiment designs a target frame regression task layer, a face pose classification task layer, and a face blur score task layer based on the YOLOv8 network. First, the target frame coordinates of the face are obtained through the target frame regression task layer, and then the face pose and face blur score are obtained through the face pose classification task layer and the face blur score task layer, respectively. This achieves a more accurate face quality assessment.
[0014] This method is improved based on the YOLOv8 neural network model, and comprehensively evaluates the face quality through face posture and face blur to improve the accuracy of the evaluation. Based on the YoloV8 network, multi-task learning is used to realize face detection tasks, face posture classification tasks, and face blur scoring tasks, which improves the reasoning efficiency and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A flowchart of a multi-task face quality assessment method based on posture and ambiguity according to an embodiment of the present disclosure is shown.
[0016] Figure 2 A schematic diagram of a network model of a face quality assessment method based on the YOLOv8 algorithm according to an embodiment of the present disclosure is shown.
[0017] Figure 3 A network structure diagram of a target box regression task layer according to an embodiment of the present disclosure is shown.
[0018] Figure 4 A network structure diagram of the face posture classification task layer according to an embodiment of the present disclosure is shown.
[0019] Figure 5 A network structure diagram of the face blur scoring task layer according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0020] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0021] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0022] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0023] See also Figure 1-Figure 5 , Figure 1 A flowchart of a multi-task face quality assessment method based on posture and ambiguity according to an embodiment of the present disclosure is shown. A multi-task face quality assessment method based on posture and ambiguity, the method comprising: S01, obtaining video data to be evaluated, and decoding the video data to obtain an image to be evaluated; S02, extract the features of the image to be evaluated through the backbone network backbone in the YOLOv8 algorithm; S03, using the neck network of the YOLOv8 algorithm to perform feature fusion on the features of the image to be evaluated extracted by the backbone network; wherein the neck network in the YOLOv8 algorithm connects the backbone network and the head network; Figure 2 A schematic diagram of a network model of a face quality assessment method based on the YOLOv8 algorithm according to an embodiment of the present disclosure is shown. For example, the backbone network adopts CSP (Cross Stage Partial networks), the neck network adopts a path fusion network PAN (Path Aggregation Network) to merge feature maps at different levels, and the head network adopts an improved multi-task head branch, including a first branch, a second branch, and a third branch.
[0024] S04, predict the image after feature fusion through the head network of the YOLOv8 algorithm, including: The image after feature fusion is processed by improving the head layer of the YOLOv8 algorithm to complete the target frame regression task, the face posture classification task, and the face ambiguity scoring task; the head layer of the improved YOLOv8 algorithm includes a first head branch, a second head branch, and a third head branch; wherein the first head branch includes a target frame regression task layer, the second head branch includes a face posture classification task layer, and the third head branch includes a face ambiguity scoring task layer; S05, scoring the task according to a multi-task face quality scoring strategy to obtain a face quality score and a face target frame coordinate; the multi-task face quality scoring strategy includes: the face quality score is the sum of the face posture score and the face ambiguity score; the face quality score has a value between 0 and 2, and a larger value indicates a higher quality; The value of the face blur score is between 0 and 1, and the larger the value, the clearer it is. The larger the face quality score, the higher the quality of the image.
[0025] The face quality is comprehensively evaluated through face posture and face blur to improve the accuracy of the evaluation; based on the YoloV8 network, multi-task learning is used to achieve face detection tasks, face posture classification tasks, and face blur scoring tasks, which improves the reasoning efficiency and robustness. It provides high-quality data for subsequent face recognition models, reduces the pressure on face recognition algorithms, and improves recognition efficiency.
[0026] In a possible implementation, the multi-task face quality scoring strategy further includes: the face blur score is directly obtained through the face blur score task layer, and the face posture score is determined through the correspondence between the face posture classification and the face posture score, and the correspondence includes: Neutral - 1 point, slightly left - 0.5 point, severely left - 0 point, slightly right - 0.5 point, severely right - 0 point, slightly above - 0.5, severely above 0 point, slightly below - 0.5 point, severely below - 0 point; Among them, the face posture classification output by the face posture classification task layer is the digital label of the corresponding classification.
[0027] Figure 3 The network structure diagram of the target frame regression task layer of an embodiment of the present disclosure is shown. In a possible implementation, the target frame regression task layer includes a convolution module A1, a first attention module, a convolution module A2, and a first fully connected module connected in sequence; Among them, the convolution module A1 includes a group normalization layer (GN), Convolutional layer (conv: 3×3, 1), activation function ReLu layer and Convolutional layer (conv: 1×1,1); the first attention module uses a self-attention layer, Figure 3-Figure 5Where Q, K, and V represent the query (query vector), key (key vector), and value (value vector) inputs in the self-attention method. The MatMul layer represents the multiplication operation. SoftMax is a normalization layer. The convolution module A2 has the same structure as the convolution module A1; the first fully connected module includes the first fully connected layer (FC), in which the input dimension is the output dimension of the convolution module A2, and the output dimension is The number of candidate boxes, 4 represents the number of coordinates of each candidate box. The first fully connected layer can also contain a ReLu activation function. The target box regression task layer implements coordinate prediction.
[0028] Figure 4 The network structure diagram of the face posture classification task layer of an embodiment of the present disclosure is shown. In a possible implementation, the face posture classification task layer includes a convolution module B1, a second attention module, a convolution module B2, and a second fully connected module connected in sequence; Among them, the convolution module B1 includes a group normalization layer (GN), Convolutional layer (conv: 3×3, 1), activation function ReLu layer and Convolutional layer (conv: 1×1,1); the second attention module uses a self-attention layer; the convolutional module B2 has the same structure as the convolutional module B1; the second fully connected module includes a second fully connected layer, in which the input dimension is the output dimension of the convolutional module B2, and the output dimension is The number of candidate boxes, 9 represents the number of pose categories of each candidate box. The face pose classification task layer implements face pose prediction.
[0029] Figure 5 The network structure diagram of the face blur scoring task layer of an embodiment of the present disclosure is shown. In a possible implementation, the face blur scoring task layer includes a convolution module C1, a third attention module, a convolution module C2, and a third fully connected module connected in sequence; Among them, the convolution module C1 includes a group normalization layer (GN, Group Normalization), Convolutional layer (conv: 3×3, 1), activation function ReLu layer and Convolutional layer (conv: 1×1,1); the third attention module uses a self-attention layer; the convolution module C2 has the same structure as the convolution module C1; the third fully connected module includes a third fully connected layer, in which the input dimension is the output dimension of the convolution module C2, and the output dimension is The number of candidate boxes, 1 represents the number of pose categories of each candidate box; the first attention module, the second attention module and the third attention module have the same structure, and the first fully connected module, the second fully connected module and the third fully connected module have the same structure. The face fuzziness scoring task layer realizes fuzziness prediction.
[0030] For example, in each head branch, the coordinates of the candidate box are expressed as (x, y, w, h), where (x, y) is the center coordinate of the candidate box, w is the width of the candidate box, and h is the height of the candidate box. The number of candidate boxes can be set to 1000.
[0031] The method is applied to a face quality assessment model. In order to train the face quality assessment model, a self-made face quality dataset can be used, which includes a face target frame, a face posture label (center, slightly left, heavily left, slightly right, heavily right, slightly above, heavily above, slightly below, heavily below), and a face blur score label (0-1). During the training process, the self-made face quality assessment dataset is used to continuously train the face quality assessment multi-task model until the optimal model parameters are reached and the value of the loss function no longer decreases, and the training is stopped.
[0032] A multi-task face quality assessment system based on posture and ambiguity, the system applies the assessment method, and the system comprises: One or more cameras for collecting video data; Video stream decoder, which decodes the video stream generated by the camera to obtain image frames; The multi-task face quality assessment module is used to perform quality assessment on the image frame according to the assessment method to obtain the coordinates of the face target frame and the face quality score.
[0033] A video stream decoder is deployed in the server cluster to decode the video stream generated by the camera to obtain image frames, and then the decoded image frames are sent to the multi-task face assessment model for inference to obtain the face target frame, face posture category and blur score. The multi-task face quality scoring strategy is then used to obtain the final face target frame and quality score result based on these three results.
[0034] Multi-task face quality scoring strategy: face quality score = face pose score + face blur score.
[0035] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A multi-task face quality assessment method based on posture and ambiguity, characterized in that: The method comprises: Obtaining video data to be evaluated, and decoding the video data to obtain an image to be evaluated; Extract the features of the image to be evaluated through the backbone network backbone in the YOLOv8 algorithm; The features of the image to be evaluated extracted by the backbone network are fused through the neck network of the YOLOv8 algorithm; wherein the neck network in the YOLOv8 algorithm connects the backbone network and the head network; The head network of the YOLOv8 algorithm is used to predict the image after feature fusion, including: The image after feature fusion is processed by improving the head layer of the YOLOv8 algorithm to complete the target frame regression task, the face posture classification task, and the face ambiguity scoring task; the head layer of the improved YOLOv8 algorithm includes a first head branch, a second head branch, and a third head branch; wherein the first head branch includes a target frame regression task layer, the second head branch includes a face posture classification task layer, and the third head branch includes a face ambiguity scoring task layer; Scoring the tasks according to a multi-task face quality scoring strategy to obtain a face quality score and a face target frame coordinate; the multi-task face quality scoring strategy includes: the face quality score is the sum of the face posture score and the face blur score; the face quality score has a value between 0 and 2, and a larger value indicates a higher quality; Among them, the value of the face blur score is between 0 and 1, and the larger the value, the clearer it is.
2. The multi-task face quality assessment method based on posture and ambiguity according to claim 1, characterized in that: The multi-task face quality scoring strategy also includes: the face blur score is directly obtained through the face blur score task layer, and the face posture score is determined through the correspondence between the face posture classification and the face posture score, and the correspondence includes: Neutral - 1 point, slightly left - 0.5 point, severely left - 0 point, slightly right - 0.5 point, severely right - 0 point, slightly above - 0.5, severely above 0 point, slightly below - 0.5 point, severely below - 0 point; Among them, the face posture classification output by the face posture classification task layer is the digital label of the corresponding classification.
3. The multi-task face quality assessment method based on posture and ambiguity according to claim 1, characterized in that: The target frame regression task layer includes a convolution module A1, a first attention module, a convolution module A2, and a first fully connected module connected in sequence; The convolution module A1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and Convolutional layer; the first attention module adopts a self-attention layer; the structure of convolutional module A2 is the same as that of convolutional module A1; the first fully connected module includes a first fully connected layer, in which the input dimension is the output dimension of convolutional module A2, and the output dimension is The number of candidate boxes. 4 indicates the number of coordinates of each candidate box.
4. The multi-task face quality assessment method based on posture and ambiguity according to claim 3 is characterized in that: The face posture classification task layer includes a convolution module B1, a second attention module, a convolution module B2, and a second fully connected module connected in sequence; The convolution module B1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and Convolutional layer; the second attention module adopts a self-attention layer; the structure of convolutional module B2 is the same as that of convolutional module B1; the second fully connected module includes a second fully connected layer, in which the input dimension is the output dimension of convolutional module B2, and the output dimension is The number of candidate boxes, 9 represents the number of pose categories for each candidate box.
5. The multi-task face quality assessment method based on posture and ambiguity according to claim 4, characterized in that: The face blur scoring task layer includes a convolution module C1, a third attention module, a convolution module C2, and a third fully connected module connected in sequence; The convolution module C1 includes a group normalization layer, Convolutional layer, activation function ReLu layer and convolution layer; the third attention module adopts a self-attention layer; the convolution module C2 has the same structure as the convolution module C1; the third fully connected module includes a third fully connected layer, in which the input dimension is the output dimension of the convolution module C2, and the output dimension is The number of candidate boxes, 1 represents the number of pose categories of each candidate box; wherein, the structures of the first attention module, the second attention module and the third attention module are the same, and the structures of the first fully connected module, the second fully connected module and the third fully connected module are the same.
6. A multi-task face quality assessment system based on posture and ambiguity, characterized in that: The system applies the evaluation method according to any one of claims 1 to 5, and the system comprises: One or more cameras for collecting video data; Video stream decoder, which decodes the video stream generated by the camera to obtain image frames; The multi-task face quality assessment module is used to perform quality assessment on the image frame according to the assessment method to obtain the coordinates of the face target frame and the face quality score.
Citation Information
Patent Citations
Deep learning-based multi-dimensional output face quality evaluation method and electronic equipment
CN113158860A
Construction method and application of face quality evaluation model
CN113436174A
Face quality evaluation model training method and device, equipment and storage medium
CN117152551A
