Facial key point evaluation method, device, equipment and storage medium

Abstract

The present application relates to the field of face recognition technology, and discloses a method, device, equipment and storage medium for evaluating key points of a face. The method includes: obtaining an image to be evaluated and a reference image; taking a screenshot of the image to be evaluated to obtain multiple sub-images to be evaluated; inputting the sub-image to be evaluated into a confidence evaluation model to perform confidence evaluation on non-outer contour key points in the sub-image to be evaluated, and obtain a first confidence evaluation result; judging whether the number of target sub-images exceeds a first quantity threshold; calculating the outer contour offset distance between the image to be evaluated and the reference image when the number of target sub-images exceeds the first quantity threshold; performing confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance, and obtain a second confidence evaluation result. The embodiments of the present application can improve the accuracy of face key point detection.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to a method, device, equipment and storage medium for evaluating key points of a face. Background Art

[0002] Facial key point detection and analysis is a key step in the field of face recognition and analysis. As the application scenarios of face recognition technology gradually become more popular, new challenges are also brought to facial key point detection and analysis.

[0003] In the related technology, the model is trained by collecting data from different application scenarios. Due to the diversity of deployment application scenarios, collecting data from different application scenarios requires a lot of resources, and manual labeling of the collected data may also result in human labeling errors, which will reduce the accuracy of facial key point detection in multiple application scenarios. Summary of the invention

[0004] The purpose of this application is to provide a facial key point evaluation method, device, equipment and storage medium, which can improve the robustness of facial key point detection.

[0005] The present application provides a method for evaluating key points of a face, including:

[0006] Acquire an image to be evaluated and a reference image; the image to be evaluated and the reference image both contain a plurality of facial key points, and the facial key points include non-outer contour key points and outer contour key points;

[0007] Taking screenshots of the image to be evaluated to obtain a plurality of sub-images to be evaluated; the sub-images to be evaluated contain all non-outer contour key points of the same type;

[0008] Inputting the sub-image to be evaluated into a confidence evaluation model to perform confidence evaluation on non-outer contour key points in the sub-image to be evaluated, and obtaining a first confidence evaluation result;

[0009] Determine whether the number of target sub-images exceeds a first number threshold; the target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible;

[0010] If yes, calculate the outer contour offset distance between the image to be evaluated and the reference image; the outer contour offset distance is calculated by a first relative distance and a second relative distance, wherein the first relative distance is the relative distance between the outer contour key points in the image to be evaluated and the non-outer contour key points in the target sub-image in the direction of the length of the bridge of the nose, and the second relative distance is the relative distance between the outer contour key points and the non-outer contour key points in the reference image in the direction of the length of the bridge of the nose;

[0011] According to the outer contour offset distance, confidence evaluation is performed on the outer contour key points in the image to be evaluated to obtain a second confidence evaluation result.

[0012] In some embodiments, before taking a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated, the method further includes:

[0013] Calculate a first barycentric coordinate and a second barycentric coordinate; the first barycentric coordinate is the barycentric coordinate of the non-outer contour key point of the left eye type in the image to be evaluated, and the second barycentric coordinate is the barycentric coordinate of the non-outer contour key point of the right eye type in the image to be evaluated;

[0014] Determine rotation angle information according to the first barycentric coordinates and the second barycentric coordinates;

[0015] According to the rotation angle information, the positions of the key points of the faces in the image to be evaluated are corrected to obtain a corrected image to be evaluated.

[0016] In some embodiments, taking a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated includes:

[0017] Taking a screenshot of the image to be evaluated to obtain a first selected rectangular image; the first selected image contains all non-outer contour key points of the same type;

[0018] Taking the center of the first selected image as the center and the long side of the first selected image as the side length, taking a screenshot in the image to be evaluated to obtain a square second selected image;

[0019] The second selected image is processed into a preset image size to obtain the sub-image to be evaluated.

[0020] In some embodiments, the confidence assessment model includes a feature extraction layer, a feature fusion layer and an output layer, and the step of inputting the sub-image to be evaluated into the confidence assessment model to perform confidence assessment on non-outer contour key points in the sub-image to be evaluated to obtain a first confidence assessment result includes:

[0021] Inputting the sub-image to be evaluated into the feature extraction layer to perform convolution extraction on the features of the sub-image to be evaluated layer by layer to obtain feature data at different scales;

[0022] Inputting the feature data at different scales into the feature fusion layer to perform feature fusion on the feature data at different scales to obtain feature fusion data;

[0023] The feature fusion data is input into the output layer to perform data classification on the feature fusion data, and the first confidence evaluation result is generated according to the classification result.

[0024] In some embodiments, the calculating the outer contour offset distance between the image to be evaluated and the reference image includes:

[0025] Calculating an image scale coefficient; the image scale coefficient is a ratio of the third relative distance to the fourth relative distance, the third relative distance being the mean relative distance between different types of non-outer contour key points in each of the target sub-images in the direction of the length of the bridge of the nose, and the fourth relative distance being the mean relative distance between different types of non-outer contour key points in the reference image that correspond one-to-one to the third relative distance in the direction of the length of the bridge of the nose;

[0026] calculating the first relative distance and the second relative distance;

[0027] The outer contour offset distance is calculated according to the image scale coefficient, the first relative distance, and the second relative distance.

[0028] In some embodiments, a non-outer contour key point is selected from the target sub-image, and the first relative distance, the second relative distance, the third relative distance and the fourth relative distance are calculated using the non-outer contour key point.

[0029] In some embodiments, the step of performing confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance to obtain a second confidence evaluation result includes:

[0030] Determine whether the number of target offset distances exceeds a second number threshold; the target offset distance is the outer contour offset distance that does not exceed a preset offset distance;

[0031] If so, obtaining a second confidence evaluation result that the outer contour key point is credible;

[0032] If not, a second confidence evaluation result is obtained indicating that the outer contour key point is unreliable.

[0033] The embodiment of the present application also provides a facial key point evaluation device, comprising:

[0034] The first module is used to obtain an image to be evaluated and a reference image; the image to be evaluated and the reference image both contain a plurality of facial key points, and the facial key points include non-outer contour key points and outer contour key points;

[0035] The second module is used to take a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated; the sub-images to be evaluated contain all non-outer contour key points of the same type;

[0036] The third module is used to input the sub-image to be evaluated into the confidence evaluation model to perform confidence evaluation on the non-outer contour key points in the sub-image to be evaluated to obtain a first confidence evaluation result;

[0037] A fourth module is used to determine whether the number of target sub-images exceeds a first number threshold; the target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible;

[0038] A fifth module is used to calculate the outer contour offset distance between the image to be evaluated and the reference image; the outer contour offset distance is calculated by a first relative distance and a second relative distance, wherein the first relative distance is the relative distance between the outer contour key points in the image to be evaluated and the non-outer contour key points in the target sub-image in the direction of the length of the bridge of the nose, and the second relative distance is the relative distance between the outer contour key points and the non-outer contour key points in the reference image in the direction of the length of the bridge of the nose;

[0039] The sixth module is used to perform confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance to obtain a second confidence evaluation result.

[0040] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned facial key point evaluation method when executing the computer program.

[0041] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned facial key point evaluation method is implemented.

[0042] The beneficial effects of the present application are as follows: by taking screenshots of an image to be evaluated, a plurality of sub-images to be evaluated containing all non-external contour key points of the same type are obtained, the sub-images to be evaluated are input into a confidence evaluation model to perform confidence evaluation on the non-external contour key points in the sub-images to be evaluated, and a first confidence evaluation result characterizing the confidence of the non-external contour key points in the image to be evaluated is obtained; when the number of sub-images to be evaluated that are credible according to the first confidence evaluation result exceeds a first quantity threshold, a contour offset distance between the image to be evaluated and a reference image is calculated, and based on the contour offset distance, a confidence evaluation is performed on the contour key points in the image to be evaluated, and a second confidence evaluation result characterizing the confidence of the contour key points in the image to be evaluated is obtained. Since the confidence assessment model is first used to perform confidence assessment on non-outer contour key points of the same category, when the confidence assessment results of the non-outer contour key points meet the preset conditions, the confidence assessment is then performed on the outer contour key points in the sub-image to be evaluated based on the outer contour offset distance between the image to be evaluated and the reference image. Therefore, the robustness of facial key point detection can be improved by selecting facial key points with high confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the facial key point evaluation method provided in an embodiment of the present application.

[0044] Figure 2 It is a flowchart of the specific method of step S102 provided in an embodiment of the present application.

[0045] Figure 3 It is a flowchart of the method before step S102 provided in an embodiment of the present application.

[0046] Figure 4 It is a flowchart of the specific method of step S103 provided in an embodiment of the present application.

[0047] Figure 5 It is a flowchart of the specific method of step S106 provided in an embodiment of the present application.

[0048] Figure 6 It is a flowchart of the specific method of step S107 provided in an embodiment of the present application.

[0049] Figure 7 It is a structural schematic diagram of a facial key point evaluation device provided in an embodiment of the present application.

[0050] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application.

[0051] Fig. 9 Schematic diagram of distribution of facial key points of an image to be evaluated provided in an embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown can be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second" and the like in the specification, claims and drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] See also Figure 1 , Figure 1 is a flow chart of a method for evaluating key points of a face provided in an embodiment of the present application. In some embodiments of the present application, Figure 1 The method may specifically include but is not limited to steps S101 to S107. Figure 1 These seven steps are introduced in detail.

[0056] Step S101: obtaining an image to be evaluated and a reference image.

[0057] Both the image to be evaluated and the reference image contain multiple facial key points. The image to be evaluated is an image obtained by performing facial key point detection on a facial image, for example, the facial image is input into a pre-trained facial key point detection model to perform facial key point detection on the facial image to obtain the corresponding image to be evaluated. The reference image is an image pre-marked with multiple facial key points, for example, based on multiple sample images containing multiple facial key points, the average value of the position coordinates of each facial key point is taken to mark the reference image. Facial key points refer to key points that represent specific parts of the face, including eyes, eyebrows, nose, mouth, and facial contours.

[0058] The key points of the face include non-contour key points and contour key points. The non-contour key points refer to the key points of the face that represent non-facial contours, such as the key points of the face that represent specific parts such as eyes, eyebrows, nose and mouth. The contour key points refer to the key points of the face that represent the facial contour. In the embodiment of the present application, the types of non-contour key points include left eyebrow, right eyebrow, nose, left eye, right eye and mouth, that is, the key points of the face that represent left eyebrow, right eyebrow, nose, left eye, right eye and mouth.

[0059] Step S102: taking a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated.

[0060] The sub-image to be evaluated contains all non-outer contour key points of the same type.

[0061] In a specific implementation, according to the type of each non-outer contour key point, a screenshot is taken of the image to be evaluated, so that the screenshotted sub-image to be evaluated contains all non-outer contour key points of the same type. The type of the non-outer contour key point can be determined by a face key point detection model. The face image is input into the face key point detection model, and the face key point detection model detects multiple face key points in the face image, and determines the type of each face key point according to the position of the face key point and the shape formed by the multiple face key points.

[0062] In the embodiment of the present application, the facial key points in the image to be evaluated are also numbered according to the type of each non-outer contour key point. Fig. 9 As shown, there are 68 facial key points in the image to be evaluated, and the labels of each outer contour key point can be 0 to 16, the labels of each type of outer contour key point of the left eyebrow can be 17 to 21, the labels of each type of outer contour key point of the right eyebrow can be 22 to 26, the labels of each type of outer contour key point of the nose can be 27 to 35, the labels of each type of outer contour key point of the left eye can be 36 to 41, the labels of each type of outer contour key point of the right eye can be 42 to 47, and the labels of each type of outer contour key point of the mouth can be 48 to 67.

[0063] See also Figure 2 , Figure 2 is a flowchart of a specific method of step S102 provided in an embodiment of the present application. In some embodiments, step S102 may specifically include but is not limited to steps S201 to S203. Figure 2 These three steps are introduced in detail.

[0064] Step S201: Take a screenshot of the image to be evaluated to obtain a first selected rectangular image.

[0065] The first selected image contains all non-outer contour key points of the same type.

[0066] Step S202 , taking the center of the first selected image as the center and the long side of the first selected image as the side length, taking a screenshot in the image to be evaluated, and obtaining a square second selected image.

[0067] Step S203: Process the second selected image into a preset image size to obtain a sub-image to be evaluated.

[0068] In a specific implementation, all non-outer contour key points to be included are first determined, and then the screenshot range is determined according to the image coordinates of the non-outer contour key points located at the outermost edge (for example, according to the image coordinates of the non-outer contour key points located at the upper left corner and the image coordinates of the non-outer contour key points located at the lower right corner), and a rectangle selection tool is used to take a screenshot in the image to be evaluated to obtain a first selected image, and then the length of the long side and the center coordinates of the first selected image are calculated, and a square selection tool is used to take a screenshot in the image to be evaluated with the center of the first selected image as the center and the long side of the first selected image as the side length to obtain a second selected image, and finally the second selected image is reduced or enlarged, and the second selected image is processed into a preset image size to obtain a sub-image to be evaluated.

[0069] In the above process, if the second selected image obtained by screenshot contains other types of non-outer contour key points, the other types of non-outer contour key points are eliminated.

[0070] See also Figure 3 , Figure 3 Flowchart of the method before step S102 provided in the embodiment of the present application. In some embodiments, before step S102, it may also include but is not limited to steps S301 to S303. Figure 3 These three steps are introduced in detail.

[0071] Step S301, calculating the first barycentric coordinates and the second barycentric coordinates.

[0072] The first centroid coordinates are the centroid coordinates of the non-outer contour key points of the left eye type in the image to be evaluated, and the second centroid coordinates are the centroid coordinates of the non-outer contour key points of the right eye type in the image to be evaluated.

[0073] Step S302: Determine rotation angle information according to the first barycentric coordinates and the second barycentric coordinates.

[0074] Step S303: Correct the positions of the key points of the faces in the image to be evaluated according to the rotation angle information to obtain a corrected image to be evaluated.

[0075] In the specific implementation, the first centroid coordinates are calculated according to the coordinates of the non-outer contour key points of the left eye, and the second centroid coordinates are calculated according to the coordinates of the non-outer contour key points of the right eye. Then, the rotation angle is calculated according to the first centroid coordinates and the second centroid coordinates, and the image coordinates of the non-outer contour key points representing the nose tip among the non-outer contour key points of the nose type are determined as the rotation center. Then, a rotation matrix is ​​constructed according to the rotation center and the rotation angle. Finally, according to the rotation matrix, the position of each facial key point in the image to be evaluated is corrected by bilinear interpolation method, and the image coordinates of each facial key point are updated to obtain the corrected image to be evaluated.

[0076] The calculation formulas for the first and second barycentric coordinates are:

[0077] ,

[0078] ,

[0079] in, is the first barycentric coordinate, is the second barycentric coordinate, is the horizontal coordinate of the i1th non-outer contour key point of the left eye type, is the ordinate of the i1th non-outer contour key point of the left eye type, is the horizontal coordinate of the i2th non-outer contour key point of the right eye type, is the ordinate of the i2th non-outer contour key point of the right eye type, i1∈[1,n1], i2∈[1,n2], n1 is the number of non-outer contour key points of the left eye type, and n2 is the number of non-outer contour key points of the right eye type.

[0080] The calculation formula for the rotation angle is:

[0081] ,

[0082] Where α is the rotation angle.

[0083] The expression of the rotation matrix is:

[0084] ,

[0085] Among them, R is the rotation matrix, x0 is the horizontal coordinate of the non-outer contour key point representing the nose tip, and y0 is the vertical coordinate of the non-outer contour key point representing the nose tip.

[0086] Step S103: input the sub-image to be evaluated into the confidence evaluation model to perform confidence evaluation on the non-outer contour key points in the sub-image to be evaluated, and obtain a first confidence evaluation result.

[0087] In a specific implementation, the sub-image to be evaluated is input into the confidence evaluation model, and the confidence evaluation model sequentially performs feature extraction, feature fusion, and feature classification on the sub-image to be evaluated, and obtains the classification result of the non-outer contour key points in the sub-image to be evaluated. According to the classification result, the first confidence evaluation result corresponding to the sub-image to be evaluated is obtained. If the classification result indicates that the category of the non-outer contour key points is inconsistent with the original category of the non-outer contour key points, the first confidence evaluation result that the sub-image to be evaluated is unreliable is obtained; if the classification result indicates that the category of the non-outer contour key points is consistent with the original category of the non-outer contour key points, the first confidence evaluation result that the sub-image to be evaluated is reliable is obtained.

[0088] In a specific embodiment, the confidence assessment model includes a feature extraction layer, a feature fusion layer and an output layer. Figure 4 , Figure 4 is a flowchart of a specific method of step S103 provided in an embodiment of the present application. Step S103 may specifically include but is not limited to steps S401 to S403. Figure 4 These three steps are introduced in detail.

[0089] Step S401: input the sub-image to be evaluated into a feature extraction layer to perform convolution extraction on the features of the sub-image to be evaluated layer by layer to obtain feature data at different scales.

[0090] In the specific implementation, the sub-image to be evaluated is input into the feature extraction layer, and the sub-image to be evaluated is pooled layer by layer in multiple maximum pooling sub-layers of the feature extraction layer. Each maximum pooling sub-layer performs maximum pooling on the input feature image and outputs the corresponding maximum pooled output images of different scales. Then, convolution extraction is performed in multiple convolution sub-layers respectively, and each convolution sub-layer performs convolution calculation on the input feature images of different scales and outputs the corresponding convolution output feature data.

[0091] Step S402: input the feature data at different scales into a feature fusion layer to perform feature fusion on the feature data at different scales to obtain feature fusion data.

[0092] In a specific implementation, feature data at different scales are input into a feature fusion layer. The feature fusion layer includes a splicing layer structure and a convolution layer structure. In the feature fusion layer, feature data at different scales are first spliced ​​and then convolutionally processed to fuse feature data at different scales to obtain feature fusion data.

[0093] Step S403: input the feature fusion data to the output layer to perform data classification on the feature fusion data, and generate a first confidence evaluation result according to the classification result.

[0094] In a specific implementation, the feature fusion data is input into the output layer, full connection and activation processing is performed in the output layer to obtain an activation result, data classification or regression is performed based on the activation result to obtain a classification result, and a first confidence assessment result is generated based on the classification result.

[0095] Among them, the activation result is a vector, which contains multiple confidence probabilities, representing the probability that the non-outer contour key points in the sub-image to be evaluated belong to the corresponding category, and the largest confidence probability is selected as the classification result, representing that the non-outer contour key points in the sub-image to be evaluated belong to the category corresponding to the confidence probability, and then the category corresponding to the classification result is compared with the original category of the non-outer contour key points to obtain the corresponding first confidence evaluation result.

[0096] Step S104: determine whether the number of target sub-images exceeds a first number threshold.

[0097] If not, execute step S105; if so, execute step S106.

[0098] The target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible.

[0099] Step S105: request to generate the image to be evaluated again. Return to step S101.

[0100] In a specific implementation, if the number of target sub-images does not exceed the first number threshold, it is determined that the key point detection has a large error and needs to be re-detected, and a request is made to generate the image to be evaluated again to generate a new image to be evaluated, or an alarm is issued after obtaining images to be evaluated with large errors for multiple times.

[0101] Step S106, calculating the outer contour offset distance between the image to be evaluated and the reference image.

[0102] Among them, the outer contour offset distance is calculated by the first relative distance and the second relative distance. The first relative distance is the relative distance between the outer contour key points in the image to be evaluated and the non-outer contour key points in the target sub-image in the direction of the length of the nose bridge. The second relative distance is the relative distance between the outer contour key points and the non-outer contour key points in the reference image in the direction of the length of the nose bridge.

[0103] In a specific implementation, by comparing the relative distances between the outer contour key points and non-outer contour key points of the image to be evaluated and the reference image in the direction of the length of the nose bridge, for the image to be evaluated, the non-outer contour key points and each outer contour key point in the target sub-image are selected to calculate the first relative distance, and for the reference image, the non-outer contour key points and each outer contour key point corresponding to the non-outer contour key point in the target sub-image are selected to calculate the second relative distance, and the outer contour offset distance between the image to be evaluated and the reference image is calculated using the first relative distance and the second relative distance.

[0104] See also Figure 5 , Figure 5 is a flowchart of a specific method of step S106 provided in an embodiment of the present application. In some embodiments, step S106 may specifically include but is not limited to steps S501 to S503. Figure 5 These three steps are introduced in detail.

[0105] Step S501, calculating the image scale coefficient.

[0106] Among them, the image scale coefficient is the ratio of the third relative distance to the fourth relative distance, the third relative distance is the mean relative distance between different types of non-outer contour key points in each target sub-image in the direction of the length of the nose bridge, and the fourth relative distance is the mean relative distance between different types of non-outer contour key points in the reference image in the direction of the length of the nose bridge.

[0107] Step S502: Calculate the first relative distance and the second relative distance.

[0108] Step S503: calculating the outer contour offset distance according to the image scale coefficient, the first relative distance and the second relative distance.

[0109] In this embodiment, a non-outer contour key point is selected from the target sub-image, and the first relative distance, the second relative distance, the third relative distance and the fourth relative distance are calculated using the non-outer contour key point.

[0110] The image scale factor is calculated as:

[0111] ,

[0112] ,

[0113] ,

[0114] in, is the image scale factor, is the third relative distance, is the fourth relative distance, is the first non-outer contour key point selected from the target sub-image, is the second non-outer contour key point selected from the target sub-image, is the first non-outer contour keypoint selected from the reference image, is the second non-outer contour keypoint selected from the reference image.

[0115] The calculation formulas for the first relative distance and the second relative distance are:

[0116] ,

[0117] ,

[0118] in, is the first relative distance, is the second relative distance, To select a non-outer contour key point from the target sub-image, To select a non-contour keypoint from the reference image, is the i3th outer contour key point on the image to be evaluated, is the i3th outer contour key point on the reference image, i3∈[1,n3], n3 is the number of outer contour key points.

[0119] The calculation formula for the outer contour offset distance is:

[0120] ,

[0121] in, Offset distance for the i3th outer contour.

[0122] Step S107 , performing confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance, to obtain a second confidence evaluation result.

[0123] See also Figure 6 , Figure 6 is a flowchart of a specific method of step S106 provided in an embodiment of the present application. In some embodiments, step S107 may specifically include but is not limited to steps S601 to S603. Figure 6 These three steps are introduced in detail.

[0124] Step S601, determining whether the number of target offset distances exceeds a second number threshold.

[0125] If yes, go to step S602; if no, go to step S603.

[0126] The target offset distance is an outer contour offset distance that does not exceed a preset offset distance.

[0127] Step S602, obtaining a second confidence evaluation result that the key points of the outer contour are credible;

[0128] Step S603: obtaining a second confidence evaluation result indicating that the key points of the outer contour are unreliable.

[0129] In a specific implementation, if the number of target offset distances exceeds a second quantity threshold, the outer contour key point is determined to be credible; if the number of target offset distances does not exceed the second quantity threshold, the outer contour key point detection is determined to have a large error and needs to be re-detected, and a request is made to generate the image to be evaluated again to generate a new image to be evaluated, or an alarm is issued after obtaining images to be evaluated with large errors for multiple times.

[0130] See also Figure 7 The embodiment of the present application also provides a facial key point evaluation device, which can implement the above-mentioned facial key point evaluation method, and the device includes:

[0131] The first module 701 is used to obtain an image to be evaluated and a reference image; the image to be evaluated and the reference image both contain a plurality of facial key points, and the facial key points include non-outer contour key points and outer contour key points;

[0132] The second module 702 is used to take a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated; the sub-images to be evaluated contain all non-outer contour key points of the same type;

[0133] The third module 703 is used to input the sub-image to be evaluated into the confidence evaluation model to perform confidence evaluation on the non-outer contour key points in the sub-image to be evaluated, and obtain a first confidence evaluation result;

[0134] The fourth module 704 is used to determine whether the number of target sub-images exceeds a first number threshold; the target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible;

[0135] The fifth module 705 is used to calculate the outer contour offset distance between the image to be evaluated and the reference image; the outer contour offset distance is calculated by a first relative distance and a second relative distance, the first relative distance being the relative distance between the outer contour key point in the image to be evaluated and the non-outer contour key point in the target sub-image in the direction of the length of the nose bridge, and the second relative distance being the relative distance between the outer contour key point and the non-outer contour key point in the reference image in the direction of the length of the nose bridge;

[0136] The sixth module 706 is used to perform confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance to obtain a second confidence evaluation result.

[0137] The specific implementation of the facial key point evaluation device is basically the same as the specific implementation of the above-mentioned facial key point evaluation method, and will not be repeated here.

[0138] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0139] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), a display unit 840, etc.

[0141] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 executes the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned face key point evaluation method part of this specification.

[0142] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8201 and / or a cache storage unit 8202 , and may further include a read-only storage unit (ROM) 8203 .

[0143] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0144] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 800 may also communicate with one or more external devices 800' (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 860. The network adapter 860 may communicate with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned facial key point evaluation method is implemented.

[0147] The facial key point evaluation method, apparatus, device and storage medium provided in the embodiments of the present application take screenshots of an image to be evaluated to obtain multiple sub-images to be evaluated that contain all non-external contour key points of the same type, input the sub-images to be evaluated into a confidence evaluation model to perform confidence evaluation on the non-external contour key points in the sub-images to be evaluated, and obtain a first confidence evaluation result characterizing the confidence of the non-external contour key points in the image to be evaluated. When the number of sub-images to be evaluated that are credible according to the first confidence evaluation result exceeds a first quantity threshold, calculate the external contour offset distance between the image to be evaluated and the reference image, and perform confidence evaluation on the external contour key points in the image to be evaluated based on the external contour offset distance to obtain a second confidence evaluation result characterizing the confidence of the external contour key points in the image to be evaluated. Since the confidence assessment model is first used to perform confidence assessment on non-outer contour key points of the same category, when the confidence assessment results of the non-outer contour key points meet the preset conditions, the confidence assessment is then performed on the outer contour key points in the sub-image to be evaluated based on the outer contour offset distance between the image to be evaluated and the reference image. Therefore, the robustness of facial key point detection can be improved by selecting facial key points with high confidence.

[0148] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation of the present disclosure.

[0149] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0151] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0152] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0153] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A facial key point evaluation method, characterized in that: include: Acquire an image to be evaluated and a reference image; the image to be evaluated and the reference image both contain a plurality of facial key points, and the facial key points include non-outer contour key points and outer contour key points; Taking screenshots of the image to be evaluated to obtain a plurality of sub-images to be evaluated; the sub-images to be evaluated contain all non-outer contour key points of the same type; Inputting the sub-image to be evaluated into a confidence evaluation model to perform confidence evaluation on non-outer contour key points in the sub-image to be evaluated, and obtaining a first confidence evaluation result; Determining whether the number of target sub-images exceeds a first number threshold; The target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible; If yes, calculate the outer contour offset distance between the image to be evaluated and the reference image; the outer contour offset distance is calculated by a first relative distance and a second relative distance, wherein the first relative distance is the relative distance between the outer contour key points in the image to be evaluated and the non-outer contour key points in the target sub-image in the direction of the length of the bridge of the nose, and the second relative distance is the relative distance between the outer contour key points and the non-outer contour key points in the reference image in the direction of the length of the bridge of the nose; According to the outer contour offset distance, confidence evaluation is performed on the outer contour key points in the image to be evaluated to obtain a second confidence evaluation result.

2. The facial key point evaluation method according to claim 1, characterized in that: Before taking a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated, the method further includes: Calculate a first barycentric coordinate and a second barycentric coordinate; the first barycentric coordinate is the barycentric coordinate of the non-outer contour key point of the left eye type in the image to be evaluated, and the second barycentric coordinate is the barycentric coordinate of the non-outer contour key point of the right eye type in the image to be evaluated; Determine rotation angle information according to the first barycentric coordinates and the second barycentric coordinates; According to the rotation angle information, the positions of the key points of the faces in the image to be evaluated are corrected to obtain a corrected image to be evaluated.

3. The facial key point evaluation method according to claim 1, characterized in that: The step of taking a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated includes: Taking a screenshot of the image to be evaluated to obtain a first selected rectangular image; the first selected image contains all non-outer contour key points of the same type; Taking the center of the first selected image as the center and the long side of the first selected image as the side length, taking a screenshot in the image to be evaluated to obtain a square second selected image; The second selected image is processed into a preset image size to obtain the sub-image to be evaluated.

4. The facial key point evaluation method according to claim 1, characterized in that: The confidence assessment model includes a feature extraction layer, a feature fusion layer and an output layer. The sub-image to be evaluated is input into the confidence assessment model to perform confidence assessment on non-outer contour key points in the sub-image to be evaluated to obtain a first confidence assessment result, including: Inputting the sub-image to be evaluated into the feature extraction layer to perform convolution extraction on the features of the sub-image to be evaluated layer by layer to obtain feature data at different scales; Inputting the feature data at different scales into the feature fusion layer to perform feature fusion on the feature data at different scales to obtain feature fusion data; The feature fusion data is input into the output layer to perform data classification on the feature fusion data, and the first confidence evaluation result is generated according to the classification result.

5. The facial key point evaluation method according to claim 1, characterized in that: The calculating the outer contour offset distance between the image to be evaluated and the reference image comprises: Calculating an image scale coefficient; the image scale coefficient is a ratio of the third relative distance to the fourth relative distance, the third relative distance being the mean relative distance between different types of non-outer contour key points in each of the target sub-images in the direction of the length of the bridge of the nose, and the fourth relative distance being the mean relative distance between different types of non-outer contour key points in the reference image that correspond one-to-one to the third relative distance in the direction of the length of the bridge of the nose; calculating the first relative distance and the second relative distance; The outer contour offset distance is calculated according to the image scale coefficient, the first relative distance, and the second relative distance.

6. The facial key point evaluation method according to claim 5, characterized in that: A non-outer contour key point is selected from the target sub-image, and the first relative distance, the second relative distance, the third relative distance and the fourth relative distance are calculated using the non-outer contour key point.

7. The facial key point evaluation method according to claim 1, characterized in that: The step of performing confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance to obtain a second confidence evaluation result includes: Determine whether the number of target offset distances exceeds a second number threshold; the target offset distance is the outer contour offset distance that does not exceed a preset offset distance; If so, obtaining a second confidence evaluation result that the outer contour key point is credible; If not, a second confidence evaluation result is obtained indicating that the outer contour key point is unreliable.

8. A facial key point evaluation device, characterized in that: include: The first module is used to obtain an image to be evaluated and a reference image; the image to be evaluated and the reference image both contain a plurality of facial key points, and the facial key points include non-outer contour key points and outer contour key points; The second module is used to take a screenshot of the image to be evaluated to obtain a plurality of sub-images to be evaluated; The sub-image to be evaluated contains all non-outer contour key points of the same type; The third module is used to input the sub-image to be evaluated into the confidence evaluation model to perform confidence evaluation on the non-outer contour key points in the sub-image to be evaluated to obtain a first confidence evaluation result; A fourth module is used to determine whether the number of target sub-images exceeds a first number threshold; The target sub-image is a sub-image to be evaluated whose first confidence evaluation result is credible; A fifth module is used to calculate the outer contour offset distance between the image to be evaluated and the reference image; the outer contour offset distance is calculated by a first relative distance and a second relative distance, wherein the first relative distance is the relative distance between the outer contour key points in the image to be evaluated and the non-outer contour key points in the target sub-image in the direction of the length of the bridge of the nose, and the second relative distance is the relative distance between the outer contour key points and the non-outer contour key points in the reference image in the direction of the length of the bridge of the nose; The sixth module is used to perform confidence evaluation on the outer contour key points in the image to be evaluated according to the outer contour offset distance to obtain a second confidence evaluation result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the facial key point evaluation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for evaluating facial key points according to any one of claims 1 to 7 is implemented.

Images