Face key point detection network training method, face key point detection method and medium
By using isotropic and anisotropic Gaussian distributions to generate ground truth heatmaps in facial landmark detection, and combining adaptive covariance matrix and improved loss function, the problems of large error and optimization difficulty in facial landmark detection under extreme conditions are solved, achieving higher detection accuracy and network convergence speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-01
AI Technical Summary
Existing facial landmark detection technologies have large errors in extreme cases, and the semantic ambiguity of landmarks makes network optimization difficult, resulting in limited improvement in detection accuracy.
We use isotropic and anisotropic Gaussian distributions to generate ground truth heatmaps, and combine them with an adaptive covariance matrix and an improved training loss function to provide semantically ambiguous prior information and optimize the face landmark detection network.
It improves the accuracy of facial landmark detection and the convergence speed of the network, enhances the sensitivity to peak error, and improves detection accuracy and generalization ability.
Smart Images
Figure CN121963276A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of facial landmark detection technology, and more specifically, relates to a facial landmark detection network training method, a facial landmark detection method, and a medium. Background Technology
[0002] With the development of computers and the continuous innovation of deep learning, facial landmark detection, as an essential prerequisite for other face-related tasks (such as facial expression recognition and head pose estimation), has also developed rapidly. Therefore, researching a facial landmark detection method is of great significance.
[0003] Current facial landmark detection techniques are mainly based on heatmap regression. This method generates ground truth heatmaps for each manually labeled facial landmark to supervise network training. In the generated ground truth heatmaps, the larger the heatmap value at each pixel, the higher the probability that the pixel at that location is a facial landmark. However, this method still faces many challenges. For example, in extreme cases such as exaggerated facial expressions, facial occlusion, or large head poses, the error in facial landmark detection is large. Furthermore, due to the semantic ambiguity of landmarks, manual annotation of datasets inevitably introduces some inherent positional bias. When using ground truth heatmaps to supervise network training, it can cause conflicting optimization directions, leading to convergence difficulties and limiting detection accuracy.
[0004] To address the issue of semantic ambiguity in keypoints, most existing methods adjust the network's optimization direction from the perspective of the loss function. While this can mitigate the impact of semantic ambiguity to some extent, the accuracy improvement is limited because the keypoint heatmap encoding method used is still based on the assumption of an isotropic Gaussian distribution and does not contain prior information about semantic ambiguity. Furthermore, previous decoding methods also rely on this assumption, and this fixed decoding approach is ill-suited to situations with diverse variance in heatmap distributions, thus limiting detection performance. Summary of the Invention
[0005] In response to the shortcomings and improvement needs of existing technologies, this invention provides a method for training a facial landmark detection network, a method for detecting facial landmarks, and a medium, with the aim of solving the technical problem of low accuracy in existing facial landmark detection methods.
[0006] To achieve the above objectives, according to one aspect of the present invention, a method for training a facial landmark detection network is provided, comprising:
[0007] S0: Obtain the face image dataset; in the face image dataset, each sample is a face image with key point coordinates annotated; S1: For each sample in the face image dataset, divide the key points into boundary endpoints and points inside the boundary, and transform the coordinates of each key point to the heatmap coordinate system; S2: In the heatmap coordinate system, generate ground truth heatmaps for each boundary endpoint using an isotropic Gaussian distribution, and generate ground truth heatmaps for each point inside each boundary using an anisotropic Gaussian distribution. The training dataset consists of face images and ground truth heatmaps of each key point therein. S3: Use the training dataset to train a facial landmark detection network to predict the heatmap of each key point in a face image, and obtain the trained facial landmark detection network.
[0008] Furthermore, in S2, the expression for the anisotropic Gaussian distribution is:
[0009] in, These are the pixel coordinates in the truth heatmap. express Heatmap values of the location; This indicates the coordinates of the key points in the heatmap coordinate system; Let be the covariance matrix of an anisotropic Gaussian distribution; the superscript T indicates matrix transpose.
[0010] Furthermore,
[0011] in, Represents the rotation matrix. The local variance matrix is calculated as follows:
[0012]
[0013] in, and These are the unit vectors for the tangential and normal directions, respectively; and These are the variances in the tangential and normal directions, respectively.
[0014] Furthermore, for key points inside the boundary , and ,as well as and according to and its preceding adjacent key point The next adjacent key point Calculations show that , , ; and ,as well as and The expressions are as follows:
[0015]
[0016]
[0017]
[0018] in, Indicates by point to The vector, Indicates by point to ; Based on the variance, For vectors sum vector The angle between them The preset angle threshold; The preset angle influence factor, This is the preset modulus influence factor.
[0019] Furthermore, if Then it is a point. The true heatmap is regenerated using an isotropic Gaussian distribution.
[0020] Furthermore, the expression for the isotropic Gaussian distribution is:
[0021] in, The variance is an isotropic Gaussian distribution. This represents the Euclidean norm.
[0022] Furthermore, in S3, when training the facial landmark detection network, the loss function for training loss is:
[0023] in, Indicates the total loss; This indicates the degree of difference between the predicted heatmap and the true heatmap; This represents the difference between the ground truth heatmap and the predicted heatmap in the target local region, which is the local region centered on the peak position in the ground truth heatmap and after removing invalid pixels outside the heatmap; This represents the weighting factor.
[0024] Furthermore,
[0025] in, This represents the number of key points in a single face image; for the th... k One key point, It represents the set of pixels within a local area of the target; N Represents a point set The number of pixels contained therein; Represents a point set The ground truth probability distribution obtained by normalizing the heatmap values of each pixel in the ground truth heatmap; Represents a point set The predicted probability distribution obtained by normalizing the heatmap values of each pixel in the predicted heatmap; Represents a point set The heatmap values of each pixel in the ground truth heatmap The weight coefficient distribution obtained after power-normalization.
[0026] According to another aspect of the present invention, a method for detecting facial landmarks is provided, comprising: Input the face image into the pre-trained face landmark detection network to obtain the predicted heatmap of each landmark in the face image; Decode the predicted heatmap of each key point to obtain the coordinates of each key point in the heatmap coordinate system; The coordinates of each key point in the heatmap coordinate system are transformed to the face image coordinate system to obtain the coordinates of each key point in the face image coordinate system, thus completing the face key point detection. The pre-trained facial landmark detection network is trained using the facial landmark detection network training method provided by this invention.
[0027] Furthermore, the decoding of the predicted heatmap for key points includes: Obtain the peak position of the predicted heatmap At peak position Centered on, extract a portion of size [size missing]. local area And remove invalid pixels outside the heatmap to obtain the point set. ; For point sets Each pixel in Calculate its position relative to the peak. relative x-coordinate and relative ordinate And the logarithm of the predicted heatmap value at that pixel. Thus constructing a matrix sum matrix ;matrix sum matrix The expressions are as follows:
[0028]
[0029] The predicted heatmap values at each pixel are set in the point set. Normalization is performed within the inner region to obtain the weight coefficients at each pixel. Thus, the weight matrix is constructed. ; according to Calculate the fitting parameters ; ,and ; according to , Calculate the Gaussian fitting center ; according to , Calculate the coordinates of facial landmarks in the heatmap coordinate system ; in, For local area dimensions, N Represents a point set The number of pixels contained therein. This represents a diagonal matrix.
[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program, which, when executed by a processor, implements the face landmark detection network training method provided by the present invention, or the face landmark detection method provided by the present invention.
[0031] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) Based on the semantic ambiguity characteristics of facial key points at different locations, that is, facial key points located at the boundary endpoints have weaker semantic ambiguity when labeled, while facial key points located inside the boundary have stronger semantic ambiguity when labeled. The invention generates ground truth heatmaps for key points at the boundary endpoints using an isotropic Gaussian distribution, and generates ground truth heatmaps for points inside each boundary using an anisotropic Gaussian distribution. This can approximate the degree of semantic ambiguity in the key point labeling process, providing a supervision signal with rich semantic ambiguity prior information for the training of the facial key point detection network, making the network pay more attention to the error in the direction with low semantic ambiguity, thereby improving the convergence speed of the network and improving the overall performance, and thus effectively improving the accuracy of facial key point detection.
[0032] (2) When generating a ground truth heatmap for facial key points within a boundary using an anisotropic Gaussian distribution, this invention adaptively constructs a Gaussian covariance matrix based on the magnitude and angle relationships between the vectors formed by the key points before and after them, thereby generating an anisotropic key point heatmap. This can more accurately simulate the semantic ambiguity of key points within the boundary. In a further preferred embodiment, the tangential variance used to construct the covariance matrix is calculated based on the key points within the boundary and their preceding and following key points. and normal variance At that time, the angle influence factor was introduced. And modulus influence factor It can adapt to different key points, further improving the accuracy of the simulation.
[0033] (3) The present invention addresses the key points inside the boundary. The keypoint points to the vectors of its preceding and following keypoints. and included angle satisfy At that time, taking this key point into consideration The local curvature is relatively high and the position is relatively fixed, resulting in a weaker degree of semantic ambiguity in the annotation. Therefore, an isotropic Gaussian distribution is used to regenerate the ground truth heatmap, which can more accurately simulate the degree of semantic ambiguity in the actual annotation.
[0034] (4) In training the face keypoint detection network, the present invention designs a joint loss function based on MSE loss and Focal_KL loss. Local regions are cropped based on the peak positions of the ground truth heatmap of keypoints, and Focal_KL loss is applied within these local regions. Specifically, by improving the formula for KL divergence, the ground truth heatmap value is... The power of the power, after being normalized within its neighborhood, is used as a weighting factor. After normalizing both the predicted and ground truth local regions to probability distributions, the Focal_KL loss is calculated, increasing the loss weight near the peak and enhancing the network's sensitivity to peak error. Simultaneously, this loss is weighted with the MSE loss calculated across the entire heatmap to form a joint loss. This approach constrains the network's predicted heatmap at both the distribution and numerical levels, improving the network's accuracy in peak prediction.
[0035] (5) After predicting the heatmap of each key point in the face image, this invention solves for the local Gaussian fitting center by performing weighted least squares fitting on the distribution of the peak position local region in the predicted heatmap, converts it into the predicted coordinates in the heatmap coordinate system, and then converts it back to the original image coordinate system to obtain the decoded position of the key point. The above-mentioned method of fitting within the peak local region is applicable to both isotropic and anisotropic Gaussian distributions. At the same time, this fitting-based method has higher generalization and robustness than the previous fixed decoding method, and the weighted method can also reduce the influence of noise on the fitting, further improving the decoding accuracy of the key point heatmap. Attached Figure Description
[0036] Figure 1 This is a flowchart of a face landmark detection network training method provided in an embodiment of the present invention.
[0037] Figure 2 This is a schematic diagram of loss calculation provided for an embodiment of the present invention.
[0038] Figure 3 This is a flowchart of a facial landmark detection method provided in an embodiment of the present invention. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0040] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0041] To effectively improve the accuracy of facial landmark detection, this invention provides a training method for a facial landmark detection network, a facial landmark detection method, and a medium. The overall concept is to adaptively generate a ground truth heatmap with semantically ambiguous prior information about the keypoints based on the labeled locations of keypoints in the dataset, building upon heatmap encoding methods. This provides better supervision for the training of the facial landmark detection network, improving its prediction performance and convergence speed, thereby effectively improving the accuracy of facial landmark detection based on this network. Furthermore, the training loss function of the facial landmark detection network is improved by designing a joint loss function. This loss function can simultaneously constrain the predicted heatmap of the facial landmark detection network at both the numerical and distributional levels, and enhances the network's sensitivity to peak error, further improving the detection accuracy of facial landmarks.
[0042] The following is an example.
[0043] A method for training a facial landmark detection network, such as Figure 1 As shown, each training round includes: S0: Obtain the face image dataset; in the face image dataset, each sample is a face image with key point coordinates annotated; S1: For each sample in the face image dataset, divide the key points into boundary endpoints and points inside the boundary, and transform the coordinates of each key point to the heatmap coordinate system; S2: In the heatmap coordinate system, generate ground truth heatmaps for each boundary endpoint using an isotropic Gaussian distribution, and generate ground truth heatmaps for each point inside each boundary using an anisotropic Gaussian distribution. The training dataset consists of face images and ground truth heatmaps of each key point therein. S3: Use the training dataset to train a facial landmark detection network to predict the heatmap of each key point in a face image, and obtain the trained facial landmark detection network.
[0044] This embodiment addresses the convergence difficulties and limited detection accuracy of existing facial landmark detection networks (FARTs) due to annotation bias during training. It proposes a more efficient FART training method. Specifically, based on the semantic ambiguity characteristics of facial landmarks at different locations—namely, weaker semantic ambiguity at boundary endpoints and stronger semantic ambiguity inside the boundary—a ground truth heatmap is generated using an isotropic Gaussian distribution for landmarks at boundary endpoints and an anisotropic Gaussian distribution for landmarks inside the boundary. This approximates the degree of semantic ambiguity during landmark annotation, providing a supervisory signal with rich semantic ambiguity prior information for the training of the FART. This allows the network to focus more on errors in directions with lower semantic ambiguity, thereby improving the network's convergence speed and overall performance, and ultimately enhancing the accuracy of facial landmark detection.
[0045] The following provides a further explanation of the specific implementation methods for each step.
[0046] Optionally, in step S1 of this embodiment, the face image dataset is constructed based on the public dataset WFLW, which contains images annotated with facial landmarks and face bounding boxes. In this embodiment, the face image dataset is constructed by cropping face images based on the face bounding boxes in the public dataset, then performing data augmentation methods such as random rotation (60% probability, ±30°), random scaling (1 ± 0.25 times), and random horizontal flipping (50% probability), and finally adjusting them to the same size (here, the resolution is 256×256), thus obtaining face images annotated with facial landmarks. These face images constitute the face image dataset.
[0047] Before generating ground truth heatmaps for each labeled facial key point through heatmap encoding, the coordinates of the facial key points need to be transformed from the image coordinate system to the heatmap coordinate system. Optionally, in step S1 of this embodiment, the ground truth coordinates of all key points are transformed from the original image coordinate system to the heatmap coordinate system through a downsampling operation. The relevant transformation expression is as follows:
[0048] in, and The x and y coordinates represent the true location of key points in the original coordinate system. and The x and y coordinates represent the true location of key points in the heatmap coordinate system. Indicate facial landmarks, This indicates the downsampling factor. Optionally, in this embodiment, the heatmap resolution is 64×64.
[0049] Based on the annotation style of the dataset, the boundary to which each keypoint belongs can be determined. For example, according to the WFLW annotation style, the boundary includes 11 groups such as cheek contours and left and right eyebrows. Furthermore, it can be determined whether the keypoint is located at the endpoint of its boundary or inside it. In step S2 of this embodiment, for facial keypoints located at the boundary endpoints, considering their relatively weak semantic model, an isotropic Gaussian distribution is used to generate ground truth heatmaps for them. The formula for the isotropic Gaussian distribution is:
[0050] in, For pixels in a Gaussian heatmap; For pixels The larger the heatmap value, the greater the likelihood that the point is a key point; The variance is an isotropic Gaussian distribution. Optionally, in this embodiment, .
[0051] In step S2 of this embodiment, for key points located inside the boundary Based on key points and its two adjacent key points before and after. , The ground truth coordinates of the heatmap are obtained from... point to and vector and : , ,
[0052] ,
[0053] Set the normal direction to and The direction pointed to by the sum of these two vectors:
[0054] Obtain the orthogonal tangential direction using a rotation matrix:
[0055] pass The function calculates the angle between two vectors. .
[0056] The normal variance is calculated using the following formula:
[0057] Among them, the first item The basic variance, which is taken as a value here. The second term is based on the vector. and included angle The calculated variance attenuation term is determined by the angle influence factor. To control this, optionally, the value here is 0.1; This is the angle threshold; optionally, it is set to a value here. .
[0058] The tangential variance is calculated using the following formula:
[0059] The second item is based on the included angle. The calculated variance gain term is also influenced by the angle factor. Control is implemented. The third term is based on vectors. and The variance gain term, calculated from the sum of the moduli, is derived from the moduli influence factor. To control this, a value of 0.2 is used here. After obtaining the directions and variances of the tangential and normal axes, the covariance matrix of the anisotropic Gaussian distribution is calculated as follows. : ,
[0060]
[0061] Finally, with the target key points Centered on the network, a ground truth heatmap of keypoints is generated according to the following anisotropic Gaussian formula, which serves as a supervision signal for network training:
[0062] This embodiment differs from conventional fixed heatmap encoding methods. First, based on the annotation style of the dataset, keypoints are grouped according to their respective boundaries. For boundary endpoints, considering their relatively weak semantic ambiguity, the original isotropic Gaussian heatmap encoding method is still used. For points inside the boundary, the tangential and normal directions are calculated based on the positional relationship between the points and their adjacent points. The tangential direction is considered as the direction with higher semantic ambiguity. By combining the magnitude and angle relationship between the two vectors, the variance of the tangential direction is adaptively calculated. At the same time, the variance of the normal direction is calculated based on the angle relationship. By assigning different variance values to the tangential and normal directions to approximate different degrees of semantic ambiguity, this semantically ambiguous prior information is integrated into the generated ground truth heatmap of keypoints. This allows the network to use this prior knowledge for better optimization during the learning process, thereby improving the convergence speed of the face keypoint network.
[0063] In this embodiment, step S2, based on the above-mentioned ground truth heatmap generation method, constructs a training dataset for training the face landmark detection network.
[0064] In a preferred embodiment, in step S2 of this example, for key points inside the boundary, after calculating the vector... and The angle between Then, it will first be compared with the angle threshold. Comparison, taking into account When the local curvature is high, the position of the key point is relatively fixed, and the corresponding semantic ambiguity is weak. Therefore, in this embodiment, for such key points inside the boundary, the isotropic Gaussian distribution is still used to generate the true value heatmap, which can more accurately simulate the semantic ambiguity in the annotation process.
[0065] In step S3 of this embodiment, the facial landmark network can be any kind of facial landmark detection network based on heatmap coding, used to predict the heatmap of each landmark in the input facial image, such as HRNet, ADNet, StackHourglass Networks, PropagationNet, etc. Optionally, in this embodiment, the facial landmark detection network is HRNet.
[0066] To further improve the prediction performance and convergence speed of the face landmark detection network, this embodiment designs a joint training loss function to calculate the loss between the predicted heatmap and the ground truth heatmap, such as... Figure 2 As shown: First, through (Retrieve the location of the maximum value) This operation retrieves the peak position of the keypoint truth heatmap. Using that position as the center, extract a... A local region of a certain size is selected, and invalid pixels outside the heatmap are removed, meaning only points belonging to the heatmap are retained, thus obtaining a point set. .in, For local region size, point set The number of points contained in the data is expressed as N .
[0067] By analyzing the true heatmap on the point set Normalize the heatmap values to obtain the true probability distribution of the local region. Meanwhile, by analyzing the predicted heatmap on the point set... The heatmap value on the soft The operation yields the predicted probability distribution of the local region. For the truth heatmap on the point set Apply an exponential factor to the heatmap values. That is, adjust it to the original heatmap value. The result is then raised to the power of the power and normalized, and used as the distribution of the weight coefficients in the KL divergence calculation formula. Subsequently, the Focal_KL loss is calculated using the following KL divergence formula. :
[0068] in, K This indicates the number of key points in a single face image.
[0069] Further calculation of the MSE loss between the predicted heatmap and the true heatmap :
[0070] in, This represents the number of key points in a single face image. Indicates the height of the heatmap. Indicates the width of the heatmap; This represents the pixel index in the heatmap. Indicates the first The true value of each pixel in the heatmap of key points heatmap values, Indicates the first Pixels in the predicted heatmap of key points The heatmap value.
[0071] Will and Integration results in a combined overall loss. :
[0072] in, This is a weighting factor used to adjust the balance of the loss term. In one optional implementation, The range of values is , The range of values is , The range of values is Preferably, , , The values are 2, 11, and 0.003 respectively.
[0073] This embodiment uses Focal_KL loss, improved based on KL divergence, to constrain the heatmap distribution within the local neighborhood of the peak, thereby increasing the network's sensitivity to peak errors. Simultaneously, it uses MSE loss to numerically constrain the predicted heatmap of the face keypoint detection network. By simultaneously constraining the network at both the numerical and distributional levels, the optimization direction of the network can be better guided, improving the quality and accuracy of the generated predicted heatmap.
[0074] Overall, this embodiment improves the convergence speed of the face landmark network and the quality of the predicted heatmap by incorporating semantic priors into the generated ground truth heatmap and designing a joint loss function, thereby effectively improving the accuracy of face landmark detection.
[0075] Example 2: A facial landmark detection method, such as Figure 3 As shown, it includes: Input the face image into the pre-trained face landmark detection network to obtain the predicted heatmap of each landmark in the face image; Decode the predicted heatmap of each key point to obtain the coordinates of each key point in the heatmap coordinate system; The coordinates of each key point in the heatmap coordinate system are transformed to the face image coordinate system to obtain the coordinates of each key point in the face image coordinate system, thus completing the face key point detection. The pre-trained facial landmark detection network was trained using the facial landmark detection network training method provided in Example 1 above.
[0076] Based on the face landmark detection network training method provided in Embodiment 1 above, the face landmark detection network used in this embodiment can predict the heatmap of each face landmark relatively accurately, so that the decoded face landmarks have high accuracy.
[0077] It's easy to understand that before inputting the face image to be detected into the facial landmark detection network, the face image needs to be preprocessed to make it suitable for the network's processing. Preprocessing includes: cropping the face image according to the face bounding box and adjusting it to the same size (256×256).
[0078] As a preferred implementation, to further improve the accuracy of facial landmark detection, this embodiment proposes a heatmap decoding method based on weighted least squares local Gaussian distribution fitting. Unlike previous conventional decoding methods, this method adaptively fits the parameters of a Gaussian distribution based on the heatmap distribution within the local region of the peak position, and uses this to calculate the center of the local Gaussian distribution. After converting it to predicted coordinates in the heatmap coordinate system, it is then converted back to the original image coordinate system to obtain the decoded position of the landmark. Compared to the previous fixed decoding method, this decoding method has better generalization and robustness, effectively improving decoding accuracy and thus improving the detection accuracy of facial landmarks.
[0079] like Figure 3 As shown, in this embodiment, the decoding of the predicted heatmap of key points includes: First, through (Retrieve the location of the maximum value) operation to obtain the peak position of the key point prediction heatmap. Using that position as the center, extract a... A local region of a certain size is selected, and invalid pixels outside the heatmap are removed to obtain a point set. .in, For local area dimensions, N Represents a point set The number of points included. In one alternative implementation, The range of values is Preferably, The value is 7.
[0080] Secondly, since the key point heatmap is generated based on a Gaussian distribution, the fitting model can be constructed as follows:
[0081] in, and For point set The Middle i The relative x and y coordinates of each point with respect to the peak position. This represents the logarithmic value of the predicted heatmap value for the corresponding location. Let be the parameters to be fitted. Therefore, the weighted least squares fitting objective function is constructed as follows:
[0082] in, These are weighting coefficients, whose values are the predicted heatmap values for the corresponding locations within the point set. The values are normalized. The fitting parameters are calculated using the following formula. :
[0083] in, Since the Gaussian center is the extreme point of the function, the local Gaussian fitting center can be obtained based on its derivative relationship. The solution formula is as follows:
[0084] Finally, based on its relative position to the peak position, it is restored to the predicted coordinates in the heatmap coordinate system:
[0085] Finally, the predicted coordinates in the heatmap coordinate system are transformed to the original image coordinate system to obtain the predicted coordinates of a single key point in the input image.
[0086] The heatmap decoding method based on weighted least squares local Gaussian distribution fitting employed in this embodiment uses only the pixel values of the heatmap within the local region of the peak position when decoding the predicted heatmap obtained by the facial landmark detection network, avoiding the influence of outliers and other abnormal values on the decoding results. Simultaneously, by using weighted least squares to increase the weight of positions near the peak, the interference of random noise from distant locations within the neighborhood is reduced, thereby improving the robustness of the decoding. Finally, this decoding method is applicable to both isotropic and anisotropic Gaussian distributions, exhibiting excellent generalization ability. Therefore, it has high decoding accuracy for each landmark, further improving the detection accuracy of facial landmarks.
[0087] Example 3: A computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the face landmark detection network training method provided in Embodiment 1 above, or the face landmark detection method provided in Embodiment 2 above.
[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for training a facial landmark detection network, characterized in that, include: S0: Obtain a face image dataset; in the face image dataset, each sample is a face image with key point coordinates annotated; S1: For each sample in the face image dataset, the key points are divided into boundary endpoints and points inside the boundary, and the coordinates of each key point are transformed to the heatmap coordinate system; S2: In the heatmap coordinate system, generate ground truth heatmaps for each boundary endpoint using an isotropic Gaussian distribution, and generate ground truth heatmaps for each point inside each boundary using an anisotropic Gaussian distribution. The training dataset consists of face images and ground truth heatmaps of each key point therein. S3: Use the training dataset to train a facial landmark detection network for predicting the heatmap of each key point in a face image, and obtain the trained facial landmark detection network.
2. The face landmark detection network training method as described in claim 1, characterized in that, In S2, the expression for the anisotropic Gaussian distribution is: in, These are the pixel coordinates in the truth heatmap. express Heatmap values of the location; This indicates the coordinates of the key points in the heatmap coordinate system; Let be the covariance matrix of an anisotropic Gaussian distribution; the superscript T indicates matrix transpose.
3. The face landmark detection network training method as described in claim 2, characterized in that, in, Represents the rotation matrix. The local variance matrix is calculated as follows: in, and These are the unit vectors for the tangential and normal directions, respectively; and These are the variances in the tangential and normal directions, respectively.
4. The face landmark detection network training method as described in claim 3, characterized in that, For key points inside the boundary , and ,as well as and according to and its preceding adjacent key point The next adjacent key point Calculations show that , , ; and ,as well as and The expressions are as follows: in, Indicates by point to The vector, Indicates by point to ; Based on the variance, For vectors sum vector The angle between them The preset angle threshold; The preset angle influence factor, This is the preset modulus influence factor.
5. The face landmark detection network training method as described in claim 4, characterized in that, like Then it is a point. The true heatmap is regenerated using an isotropic Gaussian distribution.
6. The face landmark detection network training method according to any one of claims 1 to 5, characterized in that, In S3, the loss function used for training the facial landmark detection network is: in, Indicates the total loss; This indicates the degree of difference between the predicted heatmap and the true heatmap; This represents the difference between the ground truth heatmap and the predicted heatmap in a target local region, where the target local region is a local region centered on the peak position in the ground truth heatmap and after removing invalid pixels outside the heatmap. This represents the weighting factor.
7. The face landmark detection network training method as described in claim 6, characterized in that, in, This represents the number of key points in a single face image; for the th... k One key point, This refers to the set of pixels within the target local area; N Represents a point set The number of pixels contained therein; Represents a point set The ground truth probability distribution obtained by normalizing the heatmap values of each pixel in the ground truth heatmap; Represents a point set The predicted probability distribution obtained by normalizing the heatmap values of each pixel in the predicted heatmap; Represents a point set The heatmap values of each pixel in the ground truth heatmap The weight coefficient distribution obtained after power-normalization.
8. A method for detecting facial landmarks, characterized in that, include: The face image is input into a pre-trained facial landmark detection network to obtain a predicted heatmap of each landmark in the face image. Decode the predicted heatmap of each key point to obtain the coordinates of each key point in the heatmap coordinate system; The coordinates of each key point in the heatmap coordinate system are transformed to the face image coordinate system to obtain the coordinates of each key point in the face image coordinate system, thus completing the face key point detection. The pre-trained facial landmark detection network is trained by the facial landmark detection network training method according to any one of claims 1 to 7.
9. The facial landmark detection method as described in claim 8, characterized in that, Decoding the predictive heatmap of key points, including: Obtain the peak position of the predicted heatmap At peak position Centered on, extract a portion of size [size missing]. local area And remove invalid pixels outside the heatmap to obtain the point set. ; For the point set Each pixel in Calculate its position relative to the peak. relative x-coordinate and relative ordinate And the logarithm of the predicted heatmap value at that pixel. Thus constructing a matrix sum matrix ;matrix sum matrix The expressions are as follows: The predicted heatmap values at each pixel are set in the point set. Normalization is performed within the inner region to obtain the weight coefficients at each pixel. Thus, the weight matrix is constructed. ; according to Calculate the fitting parameters ; ,and ; according to , Calculate the Gaussian fitting center ; according to , Calculate the coordinates of facial landmarks in the heatmap coordinate system ; in, For local area dimensions, N Represents a point set The number of pixels contained therein. This represents a diagonal matrix.
10. A computer-readable storage medium, characterized in that, The system includes a stored computer program, which, when executed by a processor, implements the face landmark detection network training method according to any one of claims 1 to 7, or the face landmark detection method according to claim 8 or 9.