Training methods, devices, and storage media for keypoint detection models

Abstract

This application provides a training method, apparatus, and storage medium for a keypoint detection model. The method uses the tangent direction of the contour curve passing through the facial keypoint at the keypoint as the major axis of an elliptical Gaussian kernel, and the normal direction of the contour curve at the keypoint as the minor axis. Elliptical Gaussian heatmaps are generated for each facial keypoint using the coordinate data of the facial keypoint as the center point coordinates of the elliptical Gaussian kernel. This application trains the model based on these elliptical Gaussian heatmaps, thereby increasing the constraint on the normal direction of the facial edges and reducing the constraint on the tangent direction. This makes the heatmap constraints learned by the keypoint detection model more consistent with the actual distribution of facial keypoints, thus improving the detection accuracy of the keypoint detection model.

Description

Technical Field

[0001] This application relates to the field of facial detection technology, and in particular to a training method, apparatus, storage medium, and computer equipment for a key point detection model. Background Technology

[0002] Facial landmark detection refers to locating the key regions of a given facial image. These key regions can include the outer contour of the face, eyes, nose, and mouth. Facial landmark detection algorithms can be mainly divided into two implementation methods: one is to directly regress facial landmarks using fully connected layers; the other is heatmap regression, which generates heatmaps for each facial landmark and identifies the point with the highest pixel value in each heatmap as the facial landmark. However, the accuracy of facial landmark detection using heatmap regression still needs improvement. Summary of the Invention

[0003] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the low accuracy of facial key point detection in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for training a keypoint detection model, the method comprising:

[0005] Acquire a facial image and its annotation data, wherein the annotation data includes the coordinate data of each pre-annotated facial key point;

[0006] For each of the facial key points, a target contour curve passing through the facial key point is obtained, and the tangent direction and normal direction of the target contour curve at the facial key point are determined. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, and the normal direction is used as the minor axis direction of the elliptical Gaussian kernel. The coordinate data of the facial key point is used as the center point coordinates of the elliptical Gaussian kernel to generate an elliptical Gaussian heatmap corresponding to the facial key point.

[0007] The initial detection model is iteratively trained using the facial image and the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and a key point detection model is obtained.

[0008] In one embodiment, the step of generating an elliptical Gaussian heatmap corresponding to the facial key point, using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, includes:

[0009] Calculate the radius of a circular Gaussian kernel;

[0010] Calculate the first product between a preset first coefficient and the kernel radius, and the second product between a preset second coefficient and the kernel radius; the first coefficient is greater than 1, and the second coefficient is a positive number less than 1;

[0011] Using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the first product as the semi-major axis of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, the second product as the semi-minor axis of the elliptical Gaussian kernel, and using the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, an elliptical Gaussian heatmap corresponding to the facial key point is generated.

[0012] In one embodiment, the first coefficient is 1.25 and the second coefficient is 0.75.

[0013] In one embodiment, the labeled data further includes labeled occlusion attributes for each of the facial key points, and the initial detection model includes a heatmap generation network;

[0014] The step of iteratively training the initial detection model using the facial image and each of the corresponding elliptical Gaussian heatmaps until the output of the initial detection model meets the preset training completion conditions and a keypoint detection model is obtained includes:

[0015] For each facial key point, if the occlusion attribute of the facial key point indicates that the facial key point is occluded, then a preset first weight or a preset second weight is used as the channel loss weight corresponding to the facial key point; otherwise, a preset third weight is used as the channel loss weight corresponding to the facial key point; the first weight and the second weight are both less than the third weight.

[0016] In each round of model training, the facial image is input into the initial detection model, and the predicted Gaussian heatmap generated by the heatmap generation network is obtained respectively. Based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round, the total loss value corresponding to this round of model training is determined. If the total loss value corresponding to this round of model training does not meet the training completion condition, the model parameters of the initial detection model are adjusted and the next round of model training is entered; otherwise, the key point detection model is obtained.

[0017] In one embodiment, the initial detection model further includes an occlusion detection network;

[0018] The step of determining the total loss value corresponding to the model training in this round based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round includes:

[0019] Based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round, calculate the heatmap loss value corresponding to the model training in this round;

[0020] Obtain the detected occlusion attributes of each facial key point generated by the occlusion detection network, and calculate the occlusion detection loss value corresponding to the model training in this round based on each labeled occlusion attribute and each detected occlusion attribute generated in this round.

[0021] Based on the heatmap loss value and the occlusion detection loss value corresponding to this round of model training, determine the total loss value corresponding to this round of model training.

[0022] In one embodiment, the step of calculating the heatmap loss value corresponding to the model training in this round based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round includes:

[0023] Using the Wing Loss function as the loss function, the heatmap loss value corresponding to the model training in this round is calculated based on each of the elliptical Gaussian heatmaps, the loss weights of each of the channels, and the predicted Gaussian heatmaps generated in this round.

[0024] In one embodiment, the step of using a preset first weight or a preset second weight as the channel loss weight corresponding to the facial key point if the occlusion attribute of the facial key point reflects that the facial key point is occluded includes:

[0025] If the occlusion attribute of the facial key point is a self-occlusion attribute, then the first weight is used as the channel loss weight corresponding to the facial key point.

[0026] If the occlusion attribute of the facial key point is an external object occlusion attribute, then the second weight is used as the channel loss weight corresponding to the facial key point; the first weight is less than the second weight.

[0027] Secondly, embodiments of this application provide a training apparatus for a keypoint detection model, the apparatus comprising:

[0028] The data acquisition module is used to acquire a facial image and the annotation data of the facial image, wherein the annotation data includes the coordinate data of each pre-annotated facial key point;

[0029] The elliptical Gaussian heatmap generation module is used to obtain the target contour curve passing through each facial key point, and determine the tangent direction and normal direction of the target contour curve at the facial key point. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, the normal direction is used as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point is used as the center point coordinates of the elliptical Gaussian kernel to generate the elliptical Gaussian heatmap corresponding to the facial key point.

[0030] The model training module is used to iteratively train the initial detection model using the facial image and each of the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and a key point detection model is obtained.

[0031] Thirdly, embodiments of this application provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the training method for the keypoint detection model described in any of the above embodiments.

[0032] Fourthly, embodiments of this application provide a computer device, which includes: one or more processors, and a memory;

[0033] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the training method for the keypoint detection model described in any of the above embodiments.

[0034] In the training method, apparatus, storage medium, and computer device of the keypoint detection model provided in this application, the computer device acquires a facial image and pre-annotated coordinate data of each facial keypoint. For each facial keypoint, the computer device can acquire a target contour curve passing through the keypoint and determine the tangent and normal directions of the contour curve at the keypoint. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, and the normal direction is used as the minor axis direction. The coordinate data of the keypoint is used as the center point coordinates of the elliptical Gaussian kernel to generate an elliptical Gaussian heatmap corresponding to the keypoint. The computer device can use the facial image and the elliptical Gaussian heatmaps corresponding to each keypoint to train an initial detection model to obtain a keypoint detection model. Compared to the tangent direction of the facial contour, the heatmap generated by the keypoint detection model needs to be narrowed down to a smaller area along the normal direction of the facial contour. This means the keypoint detection model increases constraints along the normal direction of the facial edges and reduces constraints along the tangent direction, making the heatmap constraints learned by the keypoint detection model more closely match the actual distribution of facial keypoints, thus resulting in more accurate facial keypoints output by the model. Furthermore, the solution proposed in this application does not increase the difficulty of model inference and is applicable to various scenarios. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is one of the schematic diagrams showing the arrangement of facial key points in one embodiment;

[0037] Figure 2 This is a second schematic diagram showing the arrangement of facial key points in one embodiment;

[0038] Figure 3 This is a flowchart illustrating the training method of a keypoint detection model in one embodiment;

[0039] Figure 4 This is a schematic diagram of the model structure of the initial detection model in one embodiment;

[0040] Figure 5 This is a flowchart illustrating the steps for generating an elliptical Gaussian heatmap in one embodiment.

[0041] Figure 6 This is a schematic diagram of the structure of a training device for a key point detection model in one embodiment;

[0042] Figure 7 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] As mentioned in the background section, the accuracy of existing technologies for detecting facial keypoints using heatmap regression still needs improvement. The inventors' research revealed that this problem stems from uncertainties in the annotation data of facial keypoints along the facial edges within the model training dataset. For example, keypoints on the outer contour of the face cannot be distributed perfectly evenly, or the annotated keypoints on the outer contour cannot be precisely located at the very edge of the face. Furthermore, existing technologies generate the heatmap dataset for the model training dataset based on a circular Gaussian kernel function. In other words, the radius of the Gaussian kernel is the same in every direction, resulting in identical constraints on the trained keypoint detection model across all directions.

[0045] However, the tolerance for facial landmark detection varies in different directions. Figure 1 and Figure 2 For example, when the trained keypoint detection model outputs... Figure 1 When multiple facial keypoints are shown, although the distribution of these keypoints is not uniform, the output is generally considered acceptable because each keypoint conforms to the outer contour of the face. When the trained keypoint detection model outputs... Figure 2 When multiple facial key points are shown, although the distribution of each facial key point is relatively... Figure 1 The results are uniform, but because some facial key points deviate from the actual facial area, this output is generally considered unacceptable.

[0046] Existing technologies do not consider the uncertainty of labeled data when generating heatmap sets for model training, resulting in the same constraints on keypoint detection models in all directions after training. This increases the difficulty of heatmap regression and reduces detection accuracy.

[0047] To address the aforementioned problems, this application provides a training method, apparatus, storage medium, and computer device for a keypoint detection model. This application determines the tangent and normal directions of the contour line at each pre-annotated facial keypoint, and uses the tangent and normal directions corresponding to each facial keypoint as parameters of an elliptical Gaussian kernel to generate an elliptical Gaussian heatmap corresponding to each facial keypoint. The initial detection model is then trained using these elliptical Gaussian heatmaps to obtain the keypoint detection model. This increases the constraint of the keypoint detection model on the normal direction of the facial edges and reduces the constraint on the tangent direction, making the heatmap constraints learned by the keypoint detection model more consistent with the actual distribution of facial keypoints, thus enabling the keypoint detection model to output more accurate facial keypoints.

[0048] In one embodiment, this application provides a method for training a keypoint detection model, which can be applied to a computer device. The computer device refers to a device with data processing capabilities, and may be, but is not limited to, a laptop computer, a desktop computer, a single server, or a server cluster consisting of multiple servers. Figure 3 As shown, the training method for the keypoint detection model in this application may include the following steps:

[0049] S302: Obtain the facial image and its annotation data, which includes the coordinate data of each pre-annotated facial key point.

[0050] The facial image refers to an image recording at least one face. Each face recorded in the facial image may be unobstructed or partially obstructed, and this application does not impose specific limitations on this. Furthermore, each face can be recorded in the facial image from any angle, such as an upward angle, a downward angle, or a side angle.

[0051] Furthermore, the facial image described herein can be a facial image that has undergone data augmentation processing. In other words, after obtaining the original image and the corresponding pre-annotated data, the computer device can use any one or more data augmentation algorithms to process the original image to obtain the facial image. In one embodiment, the computer device can also obtain the annotation data of the facial image based on the adopted data augmentation algorithm and the pre-annotated data. The data augmentation algorithms include, but are not limited to, image rotation and image scaling.

[0052] The annotation data for facial images includes the coordinates of each facial keypoint within the image. These coordinates can be pre-annotated manually or calculated using the data augmentation algorithm and the original annotation data; this paper does not impose specific limitations on this. It is understood that the number of facial keypoints corresponding to each face can be determined based on the actual situation, for example, it could be 68, 98, or more.

[0053] In one embodiment, S302 may include the following steps:

[0054] Step A1: Obtain the initial image and input it into the face detection model so that the face detection model can detect the facial region in the initial image and output the facial bounding box position information based on the facial region;

[0055] Step A2: Based on the facial bounding box position information output by the facial detection model, crop the facial region of the initial image to obtain the facial image;

[0056] Step A3: Obtain the annotation data of the facial image, which includes the coordinate data of each facial key point and the occlusion attribute of each facial key point. This annotation data is obtained by manual annotation.

[0057] S304: For each facial key point, obtain the target contour curve passing through the facial key point, and determine the tangent direction and normal direction of the target contour curve at the facial key point. Use the tangent direction as the major axis direction of the elliptical Gaussian kernel, use the normal direction as the minor axis direction of the elliptical Gaussian kernel, and use the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel to generate the elliptical Gaussian heatmap corresponding to the facial key point.

[0058] Among them, an elliptical Gaussian heatmap refers to a Gaussian heatmap with an elliptical Gaussian kernel. The elliptical Gaussian kernel is an elliptical Gaussian kernel; in other words, the semi-major axis of the elliptical Gaussian kernel is greater than its semi-minor axis. The target contour curve corresponding to each facial key point refers to the contour curve passing through that facial key point.

[0059] Furthermore, the contour curves can be mainly divided into facial edge contour curves, eye contour curves, eyebrow contour curves, and nose contour curves. Before executing S304, the computer device can determine the number of curves to be fitted based on the total number of facial key points and the distribution of each facial key point, and perform curve fitting based on the coordinate data of each facial key point to obtain each contour curve.

[0060] For example, when a facial image is labeled with 68 facial key points, the computer device can fit a chin contour curve based on the coordinate data of facial key points 0 to 16, a right eyebrow contour curve based on the coordinate data of facial key points 17 to 21, a left eyebrow contour curve based on the coordinate data of facial key points 22 to 26, a nose bridge contour curve based on the coordinate data of facial key points 27 to 30, a lower nose contour curve based on the coordinate data of facial key points 31 to 35, a right eye contour curve based on the coordinate data of facial key points 36 to 41, a left eye contour curve based on the coordinate data of facial key points 42 to 47, a first mouth contour curve based on the coordinate data of facial key points 48 to 59, and a second mouth contour curve based on the coordinate data of facial key points 60 to 67.

[0061] In this step, the computer device needs to generate an elliptical Gaussian heatmap corresponding to each facial key point based on the pre-labeled coordinate data. Specifically, for each facial key point, the computer device can execute steps B1 to B3 to generate the corresponding elliptical Gaussian heatmap:

[0062] Step B1: Obtain the target contour curve passing through the facial key point. In one embodiment, the computer device can determine the contour curve passing through the facial key point from the generated contour curves, and the determined contour curve is the target contour curve.

[0063] Step B2: Determine the tangent and normal directions of the target contour curve at the facial key point. If the target contour curve is Y and the coordinates of the facial key point are (x0, y0), then the computer device needs to determine the tangent and normal directions of Y at (x0, y0).

[0064] Step B3: Using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the normal direction as the minor axis direction, and the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, generate an elliptical Gaussian heatmap corresponding to the facial key point. After determining the major axis direction, minor axis direction, and center point coordinates of the elliptical Gaussian kernel, the computer device can generate the corresponding elliptical Gaussian heatmap based on a two-dimensional elliptical Gaussian function. In one embodiment, the computer device can randomly determine the semi-major axis and semi-minor axis of the elliptical Gaussian kernel, where the semi-major axis is greater than the semi-minor axis. The computer device can then generate the corresponding elliptical Gaussian heatmap using a two-dimensional elliptical Gaussian function based on the major axis direction, semi-major axis, minor axis direction, semi-minor axis, and center point coordinates.

[0065] S306: The initial detection model is iteratively trained using the facial image and the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and the key point detection model is obtained.

[0066] Specifically, the computer device can train an initial detection model using a facial image and its corresponding elliptical Gaussian heatmaps, ensuring that the Gaussian heatmaps generated by the initial detection model approximate elliptical Gaussian heatmaps. When the performance of the initial detection model meets preset training completion conditions, the computer device determines that training is complete and uses the trained initial detection model as a keypoint detection model. This keypoint detection model can be used to receive facial images and perform facial keypoint detection based on heatmap regression, outputting information about each facial keypoint.

[0067] It is understandable that the model structure of the initial detection model can be determined according to the actual situation. This application does not impose specific restrictions on this, as long as the initial detection model can perform facial key point detection based on Gaussian heatmap regression.

[0068] In one embodiment, such as Figure 4 As shown, the initial detection model can include a backbone network, a feature fusion network, and a heatmap generation network. The backbone network is used for feature extraction, extracting multi-scale feature maps from the input facial image. The feature fusion network fuses and upsamples the multi-scale feature maps extracted by the backbone network to obtain fused features. The heatmap generation network generates Gaussian heatmaps with N channels based on the fused features, facilitating the determination of the coordinates of each facial keypoint based on the N Gaussian heatmaps, where N is the total number of facial keypoints to be detected. In one example, the backbone network can be a CNN (Convolutional Neural Network), and the heatmap generation network can be composed of multiple stacked convolutional layers.

[0069] In this application, a computer device acquires a facial image and pre-annotated coordinate data of each facial key point. For each facial key point, the computer device can acquire a target contour curve passing through that key point and determine the tangent and normal directions of the target contour curve at that key point. The tangent direction is used as the major axis of an elliptical Gaussian kernel, and the normal direction is used as the minor axis. The coordinate data of the facial key point is used as the center coordinates of the elliptical Gaussian kernel to generate an elliptical Gaussian heatmap corresponding to that key point. The computer device can use the facial image and the elliptical Gaussian heatmaps corresponding to each key point to train an initial detection model to obtain a key point detection model. Compared to the tangent direction of the facial contour, the heatmap generated by the key point detection model needs to be narrowed down to a smaller range along the normal direction of the facial contour. That is, the key point detection model increases the constraint along the normal direction of the facial edge and reduces the constraint along the tangent direction, making the heatmap constraints learned by the key point detection model more consistent with the actual distribution of facial key points, thus making the facial key points output by the key point detection model more accurate.

[0070] In one embodiment, such as Figure 5 As shown, the steps for generating an elliptical Gaussian heatmap corresponding to a facial key point, using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point as the coordinates of the center point of the elliptical Gaussian kernel, include:

[0071] S502: Calculate the radius of a circular Gaussian kernel.

[0072] Specifically, the computer device can use any method disclosed in the prior art to calculate the kernel radius of a circular Gaussian kernel, and this application does not make any specific limitations in this regard.

[0073] S504: Calculate the first product between the preset first coefficient and the kernel radius, and the second product between the preset second coefficient and the kernel radius; the first coefficient is greater than 1, and the second coefficient is a positive number less than 1.

[0074] Since the first coefficient is greater than 1 and the second coefficient is a positive number less than 1, the first product is greater than the kernel radius, the second product is less than the kernel radius, and the first product is greater than the second product. It is understood that the specific values ​​of the first and second coefficients in this application can be determined based on actual circumstances. In one example, the first coefficient is 1.25 and the second coefficient is 0.75 to further improve the accuracy of facial key point detection.

[0075] S506: Using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the first product as the semi-major axis of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, the second product as the semi-minor axis of the elliptical Gaussian kernel, and using the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, generate an elliptical Gaussian heatmap corresponding to the facial key point.

[0076] In other words, the computer device uses the first product as the radius in the tangential direction and the second product as the radius in the normal direction to generate an elliptical Gaussian heatmap.

[0077] In this embodiment, by using the radius of the circular Gaussian kernel as a basis to determine the semi-major axis and semi-minor axis of the elliptical Gaussian kernel, the elliptical heatmap can more accurately represent the positional distribution of the corresponding facial key points, thereby further improving the detection accuracy of the key point detection model.

[0078] In one embodiment, the annotation data also includes annotation occlusion attributes for each facial keypoint. These occlusion attributes are pre-annotated information reflecting whether the corresponding facial keypoint is occluded. When the annotation occlusion attribute of a facial keypoint indicates that the keypoint is occluded, it means that the keypoint is difficult to annotate accurately; for example, self-occlusion, occlusion by external objects, or exceeding image boundaries can all be considered occlusion. When the annotation occlusion attribute of a facial keypoint indicates that the keypoint is not occluded, it means that the keypoint is visible in the facial image.

[0079] The initial detection model includes a heatmap generation network, which is a network used to output a Gaussian heatmap corresponding to each facial key point. For a detailed description, please refer to the above embodiments, which will not be repeated here.

[0080] The steps for iteratively training the initial detection model using facial images and their corresponding elliptical Gaussian heatmaps until the output of the initial detection model meets the preset training completion conditions and a keypoint detection model is obtained include:

[0081] For each facial key point, if the occlusion attribute of the facial key point indicates that the facial key point is occluded, then the preset first weight or the preset second weight is used as the channel loss weight corresponding to the facial key point; otherwise, the preset third weight is used as the channel loss weight corresponding to the facial key point; the first weight and the second weight are both less than the third weight.

[0082] In each round of model training, facial images are input into the initial detection model, and predicted Gaussian heatmaps generated by the heatmap generation network are obtained. Based on each elliptical Gaussian heatmap, the loss weights of each channel, and each predicted Gaussian heatmap generated in this round, the total loss value corresponding to this round of model training is determined. If the total loss value corresponding to this round of model training does not meet the training completion condition, the model parameters of the initial detection model are adjusted and the model training enters the next round; otherwise, the key point detection model is obtained.

[0083] Specifically, during model training, if occluded facial keypoints and unoccluded facial keypoints use the same channel loss weights, it will hinder the initial learning of the detection model. Therefore, the computer device can determine the channel loss weight corresponding to each facial keypoint based on the labeled occlusion attribute, so that occluded facial keypoints correspond to smaller channel loss weights and unoccluded facial keypoints correspond to larger channel loss weights, thereby reducing the channel loss corresponding to occluded facial keypoints.

[0084] In each training iteration, the computer can input facial images into the initial detection model to generate predicted Gaussian heatmaps for each facial keypoint using the initial model's heatmap generation network. The computer can then calculate the total loss for this training iteration based on each elliptical Gaussian heatmap, the channel loss weights corresponding to each facial keypoint, and the predicted Gaussian heatmap generated by the heatmap generation network in this iteration. The total loss for this iteration determines whether the initial detection model has been successfully trained. If so, the trained initial detection model is used as the keypoint detection model. If not, the model parameters of the initial detection model are adjusted, and the model enters the next training iteration.

[0085] In this embodiment, considering that the localization accuracy of occluded facial key points is lower than that of unoccluded facial key points, this application reduces the channel loss weight corresponding to occluded facial key points to alleviate the problem of excessive loss of occluded facial key points, thereby further improving the detection accuracy of the key point detection model.

[0086] In one embodiment, if the occlusion attribute of the facial key point reflects that the facial key point is occluded, the step of using a preset first weight or a preset second weight as the channel loss weight corresponding to the facial key point includes:

[0087] If the occlusion attribute of the facial key point is self-occlusion, then the first weight is used as the channel loss weight corresponding to the facial key point.

[0088] If the occlusion attribute of the facial key point is an external object occlusion attribute, then the second weight is used as the channel loss weight corresponding to the facial key point; the first weight is less than the second weight.

[0089] Self-occlusion refers to the situation where some facial key points are not visible in a facial image due to limitations in facial angle. For example, when the facial image is a left-side face image, some facial key points on the right side of the face are not visible. The occlusion property of these invisible facial key points can be considered a self-occlusion property.

[0090] External object occlusion refers to a situation where part of the face is obscured by an object other than the face of the person being photographed. For example, when the facial image is a frontal image, part of the face may be obscured by a hand, another face, or any other object. Facial keypoints located on the obscured portion of the face can be facial keypoints with external object occlusion attributes.

[0091] Since self-occluded facial keypoints are invisible in facial images, their annotation is more difficult. In contrast, facial keypoints with external occlusion attributes are visible in facial images, allowing annotators to roughly determine their coordinates. Therefore, this application can apply different channel loss weights to facial keypoints with different occlusion levels, ensuring that the channel loss weight corresponding to self-occluded facial keypoints is less than that corresponding to external occlusion attributes. In other words, the channel loss weight corresponding to self-occluded facial keypoints is the first weight, the channel loss weight corresponding to external occlusion attributes is the second weight, and the channel loss weight corresponding to unoccluded facial keypoints is the third weight, with the first weight less than the second weight, and the second weight less than the third weight.

[0092] In this embodiment, different channel loss weights are used for different occlusion situations, so as to take into account the impact of different occlusion situations on model training, thereby further alleviating the problem of excessive loss of occluded facial key points and further improving the detection accuracy of the key point detection model.

[0093] In one embodiment, the initial detection model further includes an occlusion detection network, which is a network used to detect whether each facial key point is occluded and output the occlusion attribute of each facial key point. Its specific implementation can be determined according to the actual situation, and this application does not make specific limitations on it.

[0094] Furthermore, the occlusion detection network can be a binary classification network to distinguish whether facial key points are occluded or not. The output of the fully connected layers of the occlusion detection network can use the sigmoid activation function to generate a visibility score for each facial key point. For each facial key point, if the visibility score of the facial key point is greater than a preset score threshold, the occlusion detection network can determine the occlusion attribute of the facial key point as an unoccluded attribute; otherwise, the occlusion attribute of the facial key point is determined as an occluded attribute.

[0095] The steps for determining the total loss value for this round of model training based on each elliptical Gaussian heatmap, the loss weights for each channel, and each predicted Gaussian heatmap generated in this round include:

[0096] Based on each elliptical Gaussian heatmap, the loss weights of each channel, and each predicted Gaussian heatmap generated in this round, calculate the heatmap loss value corresponding to the model training in this round;

[0097] Obtain the detected occlusion attributes of each facial key point generated by the occlusion detection network, and calculate the occlusion detection loss value corresponding to the model training in this round based on each labeled occlusion attribute and each detected occlusion attribute generated in this round.

[0098] Based on the heatmap loss value and the occlusion detection loss value corresponding to this round of model training, determine the total loss value corresponding to this round of model training.

[0099] Since the initial detection model includes a heatmap generation network and an occlusion detection network, to complete the training of both networks, in each training round, the computer can calculate the heatmap loss value corresponding to the current training round based on the predicted Gaussian heatmaps generated by the heatmap generation network, and calculate the occlusion detection loss value corresponding to the current training round based on the detected occlusion attributes generated by the occlusion detection network. The heatmap loss value reflects the loss value of the heatmap generation network in this training round, and the occlusion detection loss value reflects the loss value of the occlusion detection network. Based on the heatmap loss value and occlusion detection loss value corresponding to this training round, the computer can determine the total loss value corresponding to this training round, so that the determined loss value can be used to determine whether the performance of both the heatmap generation network and the occlusion detection network meets the preset training completion conditions.

[0100] It is understood that this application can adopt any method to determine the total loss value, as long as the total loss value is determined based on the heatmap loss value and the occlusion detection loss value. Furthermore, the computer device can sum the heatmap loss value and the occlusion detection loss value corresponding to this round of model training to obtain the total loss value of this round of model training. This allows the total loss value to simultaneously reflect the loss values ​​of the heatmap generation network and the occlusion detection network, and simplifies the calculation method of the total loss value, thereby improving the model training speed.

[0101] In one embodiment, the computer device can use the Wing Loss function as the loss function to calculate the heatmap loss value corresponding to each round of model training. Since the Wing Loss function is more suitable for facial key point detection tasks, calculating the heatmap loss value using the Wing Loss function and training the initial detection model based on the calculated heatmap loss value can improve the detection accuracy of the key point detection model.

[0102] In one embodiment, the computer device can use the mean squared error function (i.e., the MSE loss function) as the loss function to calculate the heatmap loss value corresponding to each round of model training, so as to reduce the amount of computation in the model training process and thus improve the model training efficiency.

[0103] In one embodiment, the computer device can calculate the occlusion detection loss value corresponding to each round of model training based on the binary cross-entropy loss function.

[0104] The training apparatus for the key point detection model provided in the embodiments of this application will be described below. The training apparatus for the key point detection model described below and the training method for the key point detection model described above can be referred to in correspondence.

[0105] In one embodiment, this application provides a training apparatus 600 for a keypoint detection model. For example... Figure 6 As shown, the training device 600 for the keypoint detection model includes a data acquisition module 610, an elliptic Gaussian heatmap generation module 620, and a model training module 630. Wherein:

[0106] The data acquisition module 610 is used to acquire facial images and annotation data of facial images. The annotation data includes the coordinate data of each pre-annotated facial key point.

[0107] The elliptical Gaussian heatmap generation module 620 is used to obtain the target contour curve passing through each facial key point, and determine the tangent direction and normal direction of the target contour curve at the facial key point. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, the normal direction is used as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point is used as the center point coordinates of the elliptical Gaussian kernel to generate the elliptical Gaussian heatmap corresponding to the facial key point.

[0108] The model training module 630 is used to iteratively train the initial detection model using the facial image and the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and the key point detection model is obtained.

[0109] In one embodiment, the elliptical Gaussian heatmap generation module 620 includes a kernel radius calculation unit, a product calculation unit, and a heatmap generation unit. The kernel radius calculation unit calculates the kernel radius of a circular Gaussian kernel. The product calculation unit calculates a first product between a preset first coefficient and the kernel radius, and a second product between a preset second coefficient and the kernel radius; the first coefficient is greater than 1, and the second coefficient is a positive number less than 1. The heatmap generation unit generates an elliptical Gaussian heatmap corresponding to the facial key point, using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the first product as the semi-major axis direction of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, the second product as the semi-minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point as the coordinates of the center point of the elliptical Gaussian kernel.

[0110] In one embodiment, the first coefficient is 1.25 and the second coefficient is 0.75.

[0111] In one embodiment, the labeled data further includes the labeled occlusion attributes of each facial keypoint, and the initial detection model includes a heatmap generation network. The model training module 630 of this application includes a channel loss weight determination unit and a training unit. The channel loss weight determination unit, for each facial keypoint, determines the channel loss weight based on whether the labeled occlusion attribute reflects that the keypoint is occluded; if so, a preset first weight or a preset second weight is used. Otherwise, a preset third weight is used. Both the first and second weights are less than the third weight. The training unit, in each round of model training, inputs a facial image into the initial detection model and obtains the predicted Gaussian heatmaps generated by the heatmap generation network. Based on each elliptical Gaussian heatmap, each channel loss weight, and each predicted Gaussian heatmap generated in this round, it determines the total loss value corresponding to this round of model training. If the total loss value corresponding to this round of model training does not meet the training completion condition, the model parameters of the initial detection model are adjusted, and the model training proceeds to the next round; otherwise, a keypoint detection model is obtained.

[0112] In one embodiment, the channel loss weight determination unit of this application includes a first occlusion attribute weight determination unit and a second occlusion attribute weight determination unit. The first occlusion attribute weight determination unit is used to determine the channel loss weight corresponding to the facial keypoint using a first weight when the labeled occlusion attribute of the facial keypoint is a self-occlusion attribute. The second occlusion attribute weight determination unit is used to determine the channel loss weight corresponding to the facial keypoint using a second weight when the labeled occlusion attribute of the facial keypoint is an external object occlusion attribute; the first weight is less than the second weight.

[0113] In one embodiment, the initial detection model further includes an occlusion detection network. The training unit of this application includes a heatmap loss calculation unit, an occlusion detection loss calculation unit, and a total loss calculation unit. The heatmap loss calculation unit calculates the heatmap loss value corresponding to the current round of model training based on each elliptical Gaussian heatmap, each channel loss weight, and each predicted Gaussian heatmap generated in this round. The occlusion detection loss calculation unit obtains the detected occlusion attributes of each facial keypoint generated by the occlusion detection network, and calculates the occlusion detection loss value corresponding to the current round of model training based on each labeled occlusion attribute and each detected occlusion attribute generated in this round. The total loss calculation unit determines the total loss value corresponding to the current round of model training based on the heatmap loss value and the occlusion detection loss value corresponding to the current round of model training.

[0114] In one embodiment, the heatmap loss calculation unit is also used to calculate the heatmap loss value corresponding to the model training in this round, based on each elliptical Gaussian heatmap, each channel loss weight, and each predicted Gaussian heatmap generated in this round, using the Wing Loss function as the loss function.

[0115] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of a training method for a keypoint detection model as described in any embodiment.

[0116] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform steps of a training method for a keypoint detection model as described in any embodiment.

[0117] Indicatively, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. In one example, the computer device can be a server. (Refer to...) Figure 7 The computer device 900 includes a processing component 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions, such as application programs, that can be executed by the processing component 902. The application programs stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 902 is configured to execute instructions to perform the steps of the training method for the keypoint detection model described in any of the above embodiments.

[0118] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0119] Those skilled in the art will understand that the internal structure of the computer device shown in this application is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0121] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0122] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a key point detection model, the method comprising: The method includes: Acquire a facial image and its annotation data, wherein the annotation data includes the coordinate data of each pre-annotated facial key point; Based on the total number of facial key points and their distribution, the number of curves to be fitted is determined, and curve fitting is performed based on the coordinate data of each facial key point to obtain each contour curve. For each facial key point, a target contour curve passing through that facial key point is determined from the contour curves, and the tangent direction and normal direction of the target contour curve at that facial key point are determined. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, and the normal direction is used as the minor axis direction of the elliptical Gaussian kernel. The coordinate data of the facial key point is used as the center point coordinates of the elliptical Gaussian kernel to generate an elliptical Gaussian heatmap corresponding to that facial key point. The elliptical Gaussian heatmap refers to a Gaussian heatmap with an elliptical Gaussian kernel. The initial detection model is iteratively trained using the facial image and the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and a key point detection model is obtained.

2. The training method for the keypoint detection model according to claim 1, characterized in that, The step of generating an elliptical Gaussian heatmap corresponding to the facial key point, using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, includes: Calculate the radius of a circular Gaussian kernel; Calculate the first product between a preset first coefficient and the kernel radius, and the second product between a preset second coefficient and the kernel radius; the first coefficient is greater than 1, and the second coefficient is a positive number less than 1; Using the tangent direction as the major axis direction of the elliptical Gaussian kernel, the first product as the semi-major axis of the elliptical Gaussian kernel, the normal direction as the minor axis direction of the elliptical Gaussian kernel, the second product as the semi-minor axis of the elliptical Gaussian kernel, and using the coordinate data of the facial key point as the center point coordinates of the elliptical Gaussian kernel, an elliptical Gaussian heatmap corresponding to the facial key point is generated.

3. The training method for the keypoint detection model according to claim 2, characterized in that, The first coefficient is 1.25, and the second coefficient is 0.

75.

4. The training method for the keypoint detection model according to claim 1, characterized in that, The labeled data also includes the labeled occlusion attributes of each of the facial key points, and the initial detection model includes a heatmap generation network; The step of iteratively training the initial detection model using the facial image and each of the corresponding elliptical Gaussian heatmaps until the output of the initial detection model meets the preset training completion conditions and a keypoint detection model is obtained includes: For each facial key point, if the occlusion attribute of the facial key point indicates that the facial key point is occluded, then a preset first weight or a preset second weight is used as the channel loss weight corresponding to the facial key point; otherwise, a preset third weight is used as the channel loss weight corresponding to the facial key point; the first weight and the second weight are both less than the third weight. In each round of model training, the facial image is input into the initial detection model, and the predicted Gaussian heatmap generated by the heatmap generation network is obtained respectively. Based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round, the total loss value corresponding to this round of model training is determined. If the total loss value corresponding to this round of model training does not meet the training completion condition, the model parameters of the initial detection model are adjusted and the next round of model training is entered; otherwise, the key point detection model is obtained.

5. The training method for the keypoint detection model according to claim 4, characterized in that, The initial detection model also includes an occlusion detection network; The step of determining the total loss value corresponding to the model training in this round based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round includes: Based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round, calculate the heatmap loss value corresponding to the model training in this round; Obtain the detected occlusion attributes of each facial key point generated by the occlusion detection network, and calculate the occlusion detection loss value corresponding to the model training in this round based on each labeled occlusion attribute and each detected occlusion attribute generated in this round. Based on the heatmap loss value and the occlusion detection loss value corresponding to this round of model training, determine the total loss value corresponding to this round of model training.

6. The training method for the keypoint detection model according to claim 5, characterized in that, The step of calculating the heatmap loss value corresponding to the model training in this round based on each of the elliptical Gaussian heatmaps, each of the channel loss weights, and each of the predicted Gaussian heatmaps generated in this round includes: Using the Wing Loss function as the loss function, the heatmap loss value corresponding to the model training in this round is calculated based on each of the elliptical Gaussian heatmaps, the loss weights of each of the channels, and the predicted Gaussian heatmaps generated in this round.

7. The training method for the keypoint detection model according to any one of claims 4 to 6, characterized in that, The step of using a preset first weight or a preset second weight as the channel loss weight corresponding to the facial key point if the occlusion attribute of the facial key point indicates that the facial key point is occluded includes: If the occlusion attribute of the facial key point is self-occlusion, then the first weight is used as the channel loss weight corresponding to the facial key point. If the occlusion attribute of the facial key point is an external object occlusion attribute, then the second weight is used as the channel loss weight corresponding to the facial key point; the first weight is less than the second weight.

8. A training device for a key point detection model, characterized in that, The device includes: The data acquisition module is used to acquire a facial image and the annotation data of the facial image, wherein the annotation data includes the coordinate data of each pre-annotated facial key point; The elliptical Gaussian heatmap generation module is used to determine the number of curves to be fitted based on the total number of facial key points and the distribution of each facial key point, and to perform curve fitting based on the coordinate data of each facial key point to obtain each contour curve; for each facial key point, a target contour curve passing through the facial key point is determined from the contour curves, and the tangent direction and normal direction of the target contour curve at the facial key point are determined. The tangent direction is used as the major axis direction of the elliptical Gaussian kernel, the normal direction is used as the minor axis direction of the elliptical Gaussian kernel, and the coordinate data of the facial key point is used as the center point coordinates of the elliptical Gaussian kernel to generate the elliptical Gaussian heatmap corresponding to the facial key point; the elliptical Gaussian heatmap refers to a Gaussian heatmap with an elliptical Gaussian kernel. The model training module is used to iteratively train the initial detection model using the facial image and each of the corresponding elliptical Gaussian heatmaps until the initial detection model meets the preset training completion conditions and a key point detection model is obtained.

9. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the training method for the keypoint detection model as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the training method for the keypoint detection model as described in any one of claims 1 to 7.

Images