An occlusion-robust face keypoint detection method

Abstract

This invention relates to an occlusion-robust facial landmark detection method, comprising: S1, acquiring an image to be detected; S2, converting the image to be detected into a sequence of blocks; S3, inputting the sequence of blocks into a trained facial landmark detection model, wherein the trained facial landmark detection model outputs the facial landmark coordinate detection results, the trained facial landmark detection model including a Transformer encoder, a CNN decoder, and an LFC regressor head. Compared with existing technologies, this invention has advantages such as improved robustness to occlusion.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to a method for detecting facial key points that is robust to occlusion. Background Technology

[0002] Facial landmark detection is a crucial step in face recognition and analysis, and a prerequisite for other face-related problems such as expression recognition, face editing, and 3D face reconstruction. However, current mainstream heatmap-based facial landmark detection algorithms often generate an isolated heatmap for each landmark, leading to a failure of the model to effectively learn the relationships between landmarks. Furthermore, when humans annotate facial landmarks, the annotation of occluded landmarks often relies on the structural constraints of the face itself, meaning the position and size of facial features are relatively fixed. However, most existing methods ignore this. Therefore, how to better learn the relationships between landmarks and fully utilize the strong structural constraints of the face to enhance the model's performance under occlusion conditions has become a research problem.

[0003] The commonly used face detection scheme is PIPNet. This scheme first defines key points with the minimum Euclidean distance on the average face as their neighbors. Then, it forces neighboring points to share local features and perform initial key point localization through small-scale heatmap regression. Finally, it adjusts each key point to its final position using coordinate regression, thus realizing a face key point detection algorithm that combines the advantages of heatmap regression and coordinate regression. Although coordinate regression can better capture the interrelationships between key points, it is difficult to cope with complex and changing environmental factors, especially severe occlusion, because its neighboring points are defined as key points with the minimum Euclidean distance on the average face, which leads to a decrease in recognition accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide an occlusion-robust facial key point detection method to achieve face detection under severe occlusion conditions and enhance the robustness to occlusion. By grouping face components, constructing strong structural constraints on the face itself, and performing two-stage localization of coarse and fine localization, the model's facial structure reasoning ability is enhanced, thereby improving its robustness to occlusion.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A robust method for facial landmark detection under occlusion, comprising:

[0007] S1. Obtain the image to be detected;

[0008] S2. Convert the image to be detected into sequence blocks;

[0009] S3. Input the sequence block into the trained facial landmark detection model. The trained facial landmark detection model outputs the facial landmark coordinate detection results. The trained facial landmark detection model includes a Transformer encoder, a CNN decoder, and an LFC regression head.

[0010] The specific steps of S3 are as follows:

[0011] S31. Input the sequence block into the Transformer encoder for feature extraction;

[0012] S32. Input the extracted features into the CNN decoder, upsample the extracted features, and obtain the upsampled features.

[0013] S33. Input the upsampled features into the two branches of the LFC regression head, where the component branch outputs the component heatmap according to the heatmap regression method, and the offset branch outputs the offset heatmap.

[0014] S34. Calculate the facial key point coordinate detection results based on the component heatmap and offset heatmap.

[0015] Furthermore, the calculation of facial key point coordinate detection results based on the component heatmap and offset heatmap is specifically as follows:

[0016] The coordinates of face components are obtained based on component heatmaps. Let the coordinates of the face component be (a, b). For a key point, there are two offset heatmaps corresponding to that key point, namely the x-axis offset heatmap and the y-axis offset heatmap. The x-axis coordinate of the face component is added to the value of the a-th pixel of the x-axis offset heatmap to obtain the x-axis coordinate of the key point. The y-axis coordinate of the component is added to the value of the b-th pixel of the y-axis offset heatmap to obtain the y-axis coordinate of the key point. The above steps are repeated until the x-axis coordinates and y-axis coordinates of all key points are calculated. The x-axis coordinates and y-axis coordinates of all key points constitute the face key point coordinate detection result.

[0017] Furthermore, the size of the component heatmap is n×n×C, where C is the number of face components, and the face components are cheek, left eyebrow, right eyebrow, left eye, right eye, nose and mouth. The pixel value at each position in the component heatmap represents the probability that the face component is located at that position, and the coordinates of the pixel with the largest pixel value in the component heatmap are the coordinates of the face component.

[0018] Furthermore, the sizes of the x-axis offset heatmap and the y-axis offset heatmap are 1×64×N and 64×1×N, respectively, where N represents the number of key points.

[0019] Furthermore, the component branch generates a component heatmap based on a 1×1 convolutional layer.

[0020] Furthermore, the offset branch generates an offset heatmap based on the fully connected layer.

[0021] Furthermore, the training process of the facial landmark detection model is as follows:

[0022] The face data used for training is input into the detection network, which includes a Transformer encoder, a CNN decoder, and an LFC regression head. The face data used for training has labeled component heatmap ground truth and offset heatmap ground truth.

[0023] After passing through the Transformer encoder and CNN decoder, the data enters the component branches and offset branches;

[0024] The component branch outputs the current component heatmap, and the offset branch outputs the current offset heatmaps corresponding to the coordinates of multiple face components. The offset loss function is calculated based on the current component heatmap and the ground truth of the offset heatmap. The parameters of the offset branch are optimized based on the offset loss function. The component loss function is calculated based on the current component heatmap and the ground truth of the component heatmap. The parameters of the component branch are optimized based on the component loss function. The face key point detection model is obtained through iterative training.

[0025] Furthermore, the true value of the component heatmap is:

[0026]

[0027] Where, μ x and μ y Here, (x, y) represents the ground truth coordinates of the central key point, (x, y) represents the coordinates of a point on the component heatmap ground truth, σ represents the Gaussian distribution parameter, and heatmap(x, y) represents the pixel value of the component heatmap ground truth at (x, y).

[0028] Furthermore, the central key point is the center point of the component corresponding to the true value of the component heatmap.

[0029] Furthermore, the true value of the offset heatmap is:

[0030] ΔX n，j =x n -j

[0031] ΔY n，j =y n -j

[0032] Where, ΔX n，j and ΔY n，j Let x and y represent the values ​​of the j-th pixel in the ground truth heatmap values ​​of the x-axis offset and y-axis offset heatmap values ​​of the n-th face component, respectively.n and y n These represent the coordinates of the nth face component.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention utilizes heatmap regression based on component heatmaps and offset heatmaps, enabling keypoints belonging to the same face component to share component heatmaps, thereby connecting keypoints that were previously isolated in heatmap regression methods. Furthermore, since offset heatmaps belonging to the same component are generated simultaneously, they can capture the interrelationships between keypoints more effectively, similar to coordinate regression, enhancing the model's facial structure reasoning ability and thus improving its robustness to occlusion. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present invention;

[0036] Figure 2 This is a flowchart of the model detection process of the present invention;

[0037] Figure 3 This is a schematic diagram of the face component of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0039] Definitions:

[0040] Facial landmark localization (FART) is the task of locating key facial regions in a given facial image. These key regions include the mouth, nose, eyes, eyebrows, and facial contours. Localization is typically achieved by detecting key points and obtaining their two-dimensional coordinates. Current FART algorithms can be broadly categorized into two types based on their regression objectives: heatmap regression methods and coordinate regression methods.

[0041] Heatmap regression is a facial landmark detection method. Specifically, for each landmark, the model generates a corresponding heatmap. The pixel value in each heatmap represents the probability that the landmark is located at that pixel, and the coordinates of the pixel with the largest response value in the heatmap are the landmark coordinates. Extensive experiments have demonstrated the accuracy advantage of heatmap regression over coordinate regression in facial landmark detection tasks, thus making it a mainstream method for facial landmark detection.

[0042] Coordinate regression is a method for facial landmark detection. Specifically, it uses a neural network model with a fully connected layer to directly convert image features into two-dimensional coordinates of all key points. Therefore, coordinate regression is simpler and more direct than heatmap regression, requiring less computation. Furthermore, because it obtains the coordinates of all key points simultaneously, it can better learn the overall structure of the face. However, numerous experiments have demonstrated that coordinate regression has a certain accuracy disadvantage compared to heatmap regression in facial landmark detection tasks.

[0043] Transformer: Proposed in 2017, the Transformer is a neural network that completely abandons the convolutional operations of convolutional neural networks, using only attention mechanisms for learning. While initially primarily used for time-series tasks and natural language processing, recent research has shown that the Transformer has achieved performance comparable to, and even surpassing, convolutional neural networks in computer vision, and in some areas, its effectiveness. Methods incorporating Transformers are also frequently seen in cutting-edge research in facial landmark detection.

[0044] This invention proposes an occlusion-robust facial landmark detection method, the flowchart of which is as follows: Figure 1 As shown. The method includes:

[0045] Acquire the image to be detected;

[0046] The image to be detected is converted into a sequence block;

[0047] The sequence block is input into the trained facial landmark detection model, and the trained facial landmark detection model outputs the facial landmark coordinate detection results. The trained facial landmark detection model includes a Transformer encoder, a CNN decoder, and an LFC regression head.

[0048] The specific steps for inputting the sequence block into the trained facial landmark detection model and for the trained facial landmark detection model to output the facial landmark coordinate detection results are as follows:

[0049] The sequence block is input into the Transformer encoder for feature extraction;

[0050] The extracted features are input into the CNN decoder, and the extracted features are upsampled to obtain the upsampled features.

[0051] The upsampled features are input into two branches of the LFC regression head, where the component branch outputs the component heatmap based on the heatmap regression method, and the offset branch outputs the offset heatmap.

[0052] The facial landmark coordinates are detected based on the component heatmap and the offset heatmap.

[0053] The specific results of calculating the facial key point coordinates based on the component heatmap and offset heatmap are as follows:

[0054] The coordinates of face components are obtained based on component heatmaps. Let the coordinates of the face component be (a, b). For a key point, there are two offset heatmaps corresponding to that key point, namely the x-axis offset heatmap and the y-axis offset heatmap. The x-axis coordinate of the face component is added to the value of the a-th pixel of the x-axis offset heatmap to obtain the x-axis coordinate of the key point. The y-axis coordinate of the component is added to the value of the b-th pixel of the y-axis offset heatmap to obtain the y-axis coordinate of the key point. The above steps are repeated until the x-axis coordinates and y-axis coordinates of all key points are calculated. The x-axis coordinates and y-axis coordinates of all key points constitute the face key point coordinate detection result.

[0055] This invention aims to fully utilize the strong structural constraints of the human face itself, enabling the model to better learn the interrelationships between key points, thereby enhancing the model's robustness to occlusion, and ultimately allowing the model to accurately locate all facial key points even under extreme occlusion conditions with an occlusion rate exceeding 50%.

[0056] In this invention, the input image is first converted into a sequence of blocks, that is, the original 224×224 image is cut into several image blocks of 16×16 size, and then these image blocks are processed into a uniform sequence of blocks through linear projection. Subsequently, the image sequence blocks are input into a Transformer encoder to extract features from the image. The extracted features are then input into a Convolutional Neural Network (CNN) decoder, where the feature map is upsampled through two deconvolutional blocks and restored to a size of 64×64. Finally, the processed features are input into a Landmark-in-Facial-Components (LFC) regression head. The LFC regression head has two branches; one branch uses a 1×1 convolutional layer to convert the feature map into a component heatmap, which is used to locate the approximate positions of facial components, such as eyebrows and eyes. Another branch uses fully connected layers to convert feature maps into offset heatmaps. These offset heatmaps, based on the coarse localization results of the component heatmaps, adjust each keypoint on the component to a finer localization result. For example, the component heatmap is used to locate the eyebrows, while the offset heatmap is responsible for adjusting the specific keypoints on the eyebrows from their original positions to detailed locations such as the eyebrow tip and tail. This results in the final facial landmark detection result.

[0057] In this invention, the position and size of the face components are relatively fixed. Therefore, grouping key points according to face components helps the model learn facial structure better and enhances its robustness in challenging environments, especially under severe occlusion. Specifically, after experimentally validating various grouping methods, we finally decided to divide the facial key points into seven parts according to face components: cheek, left eyebrow, right eyebrow, left eye, right eye, nose, and mouth. A schematic diagram of the face components is shown below. Figure 3 As shown.

[0058] In this invention, LFC regression obtains the specific localization results of facial key points through component heatmaps and offset heatmaps. Simply put, we first use component heatmaps to perform coarse localization of facial components. Then, using offset heatmaps, we adjust each key point from its coarsely located position relative to its component to its precise coordinates. The specific data processing flow within the LFC regression head is as follows: Figure 2 As shown.

[0059] The training process of the facial landmark detection model is as follows:

[0060] The face data used for training is input into the detection network, which includes a Transformer encoder, a CNN decoder, and an LFC regression head. The face data used for training has labeled component heatmap ground truth and offset heatmap ground truth.

[0061] After passing through the Transformer encoder and CNN decoder, the data enters the component branches and offset branches;

[0062] The component branch outputs the current component heatmap, and the offset branch outputs the current offset heatmaps corresponding to the coordinates of multiple face components. The offset loss function is calculated based on the current component heatmap and the ground truth of the offset heatmap. The parameters of the offset branch are optimized based on the offset loss function. The component loss function is calculated based on the current component heatmap and the ground truth of the component heatmap. The parameters of the component branch are optimized based on the component loss function. The face key point detection model is obtained through iterative training.

[0063] Specifically, for each input image, a component heatmap of size 64×64×C is generated using the traditional Gaussian heatmap regression method, where C represents the number of defined face components, which is 7 in this case. A ground truth heatmap is generated by selecting the coordinates of the center keypoints on the corresponding face components. For example, the coordinates of the keypoint on the tip of the nose are used to generate a component heatmap representing the nose. The specific calculation process for the value of each pixel position on the component heatmap is shown in the following formula:

[0064]

[0065] Where heatmap(x,y) represents the pixel value at coordinates (x,y) on the heatmap, (μx,μx) represents the ground truth coordinates of the key point, and σ is the Gaussian distribution parameter.

[0066] In this invention, the coordinates of the face component are obtained by performing argmax calculation on the component heatmap.

[0067] In addition, offset heatmaps of size 1×64×N and 64×1×N were generated, where N represents the number of keypoints. These two heatmaps represent the offset values ​​of keypoints along the two coordinate axes, respectively. Specifically, the ground truth values ​​on these two heatmaps are defined by the following formula:

[0068] ΔX n，j =x n -j

[0069] ΔY n，j =y n -j

[0070] Where, ΔX n，j and ΔY n，jLet x and y represent the values ​​of the j-th pixel in the ground truth heatmap values ​​of the x-axis offset and y-axis offset heatmap values ​​of the n-th face component, respectively. n and y n These represent the coordinates of the nth face component.

[0071] In other words, the value of each pixel on the offset heatmap represents the amount by which the corresponding keypoint needs to be offset along the coordinate axis relative to that pixel coordinate. For each keypoint, its coarse positioning coordinates, i.e., the face component coordinates, are first obtained using the component heatmap. Then, the corresponding offset value is found using the two offset heatmaps related to the keypoint, and this offset value is added to the component coordinates to obtain the final positioning result.

[0072] After training the model based on the aforementioned ground truth values, the trained regression head acquires the actual face image and outputs component heatmaps and offset heatmaps. During the training process, the ground truth values ​​for the offset heatmaps and component heatmaps (center key points) are labeled in the training data.

[0073] This invention proposes a novel keypoint grouping strategy, which groups facial keypoints according to facial components: left eyebrow, right eyebrow, left eye, right eye, nose, mouth, and cheeks, into seven parts. This strategy fully utilizes the strong structural constraints of the face itself, enabling the model to more clearly learn the relationships between and within these components, while also enhancing the model's facial structure reasoning ability, thereby improving robustness to occlusion. This invention first uses a traditional heatmap regression method to generate a corresponding component heatmap for each facial component, thus achieving coarse localization of facial components, i.e., coarse localization of facial keypoints. Subsequently, by shifting the heatmap, each keypoint is shifted from its coarse localization coordinates to fine localization coordinates, thereby achieving accurate facial keypoint detection. The above process also simultaneously implements a coarse-to-fine keypoint localization framework, effectively enhancing the model's robustness to occlusion.

[0074] This invention utilizes heatmap regression based on component heatmaps and offset heatmaps, enabling keypoints belonging to the same face component to share component heatmaps, thus connecting keypoints that were previously isolated in heatmap regression methods. Furthermore, since offset heatmaps belonging to the same component are generated simultaneously, they can capture the interrelationships between keypoints more effectively, similar to coordinate regression. Grouping keypoints according to face components introduces the strong structural constraints inherent in the face itself. Moreover, by first coarsely locating face components and then finely locating each individual face keypoint, a coarse-to-fine face keypoint detection framework based on face structure is constructed, further fully utilizing the strong structural constraints of the face itself. This invention achieves an average error of 4.82% in the occluded subset of the WFLW dataset with significant conventional occlusion and an average error of 6.33% in the Masked 300W dataset with significant extreme occlusion (occlusion area greater than 50%), demonstrating superior performance compared to existing methods and proving the excellent performance and occlusion robustness of this patent in face keypoint detection tasks.

[0075] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A robust method for detecting facial landmarks under occlusion, characterized in that the method... include: S1. Obtain the image to be detected; S2. Convert the image to be detected into sequence blocks; S3. Input the sequence block into the trained facial landmark detection model. The trained facial landmark detection model outputs the facial landmark coordinate detection results. The trained facial landmark detection model includes a Transformer encoder, a CNN decoder, and an LFC regression head. The specific steps of S3 are as follows: S31. Input the sequence block into the Transformer encoder for feature extraction; S32. Input the extracted features into the CNN decoder, upsample the extracted features, and obtain the upsampled features. S33. Input the upsampled features into the two branches of the LFC regression head, where the component branch outputs the component heatmap according to the heatmap regression method, and the offset branch outputs the offset heatmap. S34. Calculate the facial key point coordinate detection results based on the component heatmap and offset heatmap; The specific results of calculating the facial key point coordinates based on the component heatmap and offset heatmap are as follows: The coordinates of face components are obtained based on component heatmaps. Let the coordinates of the face component be (a, b). For a key point, there are two offset heatmaps corresponding to the key point, namely the x-axis offset heatmap and the y-axis offset heatmap. The x-axis coordinate of the face component coordinate is added to the value of the a-th pixel of the x-axis offset heatmap to obtain the x-axis coordinate of the key point. The y-axis coordinate of the component coordinate is added to the value of the b-th pixel of the y-axis offset heatmap to obtain the y-axis coordinate of the key point. The above steps are repeated until the x-axis coordinates and y-axis coordinates of all key points are calculated. The x-axis coordinates and y-axis coordinates of all key points constitute the face key point coordinate detection result. The offset branch generates an offset heatmap based on the fully connected layer; The training process of the facial landmark detection model is as follows: The face data used for training is input into the detection network, which includes a Transformer encoder, a CNN decoder, and an LFC regression head. The face data used for training has labeled component heatmap ground truth and offset heatmap ground truth. After passing through the Transformer encoder and CNN decoder, the data enters the component branches and offset branches; The component branch outputs the current component heatmap, and the offset branch outputs the current offset heatmaps corresponding to the coordinates of multiple face components. The offset loss function is calculated based on the current component heatmap and the ground truth of the offset heatmap. The parameters of the offset branch are optimized based on the offset loss function. The component loss function is calculated based on the current component heatmap and the ground truth of the component heatmap. The parameters of the component branch are optimized based on the component loss function. The face key point detection model is obtained through iterative training.

2. The method of claim 1, wherein, The component heatmap has a size of n×n×C, where C is the number of face components. The face components are cheek, left eyebrow, right eyebrow, left eye, right eye, nose, and mouth. The pixel value at each position in the component heatmap represents the probability that the face component is located at that position. The coordinates of the pixel with the largest pixel value in the component heatmap are the coordinates of the face component.

3. The occlusion-robust facial landmark detection method according to claim 2, characterized in that, The sizes of the x-axis offset heatmap and the y-axis offset heatmap are 1×64×N and 64×1×N, respectively, where N represents the number of key points.

4. The occlusion-robust facial landmark detection method according to claim 1, characterized in that, The component branch generates a component heatmap based on a 1×1 convolutional layer.

5. A robust facial landmark detection method for occlusion as described in claim 1, characterized in that, The true value of the component heatmap is: in, μ x and μ y The true coordinates of the central key point, ( x , y () represents the coordinates of a point on the component's heatmap truth. σ Represents the parameters of the Gaussian distribution. heatmap ( x , y )express( x , y The pixel value of the component heatmap truth at ().

6. A robust facial landmark detection method for occlusion as described in claim 5, characterized in that, The central key point is the center point of the component corresponding to the true value of the component heatmap.

7. The occlusion-robust facial landmark detection method according to claim 1, characterized in that, The true value of the offset heatmap is: Where, Δ X n，j and Δ Y n，j Represent the ground truth values ​​of the x-axis and y-axis offset heatmaps of the nth face component coordinates, respectively. j The value of each pixel. x n and y n These represent the coordinates of the nth face component.

Images