Face key point detection method and system based on auxiliary heat map

Abstract

The present application relates to a face key point detection method and system based on auxiliary heat map, wherein the face key point detection method based on auxiliary heat map comprises the following steps: obtaining an original image, processing the original image through a network model, and obtaining features based on the original image in different layers of the network model; obtaining a main heat map and an auxiliary heat map after the features are fused; obtaining an integer part of a key point coordinate based on the main heat map, obtaining a decimal part of the key point coordinate based on the auxiliary heat map, adding the two part values after normalization to obtain the final value of the key point coordinate and mapping the final value to the original image to obtain the final key point coordinate. ShuffleNetV2 is selected as the backbone network and is scaled at the same time, the features obtained by the original image through each layer of the backbone network are fused to obtain the main heat map and the auxiliary heat map, and the main heat map and the auxiliary heat map jointly represent the face key point to solve the quantization error problem.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a method and system for detecting facial key points based on auxiliary heatmaps. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Facial landmark detection can provide foundational data for other facial vision tasks, such as facial pose correction, pose recognition, expression recognition, fatigue monitoring, and lip-reading recognition. Therefore, high-precision facial landmark detection is needed to meet specific operational tasks in fields such as computer vision, pattern recognition, and image processing.

[0004] Existing heatmap-based face landmark detection models use networks with large parameter counts and high FLOPs (floating-point operations per second) as the backbone, such as Hourglass and HRNet. Furthermore, existing lightweight models do not consider the feature information fusion problem between different layers, while heatmap models are very sensitive to features. At the same time, existing heatmap methods introduce quantization errors when mapping heatmaps back to landmark coordinates. Although some post-processing methods can alleviate quantization errors, it is difficult to solve this problem. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a method and system for detecting facial key points based on auxiliary heatmaps. ShuffleNetV2 is selected as the backbone network and scaled. Features obtained from each layer of the backbone network are fused to obtain a main heatmap and an auxiliary heatmap. The main heatmap and the auxiliary heatmap are jointly used to represent facial key points to solve the quantization error problem.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a method for facial landmark detection based on auxiliary heatmaps, comprising the following steps:

[0008] Extract feature information from the original image based on the trained network model;

[0009] Features based on the original image are obtained in different layers of the network model, and after fusion, the main heatmap and auxiliary heatmap are obtained.

[0010] The integer part of the key point coordinates is obtained based on the main heatmap, and the fractional part of the key point coordinates is obtained based on the auxiliary heatmap. After normalizing the two parts, the values ​​are amplified to obtain the final values ​​of the key point coordinates, which are then mapped to the original image to obtain the final key point coordinates.

[0011] The magnification process involves normalizing the two numerical values, multiplying them by their respective magnification factors, and then adding them together to obtain the final values ​​of the keypoint coordinates.

[0012] The network model includes a first convolutional layer, a max pooling layer, and a set number of recombination channel units connected in sequence. The max pooling layer and the recombination channel units are connected to the fusion layer to output features for splicing the main heatmap and the auxiliary heatmap. The features output by the fusion layer are spliced ​​and then input into the second convolutional layer for processing to obtain the main heatmap and the auxiliary heatmap.

[0013] The fusion layer has a set number of BottleNeck layers, which are respectively connected to the max pooling layer and the recombined channel unit; the BottleNeck layer includes depth-separable convolutional blocks connected between two sets of first convolutional layers.

[0014] The first convolutional layer consists of a two-dimensional convolutional block, a batch normalization layer, and a ReLU activation function connected in sequence.

[0015] The second convolutional layer consists of a two-dimensional convolutional block, a batch normalization layer, a ReLU activation function, and a two-dimensional convolutional block connected in sequence.

[0016] The channel recombination unit comprises a first convolutional layer, a depthwise separable convolutional block, a first convolutional layer, and a channel recombination layer connected in sequence; wherein the depthwise separable convolutional block comprises a two-dimensional depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0017] Features based on the original image are obtained in different layers of the network model, and after fusion, a main heatmap and an auxiliary heatmap are obtained, specifically:

[0018] 1) The features of the original image after max pooling and recombining channel units are processed by the fusion layer, and the processed features are added together;

[0019] 2) Depending on the model size, step 1) is executed a set number of times;

[0020] 3) Concatenate the features processed in step 2) to obtain the main heatmap and the auxiliary heatmap;

[0021] The main heatmap obtains the normalized values ​​corresponding to the integer part of the keypoint coordinates using the argmax function; the auxiliary heatmap obtains the normalized values ​​corresponding to the decimal part of the keypoint coordinates using the argmax function; the two values ​​are added together to obtain the final values ​​of the keypoint coordinates and mapped to the original image to obtain the final keypoint coordinates.

[0022] A second aspect of the present invention provides a system for implementing the above-described method, comprising:

[0023] The feature extraction module is configured to extract feature information from the original image based on the trained network model.

[0024] The heatmap module is configured to: obtain features based on the original image in different layers of the network model, and then fuse them to obtain the main heatmap and auxiliary heatmap;

[0025] The key point detection module is configured to: obtain the integer part of the key point coordinates based on the main heatmap, obtain the fractional part of the key point coordinates based on the auxiliary heatmap, normalize the two parts and add them together to obtain the final value of the key point coordinates, and map it to the original image to obtain the final key point coordinates.

[0026] A third aspect of the present invention provides a computer-readable storage medium.

[0027] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the facial landmark detection method based on auxiliary heatmaps as described above.

[0028] A fourth aspect of the present invention provides a computer device.

[0029] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the facial key point detection method based on auxiliary heatmaps as described above.

[0030] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects:

[0031] 1. To address the quantization error issue in heatmap models, in addition to generating a primary heatmap, an auxiliary heatmap is also generated. This auxiliary heatmap is used to assist in coordinate regression of the primary heatmap, thus resolving the quantization error between heatmap coordinates and their conversion to true coordinates. Compared to traditional face landmark detection methods based on heatmap regression, this method significantly reduces the number of network model parameters compared to complex networks, while maintaining detection accuracy comparable to large networks. This effectively avoids the problem of excessively large network model parameters in existing methods, which is beneficial for deployment on mobile devices.

[0032] 2. For the current heatmap model, the Hourglass network is used as the backbone. By combining various lightweight networks, ShuffleNetV2 is selected as the backbone network and scaled up. This network design complies with the four lightweight network design principles, and the accuracy is not significantly reduced.

[0033] 3. To address the issue of feature fusion between different layers in existing lightweight models, a multi-scale feature information interaction block was designed. While using high-resolution feature branches, multiple low-resolution branches are executed in parallel. Combined with continuous feature fusion and information interaction between different branches, the resulting heatmap possesses strong semantic information and accurate location information. This compensates for the information loss caused by the reduced number of feature map channels in traditional neural networks, making the network more sensitive to the location of key points. Attached Figure Description

[0034] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0035] Figure 1 This is a schematic diagram of the network structure in the face key point detection process provided by one or more embodiments of the present invention;

[0036] Figure 2 This is a schematic diagram of the position with the highest value in the main heat map obtained from the original image during the face key point detection process provided by one or more embodiments of the present invention;

[0037] Figure 3 This is a schematic diagram of the position with the highest value in the auxiliary heat map obtained from the original image during the face key point detection process provided by one or more embodiments of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] As described in the background section, although heatmap-based regression models have high accuracy, they still suffer from the following problems:

[0042] (1) Most of the backbone networks of heatmap models use models with high parameter count and FLOPs, such as Hourglass, which results in slow running speed and high memory consumption.

[0043] (2) Existing lightweight models do not take into account the problem of feature information fusion between different layers, while in the heatmap model, the model is very sensitive to features.

[0044] (3) Existing heatmap methods introduce quantization error when mapping heatmaps back to key point coordinates. Although some post-processing methods can alleviate quantization error, it is difficult to solve this problem.

[0045] Therefore, the following embodiments present a method and system for facial landmark detection based on auxiliary heatmaps, which improves the heatmap regression-based model:

[0046] (1) The current heatmap model uses the Hourglass network as the backbone. By combining various lightweight networks, ShuffleNetV2 was selected as the backbone network and scaled up. This network design conforms to the four lightweight network design principles and the accuracy is not significantly reduced.

[0047] (2) To address the issue that existing lightweight models fail to consider feature fusion between different layers, a multi-scale feature information interaction block was designed. While using high-resolution feature branches, multiple low-resolution branches are executed in parallel. Furthermore, continuous feature fusion and information interaction between different branches result in a heatmap with strong semantic information and accurate location information. This compensates for the information loss caused by the reduction in the number of feature map channels in traditional neural networks, making the network more sensitive to the location of key points.

[0048] (3) To address the quantization error problem in the heatmap model, in addition to generating a main heatmap as previously described, an auxiliary heatmap is also generated. This auxiliary heatmap is used to assist the main heatmap in coordinate regression, thus resolving the quantization error problem between heatmap coordinates and real coordinates. Compared with traditional face keypoint detection methods based on heatmap regression, this method significantly reduces the number of network model parameters compared to complex networks, while still maintaining detection accuracy comparable to large networks. This effectively avoids the problem of excessively large network model parameters in existing methods, which is detrimental to deployment on mobile devices.

[0049] Example 1:

[0050] like Figure 1-3 As shown, the face landmark detection method based on auxiliary heatmaps includes the following steps:

[0051] Extract feature information from the original image based on the trained network model;

[0052] Features based on the original image are obtained in different layers of the network model, and after fusion, the main heatmap and auxiliary heatmap are obtained.

[0053] The integer part of the key point coordinates is obtained based on the main heatmap, and the fractional part of the key point coordinates is obtained based on the auxiliary heatmap. After normalization of the two parts, the values ​​are amplified to obtain the final values ​​of the key point coordinates, which are then mapped to the original image to obtain the final key point coordinates.

[0054] This embodiment provides the following:

[0055] 1) Lightweight model, using a scaled ShuffleNetV2 as the backbone network, resulting in fewer model parameters and fewer FLOPs (floating-point operations).

[0056] 2) Multi-scale fusion method, which integrates features and information exchange in the backbone network.

[0057] 3) The auxiliary heatmap method combines the main heatmap with the auxiliary heatmap to represent facial key points and solve the quantization error problem.

[0058] Specifically:

[0059] Model structure as follows Figure 1As shown, the system includes a first convolutional layer (Conv Block1, 256×256×3), a max pooling layer (MaxPool, 128×128×18), and a set number of shuffle units connected in sequence. In this embodiment, there are three sets of shuffle units. The max pooling layer (MaxPool, 128×128×18) and the three sets of shuffle units are connected to the fusion layer to output features for splicing the main heatmap and the auxiliary heatmap. The features output by the fusion layer are spliced ​​and then input into the second convolutional layer (Conv Block2) for processing to obtain the main heatmap and the auxiliary heatmap.

[0060] Among them, there are three sets of recombination channel units (Shuffle Units), namely 64×64×18 (4 sets), 32×32×36 (8 sets) and 16×16×72 (4 sets).

[0061] The fusion layer has a set number of BottleNeck layers, which are four in this embodiment, and are respectively connected to the MaxPool layer and three Shuffle Units.

[0062] The BottleNeck layer includes depth-separable convolutional blocks (DWConv Blocks) connecting two sets of first convolutional layers (Conv Block1).

[0063] The first convolutional layer (Conv Block1) includes a two-dimensional convolutional block (conv2d), a batch normalization layer (BN, or BatchNorm), and a ReLU activation function connected in sequence.

[0064] The structure of the second convolutional layer (Conv Block2) is similar to that of the first convolutional layer, except that a set of two-dimensional convolutional blocks (conv2d) is added after the ReLU activation function; that is, it includes a two-dimensional convolutional block, a batch normalization layer, a ReLU activation function and a two-dimensional convolutional block connected in sequence.

[0065] The Shuffle Unit comprises a first convolutional layer (Conv Block1), a depth-separable convolutional block (DWConv Block), another first convolutional layer (Conv Block1), and a channel shuffle, which are connected in sequence.

[0066] The depthwise separable convolutional block (DWConv Block) consists of a two-dimensional depthwise separable convolutional layer (DWConv2d), a batch normalization layer (BN, i.e., BatchNorm), and a ReLU activation function connected in sequence.

[0067] This model is implemented through the following steps:

[0068] Step 1: Construct the model backbone network. ShuffleNetV2 is selected as the backbone network, and to further reduce the number of parameters and FLOPs, it is scaled down to half its original size.

[0069] Step 2: Based on the feature fusion method, the features of different layers of the backbone network are fused.

[0070] The specific process of fusion processing is as follows:

[0071] 1) Figure 1 In the main backbone network on the left, the features from the MaxPool layer and the subsequent three ShuffleUnit layers are processed by four BottleNeck layers, and then the processed features are summed. The BottleNeck layer is a processing method in the ResNet network.

[0072] 2) Depending on the model size, the above steps may be executed multiple times, with a maximum of 4 executions in this embodiment.

[0073] 3) Concatenate the features obtained in step 2) to generate a primary heatmap and an auxiliary heatmap.

[0074] In this embodiment, a primary heatmap is generated first, followed by an auxiliary heatmap. The stitching process involves both upsampling and downsampling. Taking the primary heatmap as an example... Figure 1 The structure has four Bottleneck layers from top to bottom. The feature map size of the first layer is 64*64. The features from the second to fourth groups that have passed through the Bottleneck layer (i.e., 32*32, 16*16, 8*8) are all upsampled to 64*64 and then concatenated directly along the channel direction, resulting in a sum of 18 + 36 + 72 + 144 = 270. Similarly, the auxiliary heatmap is downsampled to 8*8 before being concatenated.

[0075] Step 2 significantly reduced the number of model parameters and FLOPs while maintaining the model's feature learning ability, thus solving the problem of insufficient feature learning ability in lightweight models.

[0076] Step 3: Based on the auxiliary heatmap method, the network generates a main heatmap and an auxiliary heatmap. In this embodiment, the generated main heatmap is 64×64 in size, and the normalized values ​​corresponding to the integer part of the coordinates are obtained through the argmax function. The auxiliary heatmap is 8×8 in size, and the normalized values ​​corresponding to the decimal part of the coordinates are obtained through the argmax function. Finally, these two values ​​are multiplied by their respective factors and added together to obtain the final coordinate values. The specific effect is as follows: Figure 2 and Figure 3 As shown in the figure, (x) p ,y p (x′) represents the coordinates of the location with the highest value on the main heatmap. p ,y′ p (x) represents the integer part of the coordinates after mapping back to the original image; similarly, (x) represents the integer part of the coordinates. a ,y a (x′) is the coordinate of the location with the highest value on the auxiliary heat map. a ,y′ a (x′) represents the decimal part of the coordinates after mapping back to the original image. p ,y′ p ) and (x′ a ,y′ a The final precise coordinates are obtained by adding them together.

[0077] The model is trained using existing keypoint images, and detection is performed after training (the training process is based on existing technology). Compared with traditional deep learning networks, the network model in this embodiment can achieve accuracy comparable to heatmap-based facial keypoint detection methods, while having fewer model parameters. Furthermore, to address the quantization error problem in facial keypoints, the network simulation in this embodiment proposes an auxiliary heatmap, which, together with the main heatmap, represents the coordinates of facial keypoints.

[0078] test:

[0079] Table 1 Comparison of error rates for each model on the WFLW test set.

[0080]

[0081] The WFLW test set is divided into six subsets: Pose, Expression, Illumination, Make-up, Occlusion, and Blur. Full is the combination of these six subsets. NME represents the standard mean squared error; a smaller NME indicates higher model accuracy. LFLD is the model proposed in this embodiment, and the numbers following it represent the number of repetitions in the feature fusion stage. Table 1 shows that the detection accuracy of the model proposed in this embodiment is comparable to large-scale networks, while its parameter count and FLOPs are lower, achieving a trade-off between accuracy and model parameter count.

[0082] Example 2:

[0083] A system for implementing the above method includes:

[0084] The feature extraction module is configured to extract feature information from the original image based on the trained network model.

[0085] The heatmap module is configured to: obtain features based on the original image in different layers of the network model, and then fuse them to obtain the main heatmap and auxiliary heatmap;

[0086] The key point detection module is configured to: obtain the integer part of the key point coordinates based on the main heatmap, and obtain the fractional part of the key point coordinates based on the auxiliary heatmap. After normalizing the two parts, the values ​​are amplified to obtain the final values ​​of the key point coordinates and mapped to the original image to obtain the final key point coordinates.

[0087] The system selected ShuffleNetV2 as the backbone network and scaled it up. The network design conforms to the four lightweight network design principles, and the accuracy is not significantly reduced.

[0088] A multi-scale feature information interaction block was designed, which uses a high-resolution feature branch while executing multiple low-resolution branches in parallel. Combined with continuous feature fusion and information interaction between different branches, the resulting heatmap possesses strong semantic information and accurate location information. This compensates for the information loss caused by the reduced number of feature map channels in traditional neural networks, making the network more sensitive to the location of key points.

[0089] In addition to generating a primary heatmap, an auxiliary heatmap is also generated. The auxiliary heatmap is used to help the primary heatmap perform coordinate regression, thereby solving the quantization error problem between restoring the heatmap coordinates to the true coordinates.

[0090] Example 3:

[0091] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the face key point detection method based on auxiliary heatmaps as described in Embodiment 1 above.

[0092] Example 4:

[0093] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the face key point detection method based on auxiliary heatmap as described in Embodiment 1 above.

[0094] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A facial landmark detection method based on auxiliary heatmaps, characterized in that, Includes the following steps: Extract feature information from the original image based on the trained network model; Features based on the original image are obtained in different layers of the network model, and after fusion, the main heatmap and auxiliary heatmap are obtained. The integer part of the key point coordinates is obtained based on the main heatmap, and the fractional part of the key point coordinates is obtained based on the auxiliary heatmap. After normalization of the two parts, the values ​​are amplified to obtain the final values ​​of the key point coordinates, which are then mapped to the original image to obtain the final key point coordinates. Features based on the original image are obtained in different layers of the network model, and after fusion, a main heatmap and an auxiliary heatmap are obtained, specifically: Step 1): The features of the original image after max pooling and recombining channel units are processed by the fusion layer, and the processed features are added together. Step 2), depending on the model size, execute step 1) a set number of times; Step 3) Concatenate the features processed in Step 2) to obtain the main heatmap and the auxiliary heatmap; The main heatmap uses the argmax function to obtain the normalized values ​​corresponding to the integer part of the key point coordinates; the auxiliary heatmap uses the argmax function to obtain the normalized values ​​corresponding to the decimal part of the key point coordinates; the two parts are added together to obtain the final values ​​of the key point coordinates and mapped to the original image to obtain the final key point coordinates. The network model includes a first convolutional layer, a max pooling layer, and a set number of recombination channel units connected in sequence. The max pooling layer and the recombination channel units are respectively connected to the fusion layer to output features for splicing the main heatmap and the auxiliary heatmap. The features output by the fusion layer are spliced ​​and then input into the second convolutional layer for processing to obtain the main heatmap and the auxiliary heatmap.

2. The facial landmark detection method based on auxiliary heatmap as described in claim 1, characterized in that, The magnification process specifically involves normalizing the two numerical values, multiplying them by their respective magnification factors, and then adding them together to obtain the final values ​​of the key point coordinates.

3. The facial landmark detection method based on auxiliary heatmap as described in claim 1, characterized in that, The fusion layer has a set number of BottleNeck layers, which are respectively connected to the max pooling layer and the recombining channel unit; the BottleNeck layer includes depth-separable convolutional blocks connected between two sets of first convolutional layers.

4. The facial landmark detection method based on auxiliary heatmap as described in claim 1, characterized in that, The first convolutional layer consists of a two-dimensional convolutional block, a batch normalization layer, and a ReLU activation function connected in sequence; the second convolutional layer consists of a two-dimensional convolutional block, a batch normalization layer, a ReLU activation function, and a two-dimensional convolutional block connected in sequence.

5. The facial landmark detection method based on auxiliary heatmap as described in claim 1, characterized in that, The channel recombination unit comprises a first convolutional layer, a depthwise separable convolutional block, a first convolutional layer, and a channel recombination layer connected in sequence; wherein the depthwise separable convolutional block comprises a two-dimensional depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

6. A facial landmark detection system based on auxiliary heatmaps, used to implement the facial landmark detection method based on auxiliary heatmaps as described in any one of claims 1-5, characterized in that, include: The feature extraction module is configured to extract feature information from the original image based on the trained network model. The heatmap module is configured to: obtain features based on the original image in different layers of the network model, and then fuse them to obtain the main heatmap and auxiliary heatmap; The key point detection module is configured to: obtain the integer part of the key point coordinates based on the main heatmap, and obtain the fractional part of the key point coordinates based on the auxiliary heatmap. After normalizing the two parts, the values ​​are amplified to obtain the final values ​​of the key point coordinates and mapped to the original image to obtain the final key point coordinates.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the face key point detection method based on auxiliary heatmap as described in any one of claims 1-5.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the face key point detection method based on auxiliary heatmap as described in any one of claims 1-5.

Images