Face alignment method based on boundary-aware face alignment neural network

Abstract

This application discloses a face alignment method based on a boundary-aware face alignment neural network, including a boundary heatmap estimation subnetwork and a coordinate regression subnetwork. The boundary heatmap estimation subnetwork includes a CoordConv layer and multiple shallow and deep feature fusion (SDFusion) modules. The coordinate regression subnetwork includes a self-attention-based feature re-extraction (SAfeature) module and a Transformer-decoder module. The boundary heatmap estimation subnetwork takes the user's original face image information as input and generates a boundary heatmap as output. The coordinate regression subnetwork fuses the original face image information, the latent features of the boundary heatmap estimation subnetwork, and the generated boundary heatmap. The SAfeature module and the Transformer-decoder module are used to map the fused features to key point coordinates to further improve the accuracy of the predicted boundary heatmap.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a face alignment method based on a boundary-aware face alignment neural network. Background Technology

[0002] Face alignment, also known as FLD (facial landmarks detection), refers to the automatic location of a predefined set of semantic feature points in a face image. Accurate face alignment plays a crucial role in various face applications, including face recognition and verification, face reconstruction, and face deformation.

[0003] Most deep learning-based FLD algorithms locate keypoint coordinates using either coordinate regression or keypoint heatmap regression. The former directly predicts the position of each keypoint, while the latter estimates the heatmap of each keypoint and positions it at the highest response point in the heatmap. However, using these algorithms, the semantic and geometric correlations between keypoints are not fully utilized to guide network learning, thus failing to accurately capture facial keypoints. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:

[0005] In a first aspect, embodiments of this application provide a face alignment method based on a boundary-aware face alignment neural network, comprising: a boundary heatmap estimation subnetwork and a coordinate regression subnetwork. The boundary heatmap estimation subnetwork includes a CoordConv layer and multiple shallow and deep feature fusion (SDFusion) modules. The coordinate regression subnetwork includes a self-attention-based feature re-extraction (SAfeature) module and a Transformer-decoder module. The boundary heatmap estimation subnetwork receives user-inputted original face image information at its input end and generates a boundary heatmap at its output end. The coordinate regression subnetwork fuses the original face image information, the latent features of the boundary heatmap estimation subnetwork, and the generated boundary heatmap. The SAfeature module and the Transformer-decoder module are used to map the fused features to keypoint coordinates.

[0006] In one possible implementation, the SDFusion module includes a first residual module. The input of the first residual module is a first image information. The output of the first residual module is connected to the inputs of a first convolution module and a pyramid pooling module. The output of the first convolution module is activated and outputs a first boundary sub-heatmap of the first image information. The first boundary sub-heatmap is input to a second convolution module and activated and output. The first image information processed by the first residual module is then concatenated with a second image information channel to obtain third image information. The pyramid pooling module is then sequentially connected to a third convolution module, a second residual module, and a fourth convolution module. The third image information is processed by the third convolution module, the second residual module, and the fourth convolution module, and then multiplied element-wise with the first boundary sub-heatmap output by the second convolution module to obtain contour-enhanced fourth image information.

[0007] In one possible implementation, the SAfeature module includes a fifth, sixth, seventh, and eighth convolutional module arranged in parallel. The output information of the fifth and sixth convolutional modules is subjected to a matrix outer product operation to obtain two sets of feature maps. The two sets of feature maps are concatenated by channels to obtain new feature map information. The output information of the seventh convolutional module is added element-wise with the new feature map information and then input into the ninth convolutional module. The output information of the eighth convolutional module is added element-wise with the output information of the ninth convolutional module and then input into the third residual module. The output of the third residual module is connected to the input of the Transformer-decoder module.

[0008] In one possible implementation, the boundary heatmap estimation subnetwork further includes a tenth convolutional module, a fourth residual module, and a first fuzzy pooling module connected in sequence. The output of the first fuzzy pooling module is connected to the input of the CoordConv layer. A fifth residual module, a sixth residual module, and a first hourglass module are sequentially arranged between the CoordConv layer and the first SDFusion module. A first hourglass module is arranged between adjacent SDFusion modules. The output of the last SDFusion module is sequentially connected to a second hourglass module and an eleventh convolutional module. The output of the eleventh convolutional module is activated to output the boundary heatmap.

[0009] In one possible implementation, the input of the first residual module in the first SDFusion module is the element-wise sum of the output of the sixth residual module and the output of the first hourglass module, and the input of the first residual module in the remaining SDFusion modules is the element-wise sum of the output of the previous SDFusion module and the output of the first hourglass module, and the second image information comes from the CoordConv layer.

[0010] In one possible implementation, the coordinate regression subnetwork further includes a twelfth convolutional module, a second fuzzy pooling module, a thirteenth convolutional module, a fourteenth convolutional module, a third fuzzy pooling module, a fifteenth convolutional module, a fourth fuzzy pooling module, a sixteenth convolutional module, and a fifth fuzzy pooling module connected in sequence, with the output of the fifth fuzzy pooling module connected to the input of the SAfeature module.

[0011] In one possible implementation, the boundary heatmap obtained by the boundary heatmap estimation subnetwork is concatenated with the original face image information channel after the nearest neighbor difference operation and the average pooling operation, and then input into the twelfth convolution module. The output information of the second fuzzy pooling module is concatenated with the output information channel of the last SDFusion module and then input into the thirteenth convolution module.

[0012] In one possible implementation, the loss function Loss is determined, and the overall Loss includes... and These correspond to the keypoint coordinate loss and the boundary heatmap loss, respectively. The total loss is defined as follows:

[0013]

[0014] in, Indicates the number of facial landmarks. and These represent the predicted keypoint coordinates and the labeled keypoint coordinates, respectively. This means that the number of times the boundary heatmap is predicted is equal to the number of stacked hourglass modules. and These represent the predicted boundary heatmap and the actual boundary heatmap, respectively. β represents the weights; β is a hyperparameter that adjusts the loss between these two types, and is set to 0.001 by default.

[0015] In one possible implementation, the face matching result needs to be restored to a coordinate system with respect to the input image size, i.e., the horizontal coordinate is multiplied by the width of the input image, and the vertical coordinate is multiplied by the width of the input image. If it is necessary to draw the predicted key points onto the original image, the parameters of the face bounding box in the data preprocessing are used to multiply the model output result by the width and height of the bounding box, and then the key points are drawn onto the original image using the position of the bounding box.

[0016] In this embodiment, the boundary heatmap estimation subnetwork generates the boundary heatmap, and the coordinate regression subnetwork first fuses the original image information and the boundary heatmap, and then uses the Transformer-decoder module to map the fused features to key point coordinates to further improve the accuracy of the predicted boundary heatmap. Attached Figure Description

[0017] Figure 1 A schematic diagram of a boundary-aware face alignment neural network provided in this application embodiment;

[0018] Figure 2 A diagram illustrating the meaning of modules in a boundary-aware face alignment neural network provided in this application embodiment;

[0019] Figure 3 This is a schematic diagram of the SDFusion module structure provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the SAfeature module provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the test image results provided in an embodiment of this application. Detailed Implementation

[0022] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0023] Figure 1 A schematic diagram of a boundary-aware face alignment neural network provided in this application embodiment is shown below. Figure 1 The boundary-aware face alignment neural network in this embodiment includes a boundary heatmap estimation subnetwork and a coordinate regression subnetwork. The boundary heatmap estimation subnetwork includes a CoordConv layer and multiple shallow and deep feature fusion SDFusion modules. The coordinate regression subnetwork includes a self-attention-based feature re-extraction SAfeature module and a Transformer-decoder module.

[0024] The boundary heatmap estimation subnetwork takes the user's original face image information as input and generates a boundary heatmap as output. The coordinate regression subnetwork fuses the original face image information, the latent features of the boundary heatmap estimation subnetwork, and the generated boundary heatmap. The SAfeature module and the Transformer-decoder module are used to map the fused features to key point coordinates.

[0025] The network proposed in this application includes two stages: boundary heatmap estimation and key point coordinate prediction. The complete network structure diagram is shown below. Figure 1 As shown in the figure, the first row represents the boundary heatmap estimation subnetwork, and the second row represents the coordinate regression subnetwork.

[0026] Specifically, let I be an input image, represented as a three-dimensional tensor of size C×H×W, where C, H, and W are the number of channels, height, and width of the image, respectively. The method of this embodiment can then be defined as a function Φ: I→(B, P), meaning that based on the input image I, this embodiment can predict a set of boundary heatmaps B = {b1, b2, ..., b...} M} and a key point coordinate matrix P=X N×2 Where bi, M, and N represent the predicted heatmap, the number of hourglass modules stacked in the first stage, and the number of keypoints detected in the second stage, respectively. Furthermore, the first stage will combine the input image I and the predicted boundary heatmap b... M The input of the last hourglass module is transmitted to the second-stage network.

[0027] Whether training or using a model, images need to be preprocessed before being fed into the model. The purpose is to remove some redundant information from the original image, improve the image quality, and enhance the detectability of useful information. This simplifies subsequent operations such as model segmentation, feature extraction, and recognition, and improves its reliability.

[0028] In the field of facial landmark detection, image preprocessing during model training is complex. First, each image needs to be cropped, based on a bounding box enclosing the face, which must contain landmark coordinates. If the dataset doesn't provide these bounding boxes, there are typically two methods to obtain them: using an existing face detection model or calculating them using a script based on the landmark coordinates provided in the dataset (the latter is more commonly used). After obtaining the bounding box, its width and height need to be scaled within a certain range, and it also needs to be translated within that range to reduce errors introduced by manual annotation and the face detection model. Then, the image within the bounding box can be cropped. To enrich the training set, data augmentation is also necessary, typically including operations such as translation, rotation, flipping, blurring, adding occlusion, adjusting lighting intensity, and adding noise to the cropped image, making the trained model more robust. While performing various image operations, corresponding operations also need to be performed on the landmark annotations to ensure accuracy. For example, when an image is rotated by 10 degrees, it is necessary to calculate the new coordinates of the key points in the image after the rotation.

[0029] In the testing phase, only the original face bounding box needs to be obtained and the image within the bounding box cropped. Once the final image and keypoint labels are obtained, the keypoint annotations are normalized using the image size. Then, the image is resized to a fixed size to match the model's input size and normalized again. Finally, the image and keypoint annotations are converted into tensors and fed into the model.

[0030] See Figure 3 The SDFusion module includes a first residual module. The input of the first residual module is the first image information. The output of the first residual module is connected to the input of the first convolution module and the pyramid pooling module. The output of the first convolution module is activated and outputs a first boundary sub-heatmap of the first image information. The first boundary sub-heatmap is input to the second convolution module and activated and output.

[0031] The first image information, processed by the first residual module, is then concatenated with the second image information channel after passing through the pyramid pooling module to obtain the third image information. The pyramid pooling module is then sequentially connected to the third convolution module, the second residual module, and the fourth convolution module. After passing through the third convolution module, the second residual module, and the fourth convolution module, the third image information is multiplied element-wise with the first boundary sub-heatmap output by the second convolution module to obtain the fourth image information with enhanced contours. All convolution modules in the SDFusion module are 1×1 convolutions.

[0032] See Figure 4 The SAfeature module includes a fifth, sixth, seventh, and eighth convolutional module arranged in parallel. The outputs of the fifth and sixth convolutional modules undergo a matrix outer product operation to obtain two sets of feature maps. These two sets of feature maps are then concatenated through channels to obtain new feature map information. The output of the seventh convolutional module is element-wise added to the new feature map information and then input into the ninth convolutional module. The output of the eighth convolutional module is element-wise added to the output of the ninth convolutional module and then input into the third residual module. The output of the third residual module is connected to the input of the Transformer-decoder module. The fifth, sixth, seventh, and eighth convolutional modules in the SAfeature module are all 1×1 convolutions.

[0033] The boundary heatmap estimation subnetwork further includes a tenth convolutional module, a fourth residual module, and a first fuzzy pooling module connected in sequence. The output of the first fuzzy pooling module is connected to the input of the CoordConv layer. Between the CoordConv layer and the first SDFusion module, a fifth residual module, a sixth residual module, and a first hourglass module are sequentially arranged. Adjacent SDFusion modules are connected by a first hourglass module. The output of the last SDFusion module is sequentially connected to a second hourglass module and an eleventh convolutional module. The output of the eleventh convolutional module is activated to output the boundary heatmap. The tenth convolutional module of the boundary heatmap estimation subnetwork is a 7×7 convolution, and the remaining convolutional modules are 1×1 convolutions.

[0034] In this embodiment, the input of the first residual module in the first SDFusion module comes from the element-wise addition of the output of the sixth residual module and the output of the first hourglass module. The input of the first residual module in the remaining SDFusion modules comes from the element-wise addition of the output of the previous SDFusion module and the output of the first hourglass module. The second image information comes from the CoordConv layer.

[0035] The coordinate regression subnetwork further includes a twelfth convolutional module, a second fuzzy pooling module, a thirteenth convolutional module, a fourteenth convolutional module, a third fuzzy pooling module, a fifteenth convolutional module, a fourth fuzzy pooling module, a sixteenth convolutional module, and a fifth fuzzy pooling module connected in sequence. The output of the fifth fuzzy pooling module is connected to the input of the SAfeature module. The thirteenth convolutional module in the coordinate regression subnetwork is a 1×1 convolution, and the remaining convolutional modules are 3×3 convolutions.

[0036] The boundary heatmap obtained by the boundary heatmap estimation subnetwork is concatenated with the original face image information channel after the nearest neighbor difference operation and the average pooling operation, and then input into the twelfth convolution module. The output information of the second fuzzy pooling module is concatenated with the output information channel of the last SDFusion module and then input into the thirteenth convolution module.

[0037] Inspired by attention mechanisms and coarse-to-fine frameworks, the enhanced HourglassNet aims to force the network to pay more attention to boundary regions for more accurate prediction of boundary heatmaps and to use the predicted heatmaps as explicit guidance to enhance feature maps. To achieve this goal, this embodiment stacks a baseline network of four hourglass modules. Then, an SDFusion module is designed. First, this module generates an attention map using the predicted boundary heatmap. Then, a pyramid pooling module captures multi-scale contextual information to obtain feature map f1. f1 and shallow feature f2 are concatenated by channels, and then the number of channels is adjusted by a 1×1 convolution. The feature is then fused by a residual module and a 1×1 convolution. Finally, the attention map is multiplied with this feature. In addition, to maintain shift invariance in the network and retain as much detail as possible from pooling layers or convolutions, this embodiment uses fuzzy pooling instead. Considering the effectiveness of positional encoding, this embodiment also introduces a CroodConv layer in the shallow layers of the network to explicitly add positional information to the network.

[0038] Specifically, after receiving the input, the network in this stage passes through convolution, residual modules, fuzzy pooling, and a CoordConv layer to obtain feature f2. Then, it consists of four stacked hourglass modules, with an SDFusion module between every two consecutive hourglass modules. This module outputs a boundary heatmap for loss function calculation. The output of the last hourglass module is directly passed through a 1×1 convolution to obtain a boundary heatmap, which is then fed into the second-stage network along with the input.

[0039] The baseline network was enhanced in several ways by introducing a CoordConv layer, which performs better than using only ordinary convolutional layers. SDFusion modules were designed and added to effectively fuse shallow and deep features. Specifically, each SDFusion module fuses features from the early-stage CoordConv layer and features from the hourglass layer in deeper layers of the network. Notably, pyramid pooling modules were used in SDFusion to capture multi-scale contextual information.

[0040] Because of the great success of the Transformer in solving various vision tasks, it has become the core of coordinate regressors, such as... Figure 1 As shown in the second line of the document.

[0041] As Figure 1 As shown, a separate network was designed before the Transformer to effectively fuse the original image I, the latent features of the first stage, and the boundary heatmap b generated in the first stage. M Specifically, the three inputs are processed using channel concatenation, stacking, a small number of convolutions, and fuzzy pooling operations, followed by continuous channel number adjustment, resolution reduction, and feature extraction. Then, a self-attention-based feature re-extraction module, SAFetatere, was designed and implemented. Figure 1 As shown, this module contains four branches. The top two branches generate two feature maps P and Q, and use the matrix outer product to obtain two new feature maps PQ and PQ'. T Finally, these features are concatenated. The other two branches are then added pointwise to the aforementioned feature maps, followed by feature fusion through convolution and residuals. Finally, after dimensionality adjustment, the fused features are decoded by the Transformer to obtain the final result.

[0042] Specifically, this stage of the network has three inputs: input I from the first stage network, and boundary heatmap b. M And an additional feature map f. First, I and b M After resizing, the channels are concatenated, followed by 3x3 convolution and fuzzy pooling, then concatenated again with f. After a small amount of convolution and fuzzy pooling, the SAFeature model re-extracts features, adjusts the dimension of the feature map to conform to the input format of the transformer, and finally sends it to the transformer to obtain the prediction result.

[0043] exist Figure 1 The design of the keypoint coordinate prediction subnetwork has several highlights. First, Blur pooling is used multiple times after ordinary convolutional layers to achieve shift invariance. Second, a self-attention-based feature re-extraction module, SAFeature, is designed and used. The self-attention mechanism used in SAFeature helps capture long-term dependencies in the feature maps. Third, the Transformer also relies heavily on self-attention mechanisms that contribute to improving prediction accuracy.

[0044] In this embodiment of the application, it further includes: determining the loss function Loss, the overall Loss including and These correspond to the keypoint coordinate loss and the boundary heatmap loss, respectively. The total loss is defined as follows:

[0045]

[0046] in, Indicates the number of facial landmarks. and These represent the predicted keypoint coordinates and the labeled keypoint coordinates, respectively. This means that the number of times the boundary heatmap is predicted is equal to the number of stacked hourglass modules. and These represent the predicted boundary heatmap and the actual boundary heatmap, respectively. β represents the weights; β is a hyperparameter that adjusts the loss between these two types, and is set to 0.001 by default.

[0047] In this embodiment, the face matching result needs to be restored to a coordinate system with the size of the input image, that is, the horizontal coordinate is multiplied by the width of the input image, and the vertical coordinate is multiplied by the width of the input image. If it is necessary to draw the predicted key points onto the original image, the parameters of the face rectangle in the data preprocessing are used to multiply the model output result by the width and height of the rectangle, and then the key points are drawn onto the original image using the position of the rectangle.

[0048] To verify the network structure proposed in this application, a series of experiments were conducted in this embodiment. First, parameter settings were implemented: each input image was cropped and resized to 256×256, and each boundary heatmap was 64×64. Data augmentation was performed on the training data using random translation (±10%), rotation (±30◦), ​​horizontal flip (50%), brightness variation (±20%), blurring (10%), and occlusion. During model training, this embodiment used an Adam optimizer with an initial learning rate of 1×10−4, and β1 and β2 were set to 0.5 and 0.9, respectively. On a GPU (NVIDIA 3090 24GB), the network was trained 150 times, with the learning rate reduced to 1 / 10 of its previous value at the 90th and 120th iterations. The batch size was 16, and the weights ωi=1, 2, 3, and 4 in the loss function were 0.25, 0.5, 0.75, and 1.0, respectively. The deep learning framework used was PyTorch.

[0049] The experiments were conducted on two standard datasets: 300W and WFLW.

[0050] The 300W dataset contains 3,148 training images and 689 test images. Based on widely used evaluation settings, the test set typically includes a common set (554 images), a challenging set (135 images), and a full set (689 images in total). Each image in the dataset corresponds to 68 keypoint annotations.

[0051] The WFLW dataset contains 7500 training images and 2500 test images, with rich attribute labels. The test set also includes six subsets of different attribute labels: extreme pose, lighting, occlusion, makeup, blur, and expression. Each image in the dataset corresponds to 98 keypoint annotations.

[0052] Evaluation metrics. The evaluation metrics used in the embodiments of this application include normalized mean error (NME), failure rate (FR), and area under the curve (AUC), and the proposed method is compared with some state-of-the-art methods.

[0053] Figure 5 The results of some test images selected from the 300W and WFLW datasets are shown. For each set of test images, the actual and predicted keypoint locations are marked with dots in the left image; the right image shows the boundary heatmap estimated by this method. Except for the last case in the lower right corner, the predictions for most cases shown here are close to the baseline case. Because the test image in the lower right corner contains complex information such as large pose and severe occlusion, the boundary heatmap estimated by the method in this embodiment is not entirely accurate, which in turn affects the accuracy of the predicted keypoint coordinates.

[0054] For the 300W dataset, this embodiment compares the proposed method with several state-of-the-art methods on three test sets: the common set, the challenging set, and the full set. The results are shown in Table 1. The best and second-best results are highlighted in red and blue, respectively. The method of this embodiment performs best on the common set and relatively well on the full set. Due to the limited number of training images, the method of this embodiment performs slightly worse than the Awing method on the challenging set.

[0055] Table 1 Comparison results with state-of-the-art methods on 300W

[0056]

[0057] For the WFLW dataset, this application's embodiments compared different methods on the test set and several subsets (large pose, expression, illumination, makeup, occlusion, blur), and the comparison results are shown in Table 2. Although the NME value of the method in this application's embodiments is slightly higher than the minimum value obtained by SLPT on the entire test set, the method in this application's embodiments performs significantly better than SLPT under the FR metric. Regarding subset results, the method in this application's embodiments performs relatively better than other methods on the occlusion and blur subsets. The results indicate that this method is effective for face alignment.

[0058] Table 2 Comparison results with state-of-the-art methods on WFLW

[0059]

[0060] To evaluate the importance of the proposed SDFusion and SAFeature modules to keypoint detection results, experiments were conducted on WFLW and 300W in this application, and four different models with and without SDFusion and SAFeature were implemented. The evaluation method used in this application is NME, and the results are summarized in Table 3. Without SDFusion and SAFeature, the NME score increased sharply from 4.14% to 4.42%. It can be easily observed that the two blocks mentioned above are effective.

[0061] Table 3 Ablation experiments of SDFusion and SAFeature

[0062]

[0063] To evaluate the effectiveness of the enhanced HourglassNet in predicting boundary heatmaps, this embodiment compares the results with a baseline model. To better quantify the results, this embodiment uses the evaluation metric Structural Similarity (SSIM). The closer the SSIM value is to 1, the more similar the two images are. Table 4 shows the corresponding SSIM values ​​for the boundary heatmaps estimated by the two models and the actual heatmaps. It can be seen that the enhanced HourglassNet can significantly improve the accuracy of the model's predicted boundary heatmaps.

[0064] Table 4 Ablation experiments of enhanced HourglassNet

[0065]

[0066] In summary, this application proposes a boundary-aware face alignment neural network. It first estimates a boundary heatmap and then uses this heatmap to guide the prediction of keypoint coordinates. Accordingly, the proposed neural network consists of two stages: boundary heatmap estimation and keypoint coordinate prediction. In the first stage, the baseline network HoutglassNet is enhanced through Blur pooling, a CoordConv layer, and shallow and deep feature fusion modules. In the second stage, this application designs a Transformer-based subnetwork that first fuses information from the original image, latent features from the first stage, and the boundary heatmap generated in the first stage. Then, the Transformer is used to map the fused features to keypoint coordinates. During training, both the boundary heatmap and keypoint coordinate losses are considered. Experimental results show that the algorithm achieves state-of-the-art performance on two public datasets, 300W and WFLW.

[0067] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0068] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A face alignment method based on a boundary-aware face alignment neural network, characterized in that, include: The system comprises a boundary heatmap estimation subnetwork and a coordinate regression subnetwork. The boundary heatmap estimation subnetwork includes a CoordConv layer and multiple shallow and deep feature fusion (SDFusion) modules. The coordinate regression subnetwork includes a self-attention-based feature re-extraction (SAfeature) module and a Transformer-decoder module. The boundary heatmap estimation subnetwork takes the user's original face image information as input and generates a boundary heatmap as output. The coordinate regression subnetwork fuses the original face image information, the latent features of the boundary heatmap estimation subnetwork, and the generated boundary heatmap. The SAfeature module and the Transformer-decoder module are used to map the fused features to keypoint coordinates. The SDFusion module includes a first residual module. The input of the first residual module is the first image information. The output of the first residual module is connected to the input of the first convolution module and the pyramid pooling module. The output of the first convolution module is activated and outputs a first boundary sub-heatmap of the first image information. The first boundary sub-heatmap is input to the second convolution module and activated and output. The first image information, processed by the first residual module, is then concatenated with the second image information channel after passing through the pyramid pooling module to obtain the third image information. The pyramid pooling module is then sequentially connected to the third convolution module, the second residual module, and the fourth convolution module. After passing through the third convolution module, the second residual module, and the fourth convolution module, the third image information is multiplied element-wise with the first boundary sub-heatmap output by the second convolution module to obtain the fourth image information with enhanced contours. The SAfeature module includes a fifth, sixth, seventh, and eighth convolutional module arranged in parallel. The output information of the fifth and sixth convolutional modules is subjected to a matrix outer product operation to obtain two sets of feature maps. The two sets of feature maps are concatenated by channels to obtain new feature map information. The output information of the seventh convolutional module is added element-wise with the new feature map information and then input into the ninth convolutional module. The output information of the eighth convolutional module is added element-wise with the output information of the ninth convolutional module and then input into the third residual module. The output of the third residual module is connected to the input of the Transformer-decoder module. The face alignment result needs to be restored to a coordinate system relative to the input image size, i.e., the horizontal coordinate is multiplied by the width of the input image, and the vertical coordinate is multiplied by the width of the input image. If the predicted key points need to be drawn onto the original image, the parameters of the face bounding box in the data preprocessing are used, the model output result is multiplied by the width and height of the bounding box, and then the key points are drawn onto the original image using the position of the bounding box.

2. The face alignment method based on a boundary-aware face alignment neural network according to claim 1, characterized in that, The boundary heatmap estimation subnetwork further includes a tenth convolutional module, a fourth residual module, and a first fuzzy pooling module connected in sequence. The output of the first fuzzy pooling module is connected to the input of the CoordConv layer. The fifth residual module, the sixth residual module, and the first hourglass module are set in sequence between the CoordConv layer and the first SDFusion module. The first hourglass module is set between adjacent SDFusion modules. The output of the last SDFusion module is connected in sequence to the second hourglass module and the eleventh convolutional module. The output of the eleventh convolutional module is output as a boundary heatmap after passing through an activation function.

3. The face alignment method based on a boundary-aware face alignment neural network according to claim 2, characterized in that, The input of the first residual module in the first SDFusion module is the element-wise sum of the output of the sixth residual module and the output of the first hourglass module. The input of the first residual module in the remaining SDFusion modules is the element-wise sum of the output of the previous SDFusion module and the output of the first hourglass module. The second image information comes from the CoordConv layer.

4. The face alignment method based on a boundary-aware face alignment neural network according to claim 2 or 3, characterized in that, The coordinate regression subnetwork further includes a twelfth convolution module, a second fuzzy pooling module, a thirteenth convolution module, a fourteenth convolution module, a third fuzzy pooling module, a fifteenth convolution module, a fourth fuzzy pooling module, a sixteenth convolution module, and a fifth fuzzy pooling module connected in sequence. The output of the fifth fuzzy pooling module is connected to the input of the SAfeature module.

5. The face alignment method based on a boundary-aware face alignment neural network according to claim 4, characterized in that, The boundary heatmap obtained by the boundary heatmap estimation subnetwork is concatenated with the original face image information channel after the nearest neighbor difference operation and the average pooling operation, and then input into the twelfth convolution module. The output information of the second fuzzy pooling module is concatenated with the output information channel of the last SDFusion module and then input into the thirteenth convolution module.

6. The face alignment method based on a boundary-aware face alignment neural network according to claim 5, characterized in that, Determine the loss function, Loss; the overall Loss includes... and These correspond to the keypoint coordinate loss and the boundary heatmap loss, respectively. The total loss is defined as follows: in, Indicates the number of facial landmarks. and These represent the predicted keypoint coordinates and the labeled keypoint coordinates, respectively. This means that the number of times the boundary heatmap is predicted is equal to the number of stacked hourglass modules. and These represent the predicted boundary heatmap and the actual boundary heatmap, respectively. β represents the weights; β is a hyperparameter that adjusts the loss between these two types, and is set to 0.001 by default.

Images