A face alignment method and system based on local-global fusion attention

Abstract

This invention relates to a face alignment method and system based on local-global fusion attention. The method involves normalizing a face image and inputting it into a convolutional encoder to extract multi-resolution feature maps. A cascaded local-global fusion attention module is constructed, defining an initial global facial feature vector and initial facial keypoint coordinates. The multi-resolution high-dimensional feature maps are sequentially input into each level of the local-global fusion attention module in ascending order of resolution, progressively updating and optimizing the global facial feature vector and facial keypoint coordinates. The output facial keypoint coordinates of the last level of the local-global fusion attention module are used as the output of the face alignment model. The output facial keypoint coordinates of each level of the local-global fusion attention module are selected and compared with the ground truth values ​​of the keypoint coordinates to construct a multi-level loss function. This invention achieves end-to-end facial keypoint detection, enhancing the robustness and applicability of face alignment under image blur and facial occlusion conditions.

Description

Technical Field

[0001] This invention relates to the field of face alignment and facial landmark detection technology, specifically to a face alignment method based on local-global fusion attention. Background Technology

[0002] Face alignment, also known as facial landmark detection, refers to obtaining the coordinates of a set of specified facial landmarks, such as facial contour landmarks, pupil corner landmarks, and lip corner landmarks, from an input face image. Face alignment technology can be used for tasks such as expression analysis, facial effects, and face modeling, and the accuracy of its detection results directly affects the execution of subsequent tasks. Currently, most face alignment methods have weak detection capabilities in extreme conditions such as large-angle head rotation, exaggerated expressions, blurred images, and partial occlusion. Their applications are still limited to simple scenarios where the camera is facing directly at the viewer, and applying face alignment methods to unrestricted outdoor environments remains a significant challenge.

[0003] Early face alignment methods were primarily based on traditional machine learning algorithms, such as active shape models, active appearance models, and cascaded regression trees. These methods utilize the statistical properties of facial key points for key point modeling, searching, and matching. Machine learning-based methods, benefiting from their high computational efficiency, have been widely used in embedded systems. However, these methods are limited by the configuration of the initial statistical model and can only be used in simple frontal face scenarios.

[0004] Deep learning-based face alignment methods offer significant advantages over machine learning methods. These methods can be further categorized into two main types: coordinate regression-based methods and heatmap-based methods. Coordinate regression-based methods use convolutional neural networks to extract facial features, employing global pooling and fully connected layers to directly output the 2D coordinates of all keypoints. While direct coordinate regression is computationally efficient, the global pooling and fully connected layer structure causes the model to activate only certain locations, making it sensitive to the face's position in the image and exhibiting poor robustness. Heatmap-based methods use fully convolutional networks to output a high-resolution heatmap for each keypoint. Each heatmap represents the spatial response distribution of a keypoint, and the coordinates of the maximum response value in each heatmap are obtained using the maximum index operation, which represents the output coordinates of that keypoint. Fully convolutional networks make the output heatmaps spatially translation-invariant, thus unaffected by the face's position and offering high accuracy. However, performing convolution operations on high-resolution feature maps incurs significant computational and memory overhead, resulting in low efficiency. Furthermore, the maximum index operation limits the keypoint detection accuracy; achieving higher accuracy requires even higher-resolution heatmaps. Currently, coordinate regression-based methods are more widely used. Although heatmap-based methods offer higher detection accuracy, the computational cost of high-resolution feature maps and heatmaps makes them difficult to apply in practice. Therefore, how to combine the advantages of both methods, maintaining spatial translation invariance while avoiding the computational overhead of high-resolution feature maps, remains a subject for further research, and no relevant literature has yet reported on this topic. Summary of the Invention

[0005] This invention addresses the problem of overcoming the shortcomings of existing technologies by providing a face alignment method based on local-global fusion attention. The designed local-global fusion attention module can be embedded into an existing convolutional backbone network, using the convolutional backbone network as a convolutional encoder to achieve end-to-end facial keypoint detection. The designed local-global fusion attention module achieves interaction between local features of keypoints and global facial features through an attention mechanism, maintaining the structural correlation between different keypoints, thereby enhancing the robustness and applicability of face alignment under extreme conditions such as image blurring and facial occlusion.

[0006] Technical solution of the present invention:

[0007] Firstly, this invention provides a face alignment method based on local-global fusion attention, such as... Figure 1 As shown,

[0008] The specific steps are as follows:

[0009] Step S1: Normalize the face image and input it into the convolutional encoder to extract high-dimensional multi-resolution feature maps;

[0010] Step S2: Construct a multi-level local-global fusion attention module, defining the initial global facial feature vector and initial facial key point coordinates. Input the multi-resolution high-dimensional feature maps into each level of the module in order of increasing resolution, and update and optimize the global facial feature vector and facial key point coordinates level by level. Use the output facial key point coordinates of the last level as the output result of the face alignment model.

[0011] Step S3: Select the output facial key point coordinates of each level of local-global fusion attention module, construct a multi-level loss function with the true values ​​of the key point coordinates, and train the face alignment model.

[0012] like Figure 2 As shown, step S2 specifically includes:

[0013] Step S21: Based on the multi-resolution feature map extracted by the convolutional encoder Construct a multi-level local-global fusion attention module corresponding to the number of modules. Define the initial global facial feature vector. and initial facial key point coordinates , where T is the number of feature maps at different resolutions. These represent the width, height, and number of channels of the feature map, respectively. and It increases with the increase of t, where D is the dimension of the global feature vector and N is the number of facial key points;

[0014] Step S22: For the first The local-to-global fusion attention module repeats steps S23-S25:

[0015] Step S23: For the t-th level module, according to the key points The location, from the feature map Sampling is performed in the process, and the sampling results are encoded and mapped using a fully connected layer to obtain local features of key points. ;

[0016] Step S24: Use local features of key points and global facial features Calculate the local-global fusion attention weights, and aggregate the local features of each keypoint according to their weight contributions. Update the global facial features to obtain Then After nonlinear mapping and broadcasting, the local features of the key points are updated to obtain... ,Right now( Figure 3 ):

[0017]

[0018]

[0019] in, express and The inner product between the transposes, This represents summation of elements. This represents the sigmoid activation function. This represents a nonlinear broadcast layer, specifically a combination of a nonlinear activation function and a fully connected layer, which broadcasts the computation results to... Same size;

[0020] Step S25: Using a multilayer perceptron From the updated keypoint local features Regression relative to input coordinates offset Finally, the coordinates of the key points output by the t-th level module are calculated. ,Right now:

[0021]

[0022]

[0023] Step S26: After updating and optimizing through T cascaded modules, the final output coordinates of the last-level facial key points are obtained. This refers to the output of the face alignment model.

[0024] like Figure 3 As shown, step S3 includes:

[0025] Step S31: Select the offset relative to the input coordinates calculated by the local-global fusion attention modules at each level. Construct a set of offset prediction values Calculate the true coordinates of key points Input coordinates at each level The difference Construct offset label sets at each level. ;

[0026] Step S32: Set the offset labels As the true values ​​of the targets at each level, a multi-level loss function is constructed according to the following formula, where Let i be the L1 norm of the vector, and the superscript i denotes the index of the offset corresponding to the i-th keypoint. The model is trained by minimizing the multi-level loss function.

[0027]

[0028] Secondly, the present invention provides a face alignment system based on local-global fusion attention, comprising: a convolutional encoder construction and feature extraction module, a cascaded local-global fusion attention module, and a multi-level loss function construction and training module;

[0029] The convolutional encoder construction and feature extraction module normalizes the face image and inputs it into the convolutional encoder to extract high-dimensional multi-resolution feature maps. Based on the multi-resolution feature maps extracted by the convolutional encoder, a multi-level local-global fusion attention module is constructed, and the initial global facial feature vector and the initial facial key point coordinates are defined.

[0030] The cascaded local-global fusion attention module inputs multi-resolution high-dimensional feature maps into each level of the module in order of resolution from low to high, and updates and optimizes the global facial feature vector and facial key point coordinates at each level. The output of the last level of the local-global fusion attention module, the facial key point coordinates, is used as the output of the face alignment model.

[0031] The multi-level loss function construction and training module selects the output facial key point coordinates of each level of local-global fusion attention module, and constructs a multi-level loss function with the true values ​​of the key point coordinates to train the face alignment model.

[0032] Thirdly, the present invention provides an electronic device (computer, server, smartphone, etc.) including a processor and a memory;

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute computer programs stored in memory, which implement the methods or systems described herein.

[0035] Fourthly, the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program thereon, wherein the computer program, when executed by a processor, implements the method or the system described herein.

[0036] The advantages of this invention compared to the prior art are:

[0037] (1) The module proposed in this invention can be embedded into existing convolutional neural networks to achieve end-to-end face alignment.

[0038] Existing heatmap-based face alignment methods have the advantage of spatial translation invariance, but their output heatmaps require further transformation to obtain key point coordinates. The method of this invention can directly utilize existing convolutional networks as convolutional encoders, retaining the advantage of spatial translation invariance through fully convolutional operations. Furthermore, an embeddable cascaded regression module is designed to directly regress facial key point coordinates end-to-end, achieving end-to-end face alignment.

[0039] (2) This invention proposes a novel local-global fusion attention module to improve the accuracy and robustness of face alignment.

[0040] The local-global fusion attention module proposed in this invention is a cascaded module. As the number of cascaded layers increases, the resolution of the feature maps used also increases progressively, refining the local features of the sampled keypoints step by step. This achieves keypoint coordinate regression from coarse to fine, improving the accuracy of face alignment. Within a single-level module, the local-global fusion attention uses a global facial feature vector to constrain the structural relationships between local features of different keypoints, preventing overall shape distortion and enhancing the accuracy and robustness of face alignment. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0042] Figure 2 This is a structural diagram of the single-level local-global fusion attention module in this invention;

[0043] Figure 3 This is a diagram showing the internal structure of the local-global fusion attention module in this invention;

[0044] Figure 4 This is the alignment effect of the present invention on a subset of images in the WFLW (Wider Facial Landmark in the Wild) dataset. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] like Figure 1 As shown, this invention presents a face alignment method based on local-global fusion attention to enhance the robustness of facial keypoint detection. The method includes a convolutional encoder, a novel local-global fusion attention module, and a learnable global facial feature vector. and a set of predefined facial key points A convolutional encoder utilizes multiple levels of convolution and pooling operations to output a set of multi-resolution feature maps. The local-global fusion attention module is a cascaded module. A single-level module samples local features of keypoints from a feature map of a single resolution. Using a local-global fusion attention mechanism, it bidirectionally fuses and updates the sampled local features with a predefined global facial feature vector. The updated local features are then used to regress the facial keypoints of the current level. Based on the number of feature maps in the multi-resolution feature map, the example uses four local-global fusion attention modules in a cascade. For the first... The local-global fusion attention module takes the global facial feature vector as its input. and facial key points The output is the updated global facial feature vector. and facial key points This enables the step-by-step updating of the global facial feature vector and facial key points, with the final level outputting the facial key points. The key points are the output key points of the face alignment. For the face alignment model constructed by this method, this invention designs a multi-level loss function, with the facial key points output by the local-global fusion attention at each level as input. and key point true values The facial key point results output by local-global fusion attention at each level are supervised to train the face alignment model.

[0047] (1) Construction of convolutional encoder and feature extraction

[0048] The designed local-global fusion attention module can be directly embedded into existing convolutional neural networks, using the convolutional neural network as a convolutional encoder. The convolutional encoder is used to extract feature maps of different resolutions at multiple scales, and therefore can be directly constructed using existing convolutional neural networks.

[0049] One approach to constructing a convolutional encoder involves using a ResNet18 model with global pooling and fully connected layers removed as the feature extraction layer. This layer comprises four convolutional downsampling operations, outputting high-dimensional feature maps downsampled by 4, 8, 16, and 32 times the input image size, respectively. The number of extracted multi-resolution feature maps should satisfy T ≤ 4. For example, T = 4, and the input face image size is set to 256 × 256 pixels. The normalized face image is input into the ResNet18 convolutional encoder to obtain multi-resolution feature maps, which are then sorted in ascending order of resolution to obtain the feature map set. .

[0050] (2) Construction of cascaded local-global fusion attention module

[0051] Set the feature dimension D=192 and define the initial global facial feature vector. The facial landmark coordinates were randomly initialized using the 98-landmark annotation standard of the WFLW (Wider Facial Landmarks in-the-wild) dataset, i.e., the number of landmarks N=98. The initial facial landmark coordinates were defined using the average landmark coordinates of the training samples in the dataset. .

[0052] For the The t-th level local-global fusion attention module sequentially performs the update operations on the global feature vector and facial key points according to the cascaded order of the modules. The internal structure of the t-th level local-global fusion attention module is as follows: Figure 2 As shown, the input feature map Key point coordinates to be updated and global facial features to be updated Output the updated keypoint coordinates and updated global facial features The specific steps are as follows:

[0053] First, based on the input coordinates In feature map Point sampling is performed on the keypoint. The sampling method can be bilinear interpolation, followed by a fully connected layer for linear encoding mapping. This ensures that the dimension of the sampled features matches the set dimension D, thus sampling and mapping the local features of each keypoint. .

[0054] Next, the local features of the sampled key points are analyzed. and input facial global features Perform local-global fusion attention calculation to update local features of key points and global facial features. The calculation process is as follows: Figure 3 As shown. The attention weights are calculated according to the following formula, and the local features of key points are aggregated to obtain the updated global facial features. :

[0055]

[0056] in, express and The inner product between the transposes, This represents summation of elements. This represents the sigmoid activation function.

[0057] Then broadcast according to the following formula This is used to update the local features of keypoints, resulting in the updated local features of keypoints. :

[0058]

[0059] in, This is a non-linear broadcast layer, containing a non-linear activation function, a fully connected layer, and a broadcast operation. The broadcast operation, also known as a copy operation, reduces the data size from [previous size] to [new size]. Copy to ,Right now The size of the output result and Consistent.

[0060] Finally, the updated local features Input a multilayer perceptron Output relative to sampling coordinates offset Update the sampling coordinates to obtain the output facial key point coordinates of the t-th module. :

[0061]

[0062]

[0063] like Figure 1 , 2 In the process, after the calculation of the 4-level local-global fusion attention module, the coordinates of the facial key points output by the last level are... This serves as the final face alignment prediction result.

[0064] (3) Constructing a multi-level loss function and training

[0065] The offsets relative to the input coordinates calculated by the local-global fusion attention modules at each level are selected to construct a set of offset prediction values. True values ​​of key point coordinates Input coordinates at each level The difference Construct an offset label set .

[0066] Using the offset label set as the true target value for each level, a multi-level loss function is constructed according to the following formula, where The vector has an L1 norm, and the superscript i represents the index of the offset corresponding to the i-th keypoint. The loss function is accumulated in four stages:

[0067]

[0068] Training is performed by minimizing the loss function, so that the prediction bias at each level approaches the target bias. That is, for the input coordinates at each level, after being updated by the single-level local-global fusion attention module, the output coordinates all approximate the true values ​​of the keypoint coordinates. .

[0069] The face image is input into the trained model, and the output key points of the last-stage fusion attention module are selected. As a result of face alignment. Figure 4This invention demonstrates the alignment effect of the present invention on partial face images in the test set of the WFLW facial landmark dataset. The present invention achieves correct face alignment even under extreme conditions such as side profiles, exaggerated expressions, facial occlusion, complex makeup, and image blur, and has high accuracy and strong robustness.

[0070] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0071] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the various steps of the method of the present invention.

[0072] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A face alignment method based on local-global fusion attention, characterized in that, The implementation is as follows: Step 1: Normalize the face image and input it into the convolutional encoder to extract multi-resolution feature maps; Step 2: Based on the extracted multi-resolution feature maps, construct a cascaded local-global fusion attention module as a face alignment model, and determine the initial global facial feature vector and the initial facial key point coordinates. Input the multi-resolution feature maps into each level of the cascaded local-global fusion attention module in order of resolution from low to high, and update and optimize the global facial feature vector and the facial key point coordinates level by level. Step 3: Select the output facial key point coordinates of each level of local-global fusion attention module, construct a multi-level loss function with the true values ​​of the key point coordinates, train the face alignment model, and the output facial key point coordinates of the last level local-global fusion attention module are the output results of the face alignment model; Step 2 specifically includes: (21) Based on the extracted multi-resolution feature map Construct a cascaded local-global fusion attention module with the same number of multi-resolution feature maps. Where t is the number of feature maps in the multi-resolution feature map, t= , Let be the resolution size and the number of channels of the t-th feature map, respectively. and It increases with the increase of t; define the initial global facial feature vector. and initial facial key point coordinates , where D is the dimension of the global feature vector and N is the number of facial key points; for the t-th level local-global fusion attention module, repeat steps (22) to (24). (22) For the t-th level local-global fusion attention module, according to key points The position from the feature map Sampling is performed in the process, and the sampling results are encoded and mapped using a fully connected layer to obtain the local features of the key points. ; (23) Using key point local features and global facial features Calculate the local-global fusion attention weights, and aggregate the local features of each key point according to the weight contributions. Update the global facial features to obtain the updated global facial features. Then update the global facial features After nonlinear mapping and broadcasting, the updated local features of the key points are obtained. ,Right now: in, express and The inner product between the transposes, This represents summation of elements. This represents the sigmoid activation function. The nonlinear broadcast layer is a combination of a nonlinear activation function and a fully connected layer, which broadcasts the results to... Same size; (24) Using a multilayer perceptron From the updated keypoint local features Regression relative to input coordinates offset Finally, the coordinates of the key points output by the t-th level module are calculated. ,Right now: (25) After updating and optimizing T cascaded local-global fusion attention modules, the final output coordinates of facial key points at time t=T are obtained. This refers to the output of the face alignment model.

2. The face alignment method based on local-global fusion attention according to claim 1, characterized in that, Step 3 includes: (31) Select the offset relative to the input coordinates calculated by the local-global fusion attention modules at each level. Construct a set of offset prediction values Calculate the true coordinates of key points Input coordinates to local-global fusion attention modules at all levels The difference Construct offset label sets at each level. ; (32) Set the offset labels As the target ground truth values ​​for each level of local-global fusion attention module, a multi-level loss function is constructed according to the following formula, where Let L1 norm be the vector, and superscript i denote the index of the offset corresponding to the i-th keypoint. Training is performed using a multi-level loss function that minimizes the multi-level loss function. as follows: 。 3. A face alignment system based on local-global fusion attention that implements the method of claim 1 or 2, characterized in that, Includes: a convolutional encoder construction and feature extraction module, a cascaded local-global fusion attention module, and a multi-level loss function construction and training module; The convolutional encoder construction and feature extraction module normalizes the face image and inputs it into the convolutional encoder to extract multi-resolution feature maps. Based on the multi-resolution feature maps extracted by the convolutional encoder, a cascaded local-global fusion attention module is constructed, and the initial global facial feature vector and the initial facial key point coordinates are defined. The cascaded local-global fusion attention module inputs multi-resolution feature maps into each level of the local-global fusion attention module in order of resolution from low to high, and updates and optimizes the global facial feature vector and facial key point coordinates at each level. The output of the facial key point coordinates of the last level of the local-global fusion attention module is used as the output result of the face alignment model. The multi-level loss function construction and training module selects the output facial key point coordinates of each level of local-global fusion attention module, and constructs a multi-level loss function with the true values ​​of the key point coordinates to train the face alignment model.

4. An electronic device, characterized in that, Including processor and memory; Memory, used to store computer programs; A processor for executing a computer program stored in memory, which, when executed, implements the method of any one of claims 1-2 or the system of claim 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-2 or the system of claim 3.

Images