A lightweight face key point detection method based on dense attention

Abstract

The present invention belongs to the field of computer vision and artificial intelligence, and specifically relates to a lightweight facial key point detection method based on dense attention. The method comprises: obtaining a facial image to be detected, inputting the image to be detected into a trained lightweight facial key point detection model based on dense void self-attention, and obtaining lightweight facial key points; the lightweight facial key point detection model based on dense void self-attention comprises four modules, wherein the first module and the second module are both composed of an image downsampling module and a dense void self-attention module, and the third module and the fourth module are composed of an image downsampling module and a channel attention module; the present invention proposes a lightweight network structure, which pays more attention to spatial information in the early stage, and as channel information increases in the later stage, pays more attention to the extraction of channel information, and can effectively extract robust features.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and in particular relates to a lightweight facial key point detection method based on intensive attention. Background Art

[0002] Facial technologies are widely used in daily life. Facial landmark technology, as the foundation of many downstream facial tasks (such as face recognition, expression analysis, and 3D reconstruction), is a key research direction in the field of computer vision. Facial landmark detection involves locating key points on a face, including eyebrows, eyes, nose, mouth, and facial contours, within a given face image. This is a challenging task due to the influence of factors such as pose and occlusion. As deep learning network models become increasingly effective in image processing, neural networks are becoming larger and more complex, requiring more hardware resources for prediction and training. Deep learning neural network models are often only run on high-performance servers. Due to hardware and computing power limitations, mobile devices struggle to run complex deep learning network models. Therefore, research on lightweight facial landmark detection models is of great significance for the development of various downstream facial recognition tasks. Summary of the Invention

[0003] To solve the problems existing in the above-mentioned prior art, the present invention proposes a lightweight facial key point detection method based on dense attention, which comprises: obtaining a face image to be detected, inputting the image to be detected into a trained lightweight facial key point detection model based on dense void self-attention, and obtaining lightweight facial key points; the lightweight facial key point detection model based on dense void self-attention comprises four modules, wherein the first module and the second module are both composed of an image downsampling module and a dense void self-attention module, and the third module and the fourth module are composed of an image downsampling module and a channel attention module;

[0004] The process of training a lightweight face landmark detection model based on dense atrous self-attention includes:

[0005] S1: Obtain a face dataset and divide the data into a training set and a test set. The training set is used to train the model, and the test set is used to test the trained model.

[0006] S2: Preprocess the data in the training set;

[0007] S3: Input the preprocessed data into the image downsampling module;

[0008] S4: Input the downsampled image into the first module and the second module for feature extraction;

[0009] S5: Input the extracted feature map into the third module and the fourth module to extract channel attention features to obtain a feature map with edge information;

[0010] S6: Input the feature map with edge information into the fully connected layer to obtain the lightweight key point recognition result;

[0011] S7: Calculate the loss function of the model based on the lightweight key point recognition results, continuously adjust the parameters, and complete the model training when the loss function is minimized.

[0012] Preferably, the process of processing data using an image downsampling module includes: the image downsampling module is a convolution layer with a convolution kernel size of 4; downsampling the input image data through the convolution layer to obtain a face image with a size of 64×64.

[0013] Preferably, the process of extracting features from an input image using the first module and the second module includes:

[0014] S41: downsample the features of the previous module using the upsampling module of the first module to obtain an initial feature map;

[0015] S42: Input the initial feature map into the dense hole self-attention module and the edge feature extraction network respectively to obtain a first feature map and an initial edge feature map;

[0016] S43: Fusing the initial edge feature with the first feature map, and inputting the fused feature map into a fully connected layer to obtain a second feature map having the initial edge feature information;

[0017] S44: upsampling the second feature map using an upsampling module of the second module;

[0018] S45: Fusing the initial edge feature map and the feature map processed by the upsampling module of the second module, and inputting the fused features into the second edge feature extraction network to obtain an edge feature map;

[0019] S46: Input the upsampled feature map into the dense hole self-attention module to obtain the third feature map;

[0020] S47: Fuse the third feature map with the edge feature map, and input the fused feature map into the fully connected layer to obtain a feature map with edge information.

[0021] Furthermore, the process of using a dense void self-attention module to process the input feature map includes: the dense void self-attention module is composed of void downsampling and multiple self-attention modules; the features are densely void downsampled, and the feature size after downsampling is 1 / 4 of the original; the operation of performing void downsampling on the features includes rearranging and re-blocking the features; each 1 / 4 feature map is input into each self-attention module, and the output of each attention module is used as the input of the next attention module to obtain the first feature map.

[0022] Preferably, the channel attention module performs attention operation on the channel dimension. The input of the module is a regularized feature Y with a size of H×W×C. After passing through three different linear layers, Q, K, and V are obtained with a size of HW×C. Its expression is:

[0023] X=softmax(Q T ·K)·V

[0024] Beneficial effects of the present invention:

[0025] This paper proposes a lightweight network structure that focuses more on spatial information in the early stages and, as channel information increases in the later stages, places greater emphasis on extracting channel information, effectively extracting robust features. This technology also uses a novel downsampling method—atrous downsampling—to reduce feature size without sacrificing receptive field and spatial information. The present invention employs dense attention modules, each with its output serving as the input to the next module to extract more robust features. The implicit edge guidance module proposed in this paper allows the model to learn more edge features early on. Since most key points fall in locations with strong edge strength, this facilitates the accurate prediction of subsequent key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of lightweight facial key point detection based on edge guidance and dense hole attention of the present invention;

[0027] Figure 2 Schematic diagram of the dense hole self-attention module method of the present invention;

[0028] Figure 3 Schematic diagram of the channel attention module method of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] A lightweight facial key point detection method based on dense attention, the method comprising: obtaining a facial image to be detected, inputting the image to be detected into a trained lightweight facial key point detection model based on dense void self-attention, and obtaining lightweight facial key points; the lightweight facial key point detection model based on dense void self-attention comprises four modules, wherein the first module and the second module are both composed of an image downsampling module and a dense void self-attention module, and the third module and the fourth module are composed of an image downsampling module and a channel attention module.

[0031] like Figure 1 As shown in Figure 2, the process of training a lightweight face landmark detection model based on dense hole self-attention includes:

[0032] S1: Obtain a face dataset and divide the data into a training set and a test set. The training set is used to train the model, and the test set is used to test the trained model.

[0033] S2: Preprocess the data in the training set;

[0034] S3: Input the preprocessed data into the image downsampling module;

[0035] S4: Input the downsampled image into the first module and the second module for feature extraction;

[0036] S5: Input the extracted feature map into the third module and the fourth module to extract channel attention features to obtain a feature map with edge information;

[0037] S6: Input the feature map with edge information into the fully connected layer to obtain the lightweight key point recognition result;

[0038] S7: Calculate the loss function of the model based on the lightweight key point recognition results, continuously adjust the parameters, and complete the model training when the loss function is minimized.

[0039] A method for synthesizing thermal infrared and visible light facial images based on prior information consists of four steps: first, preparing a dataset and performing appropriate data preprocessing; then constructing a lightweight facial landmark detection method based on edge guidance and dense dilate attention, primarily comprising an image downsampling module, a dense dilate self-attention module (DDAM), and a channel attention module. Training and optimization are then performed using a combination of various loss functions and optimizers to update network parameters. After training, the optimal model is obtained, and the facial image is input to obtain coordinate values.

[0040] The preprocessing of the data in the training set includes: cleaning the data in the training set, cropping the cleaned images, and randomly rotating and flipping the cropped images; the processed image size is 256×256.

[0041] The process of processing data using the image downsampling module includes: the image downsampling module is a convolution layer with a convolution kernel size of 4; the input image data is downsampled through the convolution layer to obtain a face image with a size of 64×64.

[0042] The process of extracting features from an input image using the first module and the second module includes:

[0043] S41: downsample the features of the previous module using the upsampling module of the first module to obtain an initial feature map;

[0044] S42: Input the initial feature map into the dense hole self-attention module and the edge feature extraction network respectively to obtain a first feature map and an initial edge feature map;

[0045] S43: Fusing the initial edge feature with the first feature map, and inputting the fused feature map into a fully connected layer to obtain a second feature map having the initial edge feature information;

[0046] S44: upsampling the second feature map using an upsampling module of the second module;

[0047] S45: Fusing the initial edge feature map and the feature map processed by the upsampling module of the second module, and inputting the fused features into the second edge feature extraction network to obtain an edge feature map;

[0048] S46: Input the upsampled feature map into the dense hole self-attention module to obtain the third feature map;

[0049] S47: Fuse the third feature map with the edge feature map, and input the fused feature map into the fully connected layer to obtain a feature map with edge information.

[0050] Image downsampling module: The detection head in this design uses a classic four-stage network architecture. To reduce computational complexity, the image is first downsampled to 64×64 using a convolutional layer with a kernel size of 4. The feature map is then downsampled again before attention calculation at each stage, with each stage downsampling to half the size of the previous stage. By the final stage, an 8×8 feature map is obtained.

[0051] like Figure 2 As shown in the figure, the Dense Dilated Self-Attention Module (DDAM) consists of a dilated downsampling module and a dense self-attention module. Without reducing the receptive field, it performs dense dilated downsampling on features. The downsampled feature size is only one-fourth of the original size and is then fed into the self-attention module. The output of each attention module serves as the input of the next attention module. The dilated downsampling operation is a reversible process of rearranging and re-blocking the features, which is the key to this module not losing position information. The downsampling calculation formula is as follows:

[0052] x1,x2,x3,x4=split(gather(fold(x)))

[0053] Among them, Unfold means sliding out the local block, which corresponds to the "roll" operation of the sliding window in the convolution. Gather means re-sorting the features by row, and Split means separating the sorted features into four features of the same size. These four features are x1, x2, x3, and x4.

[0054] The dense self-attention module first performs attention calculation on x1 to obtain x′1 of the same size. When calculating x2, the information of x1 is added, and so on:

[0055] x′1=SA(x1)

[0056] x′2=SA(x′1+x2)

[0057] x′3=SA(x′1+x′2+x3)

[0058] x′4=SA(x′1+x′2+x′3+x4)

[0059] Among them, x′1 represents, SA represents, x′2 represents, x′3 represents, and x′4 represents.

[0060] The features after attention calculation are rearranged and merged to obtain a new feature. This is the inverse process of downsampling. The formula is as follows:

[0061] x=fold(gather(join(x1,x2,x3,x4)))

[0062] Among them, fold means restoring the extracted sliding area, and join means splicing four separated features.

[0063] Since the dense void attention module calculates the four downsampled features separately and loses the connection between the features, Fourier convolution is still used after this module. This convolution is used to extract the global spatial information of the features.

[0064] like Figure 3 As shown in the figure, the channel attention module performs attention operations on the channel dimension. The input of this module is a normalized feature Y with a size of H×W×C. It passes through three different linear layers to obtain Q, K, and V with a size of HW×C.

[0065] The classic spatial attention is to Q and K T Doing the dot product yields an attention map of size HW×HW. The entire lightweight network focuses on spatial attention in the first two stages, and in the last two stages we choose to perform attention operations on channels, focusing more on spatial information. The calculation formula for channel attention is as follows:

[0066] X=softmax(Q T ·K)·V

[0067] Among them, X represents the obtained features, Q T Represents the transpose of the query matrix, K represents the key, and V represents the value.

[0068] After four stages of calculation, we finally get a feature of size C×8×8, and then pass it through several fully connected layers to get the predicted coordinates of size N×2, where N is the number of coordinates.

[0069] The implicit edge guidance module, consisting of several simple convolutional layers, forms an edge extraction network. This module aims to leverage edge information to guide the neural network's earlier layers to learn more edge information. This module is only applied to the first two layers of the proposed four-stage network. The specific operation is as follows: The downsampled features from the first stage are fed into the edge extraction network to generate a resized edge feature map. This feature map is then fused with the output features of the dense atrous attention and fed into the second stage. After downsampling, the second stage performs the same operations as the first stage.

[0070] Loss1 uses L1Loss, which uses L1 loss to supervise the generation of key point coordinates. The generated coordinates are L Pred , the true value is L GT , the calculation formula of the loss function is:

[0071] Loss1=‖L Pred -L GT ‖2

[0072] Loss2 uses ACLoss to supervise the generation of edge information. First, the sobel operator is used to extract the edge information of the original image, which is recorded as F sobel , the feature output by the edge extraction network is recorded as F mid The loss function is calculated as:

[0073] Loss2=ACLoss(F sobel ,F mid )

[0074] Among them, Loss1 represents the first loss function, L Pred Represents the predicted key point coordinates, L GT represents the true key point coordinates, ‖.‖2 represents the L2 norm, Loss2 represents the second loss function, ACLoss represents the active contour loss, F sobel represents the true edge feature, F mid Represents the predicted intermediate edge features.

[0075] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A lightweight face key point detection method based on dense attention, characterized in that: include: Obtain a face image to be detected, input the image to be detected into a trained lightweight face key point detection model based on dense void self-attention, and obtain lightweight face key points; The lightweight face key point detection model based on dense void self-attention consists of four modules, of which the first and second modules are composed of image downsampling modules and dense void self-attention modules, and the third and fourth modules are composed of image downsampling modules and channel attention modules; The process of training a lightweight face landmark detection model based on dense atrous self-attention includes: S1: Obtain a face dataset and divide the data into a training set and a test set. The training set is used to train the model, and the test set is used to test the trained model. S2: Preprocess the data in the training set; S3: Input the preprocessed data into the image downsampling module; S4: Input the downsampled image into the first module and the second module for feature extraction; specifically including: S41: downsample the features of the previous module using the upsampling module of the first module to obtain an initial feature map; S42: Input the initial feature map into the dense hole self-attention module and the edge feature extraction network respectively to obtain a first feature map and an initial edge feature map; S43: Fusing the initial edge feature with the first feature map, and inputting the fused feature map into a fully connected layer to obtain a second feature map having the initial edge feature information; S44: upsampling the second feature map using an upsampling module of the second module; S45: Fusing the initial edge feature map and the feature map processed by the upsampling module of the second module, and inputting the fused features into the second edge feature extraction network to obtain an edge feature map; S46: Input the upsampled feature map into a dense void self-attention module to obtain a third feature map; specifically, the dense void self-attention module is composed of void downsampling and multiple self-attention modules; dense void downsampling is performed on the features, and the feature size after downsampling is 1 / 4 of the original size; wherein the void downsampling operation of the features includes rearranging the features and re-blocking them; each 1 / 4 feature map is input into each self-attention module, and the output of each attention module is used as the input of the next attention module to obtain the first feature map; S47: Fusing the third feature map with the edge feature map, and inputting the fused feature map into a fully connected layer to obtain a feature map with edge information; S5: Input the extracted feature map into the third module and the fourth module to extract channel attention features to obtain a feature map with edge information; S6: Input the feature map with edge information into the fully connected layer to obtain the lightweight key point recognition result; S7: Calculate the loss function of the model based on the lightweight key point recognition results, continuously adjust the parameters, and complete the model training when the loss function is minimized.

2. A lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The preprocessing of the data in the training set includes: cleaning the data in the training set, cropping the cleaned images, and randomly rotating and flipping the cropped images.

3. The lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The process of processing data using the image downsampling module includes: the image downsampling module is a convolution layer with a convolution kernel size of 4; the input image data is downsampled through the convolution layer to obtain a face image with a size of 64×64.

4. The lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The formula for dense hole downsampling of features is: x1,x2,x3,x4=Split(Gather(Unfold(x))) Among them, Unfold means sliding out the local block, which corresponds to the "roll" operation of the sliding window in the convolution. Gather means re-sorting the features by row, and Split means separating the sorted features into four features of the same size. These four features are x1, x2, x3, and x4.

5. The lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The formula for processing the input feature map using the self-attention module is: x′1=SA(x1) x′2=SA(x′1+x2) x′3=SA(x′1+x′2+x3) x′4=SA(x′1+x′2+x′3+x4) Among them, x′4 represents the feature after SA operation; SA represents the self-attention operation, and its expression is as follows: Q=W Q X K=W K X V=W V X Among them, Q, K, and V represent query, key, and value respectively, and Softmax represents the activation function.

6. The lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The channel attention module performs attention operations on the channel dimension. The input of this module is a regularized feature Y with a size of H×W×C. After passing through three different linear layers, Q, K, and V are obtained with a size of HW×C. Its expression is: X=softmax(Q T ·K)·V Among them, X represents the obtained features, Q T Represents the transpose of the query matrix, K represents the key, and V represents the value.

7. The lightweight face key point detection method based on intensive attention according to claim 1, characterized in that: The loss function of the model is: Loss1=‖L Pred -L GT ‖2 Loss2=ACLoss(F sobel ,F mid ) Loss = Loss1 + Loss2 Among them, Loss1 represents the first loss function, L Pred Represents the predicted key point coordinates, L GT represents the true key point coordinates, ‖.‖2 represents the L2 norm, Loss2 represents the second loss function, ACLoss represents the active contour loss, F sobel represents the true edge feature, F mid Represents the predicted intermediate edge features.

Images