Transformer-based dual-branch face image key point detection method

Abstract

The present invention discloses a dual-branch face image key point detection method based on Transformer, which is specifically implemented according to the following steps: using a dual-branch network to extract features from a face image; the dual-branch network includes a convolutional neural network, a Transformer branch network, a feature fusion module, and a progressive refinement module; designing a loss function, which includes L init , L refine ; Use the training set of the face database to train the dual-branch network and save the trained model; test the saved network model and calculate the coordinates of the L group of facial key points output by the model. n The method of the present invention solves the problems that the existing self-attention Transformer-based facial key point detection method cannot extract multi-scale features and the convolution operation of the convolutional neural network can only capture local information and cannot establish long-range dependencies of the global image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and particularly relates to a dual-branch face image key point detection method based on Transformer. Background Art

[0002] The purpose of facial landmark detection is to locate a set of predefined facial landmarks from an image. Currently, facial landmark detection models can be mainly divided into traditional methods based on manual features and regression methods based on deep learning. Traditional methods mainly include AAMs (active appearance models), CLMs (constrained local models), ESR (explicit shape regression), SDM (supervised descent method), etc. These methods mainly use manually designed features to fit the positions of facial landmarks. In recent years, with the rapid development of deep learning methods represented by convolutional neural networks (CNNs) and recurrent neural networks (RNNs), facial landmark location algorithms have also rapidly shifted to methods based on deep learning.

[0003] Generally speaking, facial key point localization based on deep learning can be divided into coordinate regression-based methods and heatmap regression-based methods. Heatmap regression-based methods focus more on the accuracy of the results, but the model size is large and the speed is slow. Coordinate regression-based methods are fast and computationally intensive, but their accuracy is not as high as that of heatmap regression-based methods. In addition, coordinate regression-based methods directly regress key point coordinates from feature maps through fully connected layers, which destroys local features. Existing methods have also proposed a variety of cascade networks and recursive networks to achieve multi-stage facial key point detection. Although coordinate regression methods have the inherent potential to learn internal relationships, they usually require a large number of samples for training.

[0004] While deep learning methods have made progress, some challenges remain, particularly for images with significant occlusion, large pose variations, and lighting changes. In practical applications, especially under real-time low-quality image conditions, when pose changes dramatically and occlusion is high, the accuracy of existing methods still lags far behind practical applications, requiring further research and improvement. Furthermore, lighting, facial expressions, and makeup can also affect the accuracy of facial landmark location. Facial landmark detection is fundamental to many facial tasks, and in industrial applications, tracking is often required to improve the stability of keypoints.

[0005] Existing deep learning-based methods have two limitations: 1) Convolutional neural networks focus on local features and lack the inherent connections between global points. The structure of the face is regular, so the inherent connections between facial points are crucial for keypoint detection. 2) In regression methods, fully connected layers help the model learn the inherent connections between points, but they also destroy local features. Therefore, the current state of the art still requires a deep learning network that can provide high-quality local and global feature representations. Summary of the Invention

[0006] The purpose of the present invention is to provide a dual-branch face image key point detection method based on Transformer, which solves the problem of low accuracy of face image key point detection in existing methods.

[0007] The technical solution adopted by the present invention is a dual-branch face image key point detection method based on Transformer, which is implemented in the following steps:

[0008] Step S1: extract features from the face image using a dual-branch network; the dual-branch network includes a convolutional neural network, a Transformer branch network, a feature fusion module, and a progressive refinement module;

[0009] Step S2, design the loss function, the loss function includes L init , L refine ;

[0010] Step S3: Use the training set of the face database to train the two-branch network and save the trained model;

[0011] Step S4: Test the network model saved in step S3 and calculate the coordinates of the L groups of facial key points output by the model. n The normalized error value compared to the true value data.

[0012] The present invention is also characterized in that:

[0013] In step S1, specifically:

[0014] Step S1-1: extract features from the face image using a convolutional neural network to obtain a feature map of size W / 4 × H / 4;

[0015] The convolutional neural network includes a shallow feature extraction module and a deep feature extraction module. The shallow feature extraction module includes four encoders; the deep feature extraction module includes three decoders;

[0016] The input W×H face image is subjected to a deformable convolution operation to obtain a feature map of size W / 2×H / 2, which is then normalized and activated by the Relu function and then subjected to maximum pooling to obtain a feature map of size W / 4×H / 4; the feature map of size (W / 4×H / 4) is first subjected to a deformable convolution operation by the second encoder to obtain a feature map of size W / 4×H / 4, which is then normalized and activated by the Relu function and then subjected to maximum pooling to obtain a feature map of size W / 8×H / 8. Similarly, after passing through the third encoder, a feature map of size W / 16×H / 16 is obtained, and after passing through the fourth encoder, a feature map of size W / 32×H / 32 is obtained.

[0017] The obtained W / 32×H / 32 size feature map is input to the first decoder, and first undergoes an upsampling operation to obtain a W / 16×H / 16 size feature map, and then undergoes variable convolution, normalization operation and ReLU function activation to finally obtain a W / 16×H / 16 size feature map. Similarly, after passing through the second decoder, a W / 8×H / 8 size feature map is obtained, and after passing through the third decoder, a W / 4×H / 4 size feature map is obtained.

[0018] Step S1-2: The face image is passed through the Transformer branch network for feature extraction to obtain a feature image of size W / 16 × H / 16;

[0019] The Transformer branch network consists of three Transformer blocks, each of which includes: patch merging, LayerNorm, SW-MSA, and MLP;

[0020] Specifically, the patch merging here is a convolution operation. The input W×H image undergoes patch merging and normalization operations, and then is input into the SW-MSA module. The feature map is first divided into image blocks of m×m size, and then the self-attention is calculated for each image block to obtain a feature map of W / 4×H / 4 size. The output before normalization is connected to the input after the SW-MSA module through a residual connection, and then normalized through the Layernorm module. The normalized result is then input into the MLP module for nonlinear transformation, and the normalized result is residually connected with the output of the nonlinear transformation, and finally the feature map of W / 4×H / 4 size is output; and so on, after the second Transformer block, the feature map of W / 8×H / 8 size is output, and after the third Transformer block, the feature map of W / 16×H / 16 size is output;

[0021] Step S1-3, perform dot multiplication on the feature map output by each encoder in the convolutional neural network and the feature map of the same scale in the feature map at three scales obtained by three Transformer blocks to obtain a fused feature map;

[0022] Step S1-4, performing channel splicing on the feature map obtained after the primary fusion in step S1-3 and the feature of the same scale generated by the decoder in the convolutional neural network to obtain a secondary fusion feature map with unchanged scale and increased number of channels;

[0023] In step S1-5, the feature maps at three scales obtained by three transformer blocks are channel-concatenated with the features of the same scale generated by the decoder in the convolutional neural network to obtain a three-fold fused feature map with unchanged scale and doubled number of channels.

[0024] Step S1-6: The obtained three-fold fusion feature map is subjected to feature fusion by a feature fusion module to obtain an initial landmark L1 of M groups of key points;

[0025] Step S1-7: Input the feature map output in step S1-6 and the initial landmark L1 of the M groups of key points into the progressive refinement module, and obtain the corresponding N iterations to obtain the refined key points L2, L3...L N .

[0026] In step S1-6, specifically:

[0027] The feature map of size W / 4×H / 4 is upsampled four times. These four upsampling operations are four deconvolution operations. After the first deconvolution of the feature map of size W / 4×H / 4, a feature map of scale W / 2×H / 2 is output. After four deconvolution operations, the feature map of size W / 4×H / 4 is finally expanded to a feature map of scale W×H. The feature map obtained by upsampling passes through three convolutional layers and one fully connected layer to generate the initial landmark L1 of M groups of key points.

[0028] In step S1-7, the progressive refinement module consists of a Transformer encoder and a fully connected layer; the Transformer encoder includes: patch merging, LN, MSA or MCSA, MLP;

[0029] The progressive refinement module performs N iterations in total, and the process is as follows:

[0030] In the first iteration, the W / 2×H / 2 scale feature map output after the first deconvolution in step S1-6 is evenly divided into N×N windows, and a local block of size T×T is cropped on the corresponding window with the generated initial key point L1 as the center; first, the T×T dimension of the T×T size feature local block is flattened by the fatten operation to obtain a one-dimensional sequence feature; the absolute position of the local block vector is encoded from 1 to the total number of key points L; that is, each position has a separate encoding vector; then the encoding vector is summed with the local block vector to output a feature sequence, the feature sequence is processed using layer normalization, and the processed features are sent to the kernel. The feature sequence that has not been subjected to the LN operation is residually connected with the output of the MSA. The result of the residual connection is subjected to the LN operation and then sent to the MCSA block. The feature sequence that has not been subjected to the LN operation is residually connected with the output of the MCSA, and the feature sequence is processed using LN. The processed features are then sent to the MLP block. The feature sequence that has not been subjected to the LN operation is residually connected with the output of the MLP to finally obtain the output result of the Transformer encoder layer; the result is input to the fully connected layer to output the relative position of the key point coordinates relative to the output key point of the previous iteration, thereby obtaining the refined coordinates of this iteration;

[0031] In the second iteration, the number of windows is 2N×2N, and the local block size becomes T / 2×T / 2. Similarly, the number of windows doubles and the local block size decreases by two times. In each iteration, this local block is projected as a patch into a vector of fixed length, and the result is sent to a layer of Transformer encoder. The iterative process is repeated to finally obtain N groups of refined key points L2, L3...L N .

[0032] In step 2, specifically:

[0033] Step S2-1, calculate the normalized average error value L using the initial key point position L1 and the true value data init , as shown in formula (5):

[0034]

[0035] Among them, L i is the key point position, i=1,2…N, N is the number of key points; L g is the true value data, d is the distance between the pupils of the two eyes of the face;

[0036] Step S2-2, using N iterations to obtain the refined key points L2, L3...L N Calculate the normalized average error value L with the true value data respectively refine , as shown in formula (6):

[0037]

[0038] Among them, λ1, λ2, ....λ n are different weight values; λ0, λ1, λ2, ....λ n The sum of the weights is 1;

[0039] Step S2-3, L init , L refine The Loss is obtained by adding them together, and the network parameters are updated according to the Loss, so that the network gradually converges.

[0040] In step S3, specifically:

[0041] Step S3-1: Data augmentation is performed on the original images taken from the training set. This is done by rotating each sample from -30° to 30° in intervals of 10° to increase the training data. In addition, each face sample is subjected to random occlusion and noise addition operations. The images in the dataset are then fed into the network for training.

[0042] Step S3-2, select the network optimizer as ADAM optimizer, set the initial learning rate to 0.1, and the scheduler strategy to multi-step decay strategy. The decay steps are 25 and 100 respectively, and the decay is half each time. A total of 200 Epochs are trained. By observing L init , L refine Two indicators dynamically adjust hyperparameters, such as learning rate lr and training round number Epoch;

[0043] Step S3-3, use the SummaryWriter of the Python third-party library tensorboard to write the training process L init , L refine The reference indicators are output to tensorboard, and the network parameters trained in step S3-2, the number of epochs being trained, the optimizer ADAM, and the scheduler are saved to obtain the trained network model.

[0044] The beneficial effect of the present invention is that the method of the present invention solves the problem that the existing self-attention Transformer-based facial key point detection method cannot extract multi-scale features and the convolution operation of the convolutional neural network can only capture local information and cannot establish long-range dependencies of the global image. In the progressive refinement module, the feature map is divided into several windows and local blocks are set respectively, which allows the network to focus only on the adjacent areas of the key points, which reduces the number of model parameters and the amount of calculation compared to the entire image. The data enhancement and data complexity of the data set are selected to reduce the overfitting phenomenon of the network, and a network with stronger generalization ability can be obtained to better adapt to application scenarios. The loss function gives higher weights to small number samples and large posture samples, which solves the problem that the existing training samples are unbalanced in each category and have diverse posture changes. The present invention can improve the performance of face detection and recognition algorithms in applications such as video surveillance, fatigue driving, and face recognition, and provide high-precision algorithm support for expanding digital face applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the overall structure of the Transformer-based dual-branch face image key point detection method of the present invention;

[0046] Figure 2 It is a schematic diagram of the encoder-decoder and Transformer block structure of the convolutional neural network of the present invention;

[0047] Figure 3 It is a structural diagram of the feature fusion module in the present invention;

[0048] Figure 4 It is a structural diagram of the progressive refinement module in the present invention;

[0049] Figure 5 Schematic diagram of the structure of the Transformer layer of the progressive refinement module in the present invention; DETAILED DESCRIPTION

[0050] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] The present invention is based on the Transformer dual-branch face image key point detection method, which is specifically implemented according to the following steps:

[0053] Step S1: extract features from the face image using a dual-branch network; the dual-branch network includes a convolutional neural network, a Transformer branch network, a feature fusion module, and a progressive refinement module;

[0054] Step S2, design the loss function, the loss function includes Linit , L refine ;

[0055] Step S3: Use the training set of the face database to train the two-branch network and save the trained model;

[0056] Step S4: Test the network model saved in step S3 and calculate the coordinates of the L groups of facial key points output by the model. n The normalized error value compared to the true value data.

[0057] Example 2

[0058] The present invention is based on the dual-branch face image key point detection method of Transformer, such as Figure 1 As shown, the specific steps are as follows:

[0059] Step S1: extract features from the face image using a dual-branch network, which includes a convolutional neural network, a Transformer branch network, a feature fusion module, and a progressive refinement module; specifically:

[0060] Step S1-1: extract features from the face image using a convolutional neural network to obtain a feature map of size W / 4 × H / 4;

[0061] Where W is the image width and H is the image height;

[0062] like Figure 2 As shown in the figure, the convolutional neural network is improved based on the U-Net model, including a shallow feature extraction module and a deep feature extraction module. The shallow feature extraction module includes four encoders; the deep feature extraction module includes three decoders;

[0063] Each encoder includes a deformable convolution (DCN), batch normalization (BN), Relu activation function, and max pooling to extract image features layer by layer. The convolution kernel (kernel_size) of the deformable convolution (DCN) is 3, the padding size parameter (padding) is set to 1, the first encoder stride is set to 2, and the remaining strides are set to 1; the pooling kernel size of the max pooling (MaxPooling) is 2×2, and the padding size parameter (padding) is set to 0;

[0064] Each decoder includes upsampling, variable convolution (DCN), normalization (BN), and Relu activation function to restore the position information of the image. The decoder will perform upsampling (Upsample), which is a deconvolution (ConvTranspose2d) operation. The deconvolution kernel (kernel_size) size is 3, the stride is set to 2, and the padding size parameter (padding) is set to 1. The variable convolution (DCN) kernel (kernel_size) size is 3, the padding size parameter (padding) is set to 1, and the stride of the first encoder is set to 1.

[0065] The input face image (W×H) is subjected to a deformable convolution operation to obtain a feature map of size (W / 2×H / 2), which is then normalized and activated by the Relu function and then subjected to maximum pooling to obtain a feature map of size (W / 4×H / 4); the feature map of size (W / 4×H / 4) is first subjected to a deformable convolution operation by the second encoder to obtain a feature map of size (W / 4×H / 4), which is then normalized and activated by the Relu function and then subjected to maximum pooling to obtain a feature map of size (W / 8×H / 8), and so on. After passing through the third encoder, a feature map of size (W / 16×H / 16) is obtained, and after passing through the fourth encoder, a feature map of size (W / 32×H / 32) is obtained.

[0066] The obtained feature map of size (W / 32×H / 32) is input into the first decoder, and first undergoes an upsampling operation to obtain a feature map of size (W / 16×H / 16), and then undergoes variable convolution, normalization operation and ReLU function activation to finally obtain a feature map of size (W / 16×H / 16). Similarly, after passing through the second decoder, a feature map of size (W / 8×H / 8) is obtained, and after passing through the third decoder, a feature map of size (W / 4×H / 4) is obtained.

[0067] Step S1-2: The face image is passed through the Transformer branch network for feature extraction to obtain a feature image of size W / 16 × H / 16;

[0068] The Transformer branch network consists of three Transformer blocks. Each Transformer block includes: patch merging, layer normalization (LayerNorm), moving window self-attention (SW-MSA), and multi-layer perceptron (MLP).

[0069] Specifically, the patch merging here is a convolution operation. The convolution kernel (kernel_size) of the first Transformer block is 3 and the stride is 4. The convolution kernel (kernel_size) of the remaining two Transformer blocks is 3 and the stride is 2. First, the input (W×H) size image is processed by the first Transformer block: the input (W×H) size image undergoes patch merging and normalization operations, and then input into the SW-MSA module. The feature map is first divided into image blocks of size m×m, and then self-attention is calculated for each image block to obtain a feature map of size (W / 4×H / 4). The output before normalization is connected to the input after the SW-MSA module through a residual connection, and then normalized through the Layernorm module. The normalized result is then input into the MLP module for nonlinear transformation, and the normalized result is residually connected with the output of the nonlinear transformation, and finally the feature map of size (W / 4×H / 4) is output; and so on, after the second Transformer block, the feature map of size (W / 8×H / 8) is output, and after the third Transformer block, the feature map of size (W / 16×H / 16) is output.

[0070] Step S1-3, perform dot multiplication on the feature map output by each encoder in the convolutional neural network and the feature map of the same scale in the feature map at three scales obtained by three Transformer blocks to obtain a fused feature map;

[0071] Step S1-4, performing channel splicing on the feature map obtained after the primary fusion in step S1-3 and the feature of the same scale generated by the decoder in the convolutional neural network to obtain a secondary fusion feature map with unchanged scale and increased number of channels;

[0072] In step S1-5, the feature maps at three scales obtained by three transformer blocks are channel-concatenated with the features of the same scale generated by the decoder in the convolutional neural network to obtain a three-fold fused feature map with unchanged scale and doubled number of channels.

[0073] Step S1-6: The obtained three-fold fusion feature map is subjected to feature fusion by a feature fusion module to obtain an initial landmark L1 of M groups of key points.

[0074] like Figure 3As shown, the feature map of size (W / 4×H / 4) is upsampled four times. These four upsamplings are four deconvolution (ConvTranspose2d) operations. The convolution kernel (kernel_size) of the first deconvolution is 3, the stride is set to 2, and the padding size parameter is set to 1. The feature map of size (W / 4×H / 4) is expanded to a feature map of scale (W / 2×H / 2). The convolution kernel (kernel_size) of the second deconvolution is 3, the stride is set to 1, and the padding size parameter is set to 1. ing) is set to 1, the convolution kernel (kernel_size) of the third deconvolution is 3, the stride (stride) is set to 1, the padding size parameter (padding) is set to 1, the convolution kernel (kernel_size) of the fourth deconvolution is 3, the stride (stride) is set to 2, and the padding size parameter (padding) is set to 1, and the feature map of the (W / 2×H / 2) scale is expanded to the feature map of the (W×H) scale; the feature map obtained by upsampling passes through three convolution layers (convolution kernel size is 3, stride is 2) and a fully connected layer (FC) to generate the initial landmark L1 of M groups of key points.

[0075] Step S1-7: Input the (W / 2×H / 2) scale feature map output after the first deconvolution in step S1-6 and the initial landmark L1 of the M groups of key points into the progressive refinement module, and obtain the corresponding N iterations to obtain the refined key points (L2, L3...L N ).

[0076] like Figure 4 and Figure 5 As shown in Figure 1, the progressive refinement module consists of a Transformer encoder and a fully connected layer. The Transformer encoder includes: patch embedding (patch merging), normalization (LN), multi-head self-attention (MSA) or multi-head cross self-attention (MCSA), and a multi-layer perceptron (MLP).

[0077] The formulas for MSA are shown in (1)-(3):

[0078] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (1);

[0079] head i =Attention(QW i Q , KW i K , VWi V ) (2);

[0080]

[0081] in, d model Indicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key; Q, K, V are divided into num_heads in the feature dimension, divided into head1, ..., head h .

[0082] The progressive refinement module performs N iterations in total, and the process is as follows:

[0083] In the first iteration, the (W / 2 × H / 2)-scale feature map output after the first deconvolution in step S1-6 is evenly divided into N × N windows. A T × T local patch is cropped from the window centered on the generated initial keypoint L1. The T × T dimensions of the T × T local patch are first flattened using a fatten operation to produce a one-dimensional sequence feature. The absolute position of the local patch vector is encoded from 1 to the total number of keypoints, L. In other words, each position has a separate encoding vector. The encoding vector is then summed with the local patch vector to produce a feature sequence. Layer normalization is applied to the feature sequence, which is then fed into the MSA block. The unprocessed feature sequence is residually connected with the MSA output. The resulting residual connection is then processed by the LN operation and fed into the MCSA block. The unprocessed feature sequence is residually connected with the MCSA output and then processed by LN. The processed feature sequence is then fed into the MLP block. The unprocessed feature sequence is residually connected with the MLP output, ultimately resulting in the output of the Transformer encoder layer. The result is input into the fully connected layer to output the relative position of the key point coordinates relative to the key point output of the previous iteration, thereby obtaining the refined coordinates of this iteration.

[0084] In the second iteration, the number of windows is 2N×2N, and the local block size becomes T / 2×T / 2. Similarly, the number of windows doubles and the local block size doubles. In each iteration, this local block is projected as a patch into a fixed-length vector, and the result is sent to a layer of Transformer encoder. The iterative process is repeated, and finally N groups of refined key points (L2, L3...L N ).

[0085] Step S2, design the loss function, which consists of two parts, including L init, L refine , to achieve unsupervised learning;

[0086] The total loss function is shown in formula (4):

[0087]

[0088] Since the two losses have different effects on the network, a hyperparameter λ1 is set to affect their weights.

[0089] is the weight term, which consists of two parts, including ω n and (1-cosθ).

[0090] is the sample weight value of the cth category. According to different conditions such as different facial postures, lighting changes, occlusion, and expressions, faces can be divided into c categories. The number distribution of these c categories in the training set is counted and the proportion of each category in the total number of samples is calculated. If the proportion of the cth category is n, then The size is 1-n.

[0091] 1-cosθ is the angle weight. θ1, θ2, and θ3 (K=3) refer to the yaw, roll, and pitch angles. Larger angles in large gesture samples have smaller cosθ values ​​and larger 1-cosθ values, giving them a higher weight compared to other samples.

[0092] K is the number of groups of facial key points, commonly used ones are 98 groups, 68 groups, etc.

[0093] Specifically: Step S2-1, use the initial key point position L1 and the true value data (GroundTruth) to calculate the normalized average error value L init , as shown in formula (5):

[0094]

[0095] Among them, L i is the key point position, i=1,2…N, N is the number of key points; L g is the true value data, d is the distance between the pupils of the two eyes of the face;

[0096] Step S2-2, using N iterations to obtain the refined key points (L2, L3...L N ) and the true value data (GroundTruth) to calculate the normalized average error value L refine , as shown in formula (6):

[0097]

[0098] Among them, L1, L2...L N is the position of N groups of refined key points, L g is the true value data, N is the number of key point groups, and d is the distance between the pupils of the two eyes of the face. n are different weight values. λ0, λ1, λ2, ....λ n The sum of the weights is 1.

[0099] Step S2-3, L init , L refine The Loss is obtained by adding them together, and the network parameters are updated according to the Loss, so that the network gradually converges.

[0100] Step S3: Use a large-scale face database including multi-pose and multi-view to train the neural network built by the Transformer-based dual-branch face image key point detection algorithm for 200 epochs, verify the training results, and save the neural network model; specifically:

[0101] Step S3-1: Perform data augmentation on the original images extracted from the training set of the dataset used, that is, rotate each sample from -30° to 30° in intervals of 10° to increase the training data. In addition, each face sample is subjected to random occlusion and noise addition operations, and the images in the dataset are input into the network for training; Step S3-2: Select the network optimizer as ADAM optimizer, set the initial learning rate to 0.1, and adjust the scheduler strategy to multi-step decay strategy, with the decay steps of 25 and 100 respectively, and decay half each time, for a total of 200 epochs. By observing L init , L refine Two indicators dynamically adjust hyperparameters: learning rate lr and number of training rounds Epoch;

[0102] Step S3-3, use the SummaryWriter of the Python third-party library tensorboard to write the training process L init , L refine Output the reference indicators to tensorboard, save the neural network parameters trained in step S3-2, the number of epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0103] Step S4: Use the images in the test set of the dataset as test data to test the network. Test the dual-branch face key point detection network model saved in step S3, and calculate the L groups of face key point coordinates (L n ) and the normalized error between the true data. The lower the error, the more accurate the facial key points predicted by the model.

[0104] The detection method of the present invention solves the problem of imbalance in various categories and diverse posture changes in existing training samples, improves the detection accuracy of key points, and realizes robust facial key point detection. It can be used in face-related application fields such as face alignment, face reproduction, face swapping and driver fatigue detection.

[0105] Example 3

[0106] The network was trained using 45,000 augmented images from the WFLW dataset. The image size was set to 128×128 pixels, the initial learning rate was set to 0.1, and the number of epochs was set to 200 when the loss function converged.

[0107] The network was tested using 2,500 images from the WFLW dataset test set. The image size was set to 128×128 pixels. The two-branch facial landmark detection network model saved in step S3 was tested, and the normalized error between the 98 sets of facial landmark coordinates output by the model and the ground truth was calculated. Lower error values ​​indicate more accurate facial landmark predictions.

[0108] The proposed method was compared with existing methods for facial landmark detection using the Normalized Mean Error (NME), Failure Rate (Failure Rate), and Area Under Correlation (AUC). The test results for the official WFLW and 300W datasets are shown in Tables 1 and 2, respectively. This demonstrates the significant potential of the proposed method in the field of facial landmark detection, outperforming competing methods and achieving superior results on these datasets. As shown in Table 3, the proposed method also outperforms existing methods in terms of model size and processing speed.

[0109] Table 1 Comparison of NME, Failure Rate, and AUC between the present invention and existing methods on the WFLW dataset

[0110] Metric Method Test set Pose Expression Illumination Make-U OcclusionBlur

[0111] p

[0112]

[0113] Table 2 Comparison of NME indicators between the present invention and the existing method on the 3 million dataset

[0114]

[0115]

[0116] NME: normalized mean error;

[0117] Failure Rate represents the percentage of images in the test set whose NME is higher than a certain threshold;

[0118] AUC is calculated based on the Cumulative Error Distribution (CED) curve. It represents the proportion of test images where the NME (%) is less than or equal to the value on the horizontal axis. AUC is the area under the CED curve, from 0 to the Failure Rate threshold.

[0119] Table 3 Comparison of parameters and computational complexity between this method and existing methods

[0120]

Claims

1. A dual-branch face image key point detection method based on Transformer, characterized by: The specific implementation steps are as follows: Step S1: Extract features from facial images using a dual-branch network; the dual-branch network includes a convolutional neural network, a Transformer branch network, a feature fusion module, and a progressive refinement module; specifically: Step S1-1: extract features from the face image using a convolutional neural network to obtain a feature map of size W / 4 × H / 4; The convolutional neural network includes a shallow feature extraction module and a deep feature extraction module. The shallow feature extraction module includes four encoders; the deep feature extraction module includes three decoders; The input W×H face image is subjected to a deformable convolution operation to obtain a feature map of size W / 2×H / 2, which is then normalized and activated by the ReLU function and then max pooled to obtain a feature map of size W / 4×H / 4. Similarly, after passing through the second encoder, a feature map of size W / 8×H / 8 is obtained, after passing through the third encoder, a feature map of size W / 16×H / 16 is obtained, and after passing through the fourth encoder, a feature map of size W / 32×H / 32 is obtained. The obtained W / 32×H / 32 size feature map is input to the first decoder, and first undergoes an upsampling operation to obtain a W / 16×H / 16 size feature map, and then undergoes variable convolution, normalization operation and ReLU function activation to finally obtain a W / 16×H / 16 size feature map. Similarly, after passing through the second decoder, a W / 8×H / 8 size feature map is obtained, and after passing through the third decoder, a W / 4×H / 4 size feature map is obtained. Step S1-2: The face image is passed through the Transformer branch network for feature extraction to obtain a feature image of size W / 16 × H / 16; The Transformer branch network consists of three Transformer blocks, each of which includes: patch merging, LayerNorm, SW-MSA, and MLP; Specifically, the input W×H image undergoes patch merging and LayerNorm operations, and then is input into the SW-MSA module to first divide the feature map into m×m blocks. Then, self-attention is calculated for each image block to obtain a W / 4×H / 4 feature map. The output before normalization is connected to the input after the SW-MSA module through a residual connection, and then normalized through the Layernorm module. The normalized result is then input into the MLP module for nonlinear transformation, and the normalized result is connected to the output of the nonlinear transformation through a residual connection, and finally a W / 4×H / 4 feature map is output. Similarly, after the second Transformer block, a W / 8×H / 8 feature map is output, and after the third Transformer block, a W / 16×H / 16 feature map is output. Step S1-3, perform dot multiplication on the feature map output by each encoder in the convolutional neural network and the feature map of the same scale in the feature map at three scales obtained by three Transformer blocks to obtain a fused feature map; Step S1-4, performing channel splicing on the feature map obtained after the primary fusion in step S1-3 and the feature of the same scale generated by the decoder in the convolutional neural network to obtain a secondary fusion feature map with unchanged scale and increased number of channels; Step S1-5: perform channel concatenation on the feature maps at three scales obtained by three transformer blocks and the features of the same scale generated by the decoder in the convolutional neural network to obtain a three-dimensional fused feature map with unchanged scale and doubled number of channels; Step S1-6: The obtained three-fold fusion feature map is subjected to feature fusion by a feature fusion module to obtain an initial landmark L1 of M groups of key points; Step S1-7: Input the feature map output in step S1-6 and the initial landmark L1 of the M groups of key points into the progressive refinement module, and obtain the corresponding N iterations to obtain the refined key points L2, L3...L N ; Step S2, design the loss function, the loss function includes L init 、 L refine ; Step S3: Use the training set of the face database to train the two-branch network and save the trained model; Step S4: Test the network model saved in step S3 and calculate the coordinates of the L groups of facial key points output by the model. n The normalized error value compared to the true value data.

2. The dual-branch face image key point detection method based on Transformer according to claim 1 is characterized in that: The step S1-6 is specifically as follows: The feature map of size W / 4×H / 4 is upsampled four times. These four upsampling operations are four deconvolution operations. After the first deconvolution of the feature map of size W / 4×H / 4, a feature map of scale W / 2×H / 2 is output. After four deconvolution operations, the feature map of size W / 4×H / 4 is finally expanded to a feature map of scale W×H. The feature map obtained by upsampling passes through three convolutional layers and one fully connected layer to generate the initial landmark L1 of M groups of key points.

3. The Transformer-based dual-branch face image key point detection method according to claim 2, characterized in that: In step S1-7, the progressive refinement module is composed of a Transformer encoder and a fully connected layer; the Transformer encoder includes: patch merging, LN, MSA or MCSA, MLP; The progressive refinement module performs N iterations in total, and the process is as follows: In the first iteration, the W / 2×H / 2 scale feature map output after the first deconvolution in step S1-6 is evenly divided into N×N windows, and a local block of size T×T is cropped on the window with the generated initial key point L1 as the center; first, the T×T dimension of the T×T size feature local block is flattened by the fatten operation to obtain a one-dimensional sequence feature; the absolute position of the local block vector is encoded from 1 to the total number of key points L; each position has a separate encoding vector; then the encoding vector is summed with the local block vector to output a feature sequence, the feature sequence is processed using layer normalization, and the processed features are then sent to M SA, the feature sequence without LN operation is residually connected with the output of MSA, the result of the residual connection is LN-operated and then sent to MCSA, the feature sequence without LN operation is residually connected with the output of MCSA, and the feature sequence is processed using LN, and then the processed features are sent to the MLP block, the feature sequence without LN operation is residually connected with the output of MLP, and finally the output result of the Transformer encoder layer is obtained; the result is input into the fully connected layer to output the relative position of the key point coordinates relative to the output key point of the previous iteration, thereby obtaining the refined coordinates of this iteration; In the second iteration, the number of windows is 2N×2N, and the local block size becomes T / 2×T / 2. Similarly, the number of windows doubles and the local block size decreases by two times. In each iteration, this local block is projected as a patch into a vector of fixed length, and the result is sent to a layer of Transformer encoder. The iterative process is repeated to finally obtain N groups of refined key points L2, L3...L N .

4. The dual-branch face image key point detection method based on Transformer according to claim 1, characterized in that: In the step 2, specifically: Step S2-1, calculate the normalized average error value using the initial key point position L1 and the true value data L init , as shown in formula (5): （5）； Among them, L i is the key point position, i =1,2…N, N is the number of key points; L g is the true value data, d is the distance between the pupils of the two eyes of the face; Step S2-2, using N iterations to obtain the refined key points L2, L3...L N Calculate the normalized average error value with the true value data respectively L refine , as shown in formula (6): （6） in, For different weight values; The sum of the weights is 1; Step S2-3, L init 、 L refine Add together to get , and according to Update the network parameters so that the network gradually converges.

5. The Transformer-based dual-branch facial image key point detection method according to claim 4, characterized in that: In the step S3, specifically: Step S3-1: Perform data augmentation on the original images taken from the training set by rotating each sample from −30 o Start with 10 o Rotate to 30 for interval o To increase the training data, in addition, each face sample is randomly blocked and noise is added, and the images in the dataset are input into the network for training; Step S3-2, select the network optimizer as ADAM optimizer, set the initial learning rate to 0.1, adjust the scheduler strategy to multi-step decay strategy, the decay steps are 25 and 100 respectively, decay half each time, and train a total of 200 Epochs. By observing L init 、L refine Two indicators dynamically adjust hyperparameters, such as learning rate lr and training round number Epoch; Step S3-3, use the SummaryWriter of the Python third-party library tensorboard to write the training process L init 、L refine The reference indicators are output to tensorboard, and the network parameters trained in step S3-2, the number of epochs being trained, the optimizer ADAM, and the scheduler are saved to obtain the trained network model.

Images