A new lightweight head pose estimation method

Abstract

In view of the problems of poor real-time performance and low recognition rate of existing head pose estimation algorithms in complex scenes, the application provides a new lightweight head pose estimation method. The method has a multi-level output structure, uses three different types of branch networks to extract features from the input image, and each branch has three stages, and each stage only needs to refine the features of the previous stage. The features extracted by each branch at the same stage are fused by a feature fusion module to generate a feature map, effectively avoiding the problem of feature loss. The feature extraction module selects the Ghost module as the feature extraction network, uses model compression to reduce the network parameters and computational complexity while ensuring the accuracy of the network; in order to extract more important effective features, an efficient channel attention module ECA-Net is introduced, thereby improving the accuracy of head pose estimation. The method of the application can achieve excellent head pose estimation efficiency and has wide applicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, pattern recognition, artificial intelligence, and particularly relates to a new lightweight head pose estimation method. BACKGROUND

[0002] Head pose is an important factor in judging human behavior, and is also very helpful for understanding human behavior. Head pose estimation refers to the computer analyzing and predicting the input image or video sequence to determine the position and pose parameters of the human head in three-dimensional space. The pose parameters here refer to the rotation angles of the face image in three-dimensional space, which are yaw, pitch, and roll. In the past two decades, the related research on head pose has promoted the development of three-dimensional reconstruction visual technology and multimedia content operation, and has a wide range of applications in human-computer interaction, biometric identification, virtual reality, and fatigue detection.

[0003] Head pose estimation as one of the key technologies of computer vision (CV) has been widely concerned and researched, and many research methods have emerged in its development process. According to whether key point detection is needed, head pose estimation can be mainly divided into two categories: one is a method based on traditional face key point detection, and the other is a method of key point-free head pose estimation based on deep neural network (DNN). The method based on face key point detection mainly adopts the way of face detection and face key point marking to represent the face, and then solves the head pose through the relative relationship between two-dimensional space and three-dimensional space. However, this method must be under the premise that face detection and feature point marking reach sufficient accuracy, and the model complexity is large. And in actual application, the face feature point detection accuracy will be affected by factors such as light change, complex background, head deflection and occlusion, resulting in a significant decrease in detection accuracy, and even the face feature points cannot be detected. Therefore, the head pose estimation algorithm based on the model is not very accurate. The key point-free head pose estimation method based on deep neural network is to train different pose samples, describe the pose samples to be processed by vectors, obtain a set of pose and feature description mapping, and convert the head pose recognition problem into a classification problem, so as to obtain the corresponding relationship between face pose and head pose, which does not depend on the accurate positioning of face feature points and can predict the head pose in a large deflection range. However, most of the current algorithms use depthwise separable convolution (DSC) model, which will occupy a large amount of memory and floating point operation, and a part of the features processed by DSC is redundant, and the correlation and redundancy between feature mappings are not well utilized. SUMMARY

[0004] The purpose of the present application is to solve the problem of insufficient feature extraction, difficulty in balancing precision and operation amount in the current head pose estimation method, and to quickly and accurately realize head pose estimation by the method.

[0005] In order to achieve the above purpose, the present application provides a new lightweight head pose estimation method, which mainly includes five parts: the first part is to preprocess the data set; the second part is to design a multi-stage three-branch feature extraction network; the third part is to fuse the extracted features; the fourth part is to map the fused features through a fine-grained structure to obtain representative features, and then input them into SSR-Net to obtain the prediction result of head pose; the fifth part is network training and testing, and the final head pose estimation result is output.

[0006] The first part includes two steps:

[0007] Step 1, download the commonly used dataset for head pose estimation: 300W-LP, AFLW2000 and BIWI dataset, and then set these datasets to two protocols for better training and testing. The first protocol is to train on the synthetic dataset 300W-LP and test on two real datasets AFLW2000 and BIWI; the second protocol is to train using 16 videos on the BIWI dataset and test the remaining 8 videos;

[0008] Step 2, use Multi-Task Convolutional Neural Network (MTCNN) for face detection, select the picture containing the face, frame the face and crop it into a 64x64 pixel face picture;

[0009] The second part includes two steps:

[0010] Step 3, input the cropped picture in step 2 into the multi-stage three-branch feature extraction network, and perform multi-size feature extraction on the input image, the specific implementation is as follows:

[0011] (1) Introduce Ghost module in the improved three-branch feature extraction network, use model compression to reduce network parameters and computational complexity while ensuring network accuracy, thereby improving the calculation speed;

[0012] (2) Input the module processed in (1) to the batch normalization and activation function module for processing, to improve the network convergence speed and prevent overfitting;

[0013] (3) Input the features processed in (2) to ECA-Net to improve the effectiveness of feature information and obtain robust features;

[0014] (4) Input the robust features obtained in (3) to the pooling layer for cropping, and use the cropped fine image as the input of the next stage network;

[0015] Step 4, repeat the operation of step 3 three times to form a three-stage feature extraction network, and only refine the features of the previous stage in each stage, the channel size of the three stages is set to 32, 64 and 128 respectively, to gradually refine the input features and preserve complete feature information;

[0016] The third part includes two steps:

[0017] Step 5, fuse the different features extracted in step 4 to obtain a new feature map, to enable the network to learn more features;

[0018] Step 6, the fused feature maps in step 5 are respectively input into a convolution kernel with a size of 1x1 for convolution operation, and the feature maps are converted to c After the channel, the pooling operation is performed, and the size of the feature map becomes w × h , w and h represent the width and height of the feature map respectively, so that the feature maps with a size of w × h × c at each stage can be obtained U K ;

[0019] The fourth part includes four steps:

[0020] Step 7, the feature maps obtained in step 6 are input into the attention structure A K to calculate the importance of each feature, thereby improving the network's ability to extract effective features;

[0021] Step 8, the features processed in steps 6 and 7 are jointly input into the fine-grained structure mapping module, and more effective and representative features are extracted through the fine-grained structure mapping module , and each is spliced into a representative feature ;

[0022] Step 9, the representative feature obtained in step 8 is sent to the feature aggregation module to obtain the final feature set V , so as to better group the feature mapping;

[0023] Step 10, the feature set V obtained in step 9 is input into the soft stage regression network (SSR-Net), thereby obtaining the final head pose estimation result;

[0024] The fifth part includes two steps:

[0025] Step 11, the network structure hyperparameters from step 3 to step 10 are debugged, wherein the initial learning rate is set to 0.01, there are a total of 100 cycles, the learning rate is reduced by 10 times every 30 cycles, and the final training model is obtained;

[0026] Step 12, the two training sets set in step 1 are respectively input into the training model in step 11, and the head pose estimation result is obtained.

[0027] This invention presents a novel lightweight head pose estimation algorithm. First, three feature extraction networks are designed, employing different activation functions and pooling methods to extract different types of features from the input face image, addressing the problem of insufficient feature extraction in head pose estimation networks. Each branch also includes three stages, where feature maps extracted from the same stage across the three branches are fused through dot product, preventing feature loss. Second, to avoid excessive redundant features, a Ghost module and ECA-Net are introduced into the feature extraction network, reducing network parameters and computational cost while maintaining network accuracy, thus improving computational speed. Then, the features extracted from each stage are input into a feature fusion module, where different features are fused through dot product to obtain new feature maps, enabling the network to learn more features. Finally, the fused features are passed through a fine-grained mapping module to obtain more representative features, which are then input into SSR-Net to output the predicted parameters from each stage, thus obtaining the final head pose estimation result. Attached Figure Description

[0028] Figure 1 This is a diagram of the overall network framework of the present invention;

[0029] Figure 2 This is a diagram of the feature extraction network framework of the present invention;

[0030] Figure 3 This is a network framework diagram of the Ghost module of the present invention;

[0031] Figure 4 This is a diagram of the ECA-Net network framework of the present invention;

[0032] Figure 5 This is a diagram of the feature fusion network framework of the present invention;

[0033] Figure 6 This is a diagram of the feature mapping network framework of the present invention. Detailed Implementation

[0034] To better understand this invention, a novel lightweight head pose estimation algorithm of this invention will be described in more detail below with reference to specific embodiments. In the following description, detailed descriptions of existing technologies may obscure the subject matter of this invention, and such descriptions will be omitted here.

[0035] Figure 1 This is a specific network model diagram of a novel lightweight head pose estimation algorithm of the present invention. In this embodiment, it is carried out according to the following steps:

[0036] Step 1, download the commonly used dataset for head pose estimation: 300W-LP, AFLW2000 and BIWI dataset, and then set these datasets to two protocols for better training and testing, the first is to train on the synthetic dataset 300W-LP and test on two real datasets AFLW2000 and BIWI, the second is to train on 16 videos on the BIWI dataset and test on the other 8 videos;

[0037] Step 2, determine whether the picture in the dataset contains a human face, use a multi-task convolutional neural network for face detection, delete pictures that do not contain a human face, select pictures that contain a human face, and frame and crop the human face to a size of 64x64 pixels;

[0038] Step 3, input the cropped picture in step 2 into the designed multi-stage three-branch feature extraction network, as shown in Figure 2 , the basic building block is composed of a Ghost module, batch normalization (BN), a nonlinear activation function, an ECA-Net, and a 2x2 pooling layer. The network structure and stage settings of each branch are the same, which are established based on the following formula:

[0039]

[0040] wherein, Ghost (3x3, c ) represents Ghost the Ghost module, 3x3 indicates a convolution kernel with a size of 3x3, the channel size is c , and the activation function is ReLU, Tanh, and Sigmoid respectively.

[0041] However, the internal parameters of each branch are different, as shown in Table 1.

[0042]

[0043] The network adopts a coarse-to-fine classification strategy, with a channel size of 32 in the first stage, 64 in the second stage, and 128 in the third stage. The purpose of this is to gradually extract fine features from the input features by continuously expanding the channel size, and the specific implementation is as follows:

[0044] (1) Introduce the Ghost module into the three-branch feature extraction network, as shown in Figure 3 , use model compression to reduce network parameters and computational complexity while ensuring network accuracy, thereby improving computational speed.

[0045] (2) The processed convolutional layer input in (1) is processed by a batch normalization and activation function module to improve network convergence speed and prevent overfitting. Stream1 adopts a ReLU activation function and an Average Pooling pooling method; Stream2 adopts a Tanh activation function and a MaxPooling pooling method; and Stream3 adopts a Sigmoid activation function and an Average Pooling pooling method. The purpose of such setting is to extract image information through different activation functions and pooling methods, and then fuse them to improve the performance of feature extraction;

[0046] (3) The processed features in (2) are input to an ECA-Net, as shown in Figure 4 , which can improve the effectiveness of feature information and obtain robust features.

[0047] (4) The robust features obtained after processing in (3) are input to a pooling layer for cropping, and the cropped fine image is used as the input of the next stage network.

[0048] Step 4. The operation of step 3 is repeated three times to form a three-stage feature extraction network, and each stage only refines the features of the previous stage. The channel sizes of the three stages are set to 32, 64 and 128 respectively to gradually refine the extraction of input features;

[0049] Step 5. Different features extracted in step 4 are fused to obtain a new feature map to enable the network to learn more features.

[0050] Step 6. As shown in Figure 5 , the feature map fused in step 5 is input into a convolution kernel with a size of 1x1 for convolution operation, and the feature map is converted to c channels for pooling operation, changing the size of the feature map to w × h , w and h represent the width and height of the feature map, respectively, so that the feature map with a size of w × h × c U K ;

[0051] Step 7. The feature map obtained in step 6 is input into an attention structure A K to calculate the importance of each feature, thereby improving the network's ability to extract effective features. Among them:

[0052] ​

[0053] The variance is chosen as the scoring function to measure the importance of each feature. Among them , u is the given pixel-level feature u = ( u 1, u 2, …, u c ), is the average value of all pixel-level features.

[0054] Step 8, as shown in Figure 6 , the features processed in steps 5 and 6 are jointly input into the fine-grained structure mapping module to extract more effective and representative features , and each patch is . The specific implementation is as follows:

[0055] (1) Flatten U K into a two-dimensional matrix U , where , w and h represent the width and height of the feature map respectively, K represents the K th stage. U contains all the pixel-level features in all stage feature maps.

[0056] (2) For the K th stage, find a mapping S K to extract S K representative features from , where , . That is, through the linear combination of n pixel-level features, we get representative features. The mapping S K is the product of two learnable mappings C and M K , C and M K is in the form as follows:

[0057]

[0058] where , is a sigmoid function, f M and f C are two fully connected layers with different functions, is the concatenation of all attention maps. The maps M K are specific to the i-th stage, while the maps K are shared by all stages. C

[0059] (3) concatenate all to form the last set of representative features where, .

[0060] Step 9, the representative features from step 7 are sent to the feature aggregation module to obtain the final feature set V to better group the feature maps;

[0061] Step 10, the feature set V from step 9 is input into the Soft Stagewise Regression Network (SSR-Net) to obtain the final head pose estimation result;

[0062] Step 11, the network structure hyperparameters from step 3 to step 10 are adjusted, wherein the initial learning rate is set to 0.01, there are a total of 100 cycles, the learning rate is reduced by 10 times every 30 cycles, and the final trained model is obtained;

[0063] Step 12, the two training sets set in step 1 are respectively input into the trained model in step 11 to obtain the head pose estimation result.

[0064] Although the above describes the specific embodiments of the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.​

Claims

1. A new lightweight head pose estimation method characterized by Among the five parts, the dataset preprocessing, three different types of feature extraction network are carried out in three stages respectively to extract multi-scale features, the features extracted in each stage are fused, and the fused features are mapped to fine-grained structure to obtain representative features, network training and testing. The first part includes two steps: Step 1, download the commonly used dataset for head pose estimation: 300W-LP, AFLW2000 and BIWI dataset, and then set these datasets to two protocols for better training and testing, the first is to train on the synthetic dataset 300W-LP and test on two real datasets AFLW2000 and BIWI, the second is to train on 16 videos on BIWI dataset and test on the other 8 videos; Step 2, use Multi-Task Convolutional Neural Network (MTCNN) for face detection, select the picture containing the face, and crop the face after framing, and crop it into a 64x64 pixel face picture; The second part includes two steps: Step 3, input the cropped picture in step 2 into the designed three-stream feature extraction network, and extract sufficient features from the input image, the specific implementation is as follows: (1) Introduce Ghost module in the three-stream feature extraction network, use model compression to reduce network parameters and calculation amount while ensuring network accuracy, so as to improve the calculation speed; (2) The convolutional layer processed in (1) is input to the batch normalization and activation function module for processing to improve the network convergence speed and prevent overfitting; (3) The features processed in (2) are input to ECA-Net to improve the effectiveness of feature information and obtain robust features; (4) The robust features obtained in (3) are input to the pooling layer for cropping, and the cropped fine image is used as the input of the next stage network; Step 4, repeat the operation of step 3 three times to form a three-stage feature extraction network, and only refine the features of the previous stage in each stage, the channel size of the three stages is set to 32, 64 and 128 respectively, and the input features are gradually refined and extracted; The third part includes two steps: Step 5, fuse the different features extracted in step 4 to obtain a new feature map to enable the network to learn more features; Step 6, the fused feature maps in step 5 are respectively input into a convolution kernel with a size of 1x1 for convolution operation, and the feature maps are converted to c channels and then subjected to pooling operation, so that the size of the feature maps becomes w x h, where w and h represent the width and height of the feature maps respectively, and thus the feature maps with a size of w x h x c at each stage U can be obtained K where c represents the channel size; The fourth part includes four steps: Step 7, input the feature map obtained in step 6 into an attention structure A K to calculate the importance of each feature, thereby improving the extraction ability of the network for effective features; Step 8, the features processed in step 6, 7 are input to the fine-grained structure mapping module, and more effective and more representative features are extracted through the fine-grained structure mapping module and each Spliced into representative features Step 9, representative features from step 8 are selected are fed into a feature aggregation module to obtain the final feature set V for better grouping of feature maps; Step 10, input the feature set V obtained in step 9 into the Soft Stagewise Regression Network (SSR-Net) to obtain the final head pose estimation result; The fifth part includes two steps: Step 11, debug the network structure hyperparameters from step 3 to step 10, where the initial learning rate is set to 0.01, there are 100 cycles in total, the learning rate is reduced by 10 times every 30 cycles, and the final training model is obtained; Step 12, input the two training sets set in step 1 into the training model in step 11 respectively, and obtain the head pose estimation result.

Images