Human head pose estimation method based on classification followed by regression

Abstract

The present invention discloses a method for estimating human head posture based on classification and regression, comprising the following steps: step 1, processing a data set, constructing a multi-classification neural network based on GhostNet, and training to obtain a multi-classification network model based on GhostNet; step 2, constructing a neural network model based on classification and regression; step 3, training a neural network based on classification and regression and a classification neural network based on GhostNet, and fusing the two network models; step 4, using a neural network based on classification and regression to estimate head posture. The method of the present invention proposes a task splitting mechanism, which splits the three angle values ​​predicted by head posture estimation into three subtasks, respectively predicting the pitch angle, yaw angle, and roll angle, thereby improving the accuracy of the prediction results and greatly reducing the amount of calculation, and is suitable for AI automatic evaluation tasks such as student learning or experimental operation.

Description

Technical Field

[0001] The present invention relates to a head posture estimation method, in particular to a head posture estimation method based on classification followed by regression. Background Art

[0002] In recent years, with the rapid development of deep learning, deep learning has played a huge role in various fields, and has also played a relatively good effect in head pose estimation. At present, compared with traditional methods, deep learning methods for head pose estimation are not only more accurate, but also more generalizable.

[0003] Currently, the main directions based on deep learning methods are as follows: head pose estimation method based on a single picture; head pose estimation method based on facial key points; head pose estimation method based on multimodal information; head pose estimation method based on multi-task.

[0004] The head posture estimation method based on a single picture has low hardware cost, does not require special camera equipment, and only needs a camera function. In addition, the model complexity is relatively low. However, this method also has obvious disadvantages, that is, the accuracy is significantly lower than other methods, especially compared with methods based on multimodal information.

[0005] The prior art application publication number is: CN111339941 A, which discloses a head posture detection method, including putting an image into a neural network for classification, and using a deep learning model to use a composite loss function to calculate losses for three angles respectively. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a head posture estimation method based on classification followed by regression, which includes three parts: classification followed by regression, task splitting mechanism, and angle calculation method. It can effectively improve the accuracy of head posture estimation of a single image.

[0007] In order to achieve the above object, the present invention is achieved through the above technical solutions:

[0008] The present invention is a method for estimating a human head posture based on classification followed by regression, comprising the following steps:

[0009] Step 1: Process the data set and build a multi-classification neural network based on GhostNet;

[0010] Step 2, construct a neural network based on classification followed by regression;

[0011] Step 3: train a neural network based on classification followed by regression and a classification neural network based on GhostNet, and integrate the two trained network models;

[0012] Step 4: Use the network model obtained in step 3 to estimate the head posture.

[0013] The further improvement of the present invention is that the specific steps of step 2 are as follows:

[0014] Step 2.1, input the RGB image with the face into the feature extraction network to obtain the feature matrix M1 of the RGB image;

[0015] Step 2.2: Input the feature matrix M1 into the feature enhancement network. The feature enhancement network contains two convolutional layers. After inputting into the first convolutional layer, the feature matrix M is obtained. 11 , and then input into the second convolutional layer to obtain the feature enhancement matrix M2;

[0016] Step 2.3, input the feature enhancement matrix M2 into the attention mechanism module to obtain the yaw attention enhancement feature matrix M y , pitch attention enhancement feature matrix M p and roll attention enhancement feature matrix M r ;

[0017] Step 2.4, the yaw attention enhanced feature matrix M y , pitch attention enhancement feature matrix M p , roll attention enhancement feature matrix M r Input into the regression network module to obtain the yaw feature vector M ypf , pitch feature vector M ppf , roll feature vector M rpf ;

[0018] Step 2.5, through the yaw feature vector M ypf Calculate the yaw angle value through the pitch eigenvector M ppf Calculate the pitch angle value through the roll feature vector M rpf Calculate the roll angle value.

[0019] A further improvement of the present invention is that the attention mechanism module includes three attention branches.

[0020] A further improvement of the present invention is that the regression network module includes three regression branches, and these three branches are respectively connected in series behind the three attention branches.

[0021] A further improvement of the present invention is that in step 2.5, the yaw feature vector M ypf , pitch feature vector M ppf , roll feature vector Mrpf The size of each eigenvector is 1×132, and the first 66 elements of each eigenvector are denoted as p i , the last 66 elements are recorded as q i , the p of each vector i and q i Substitute the formulas to calculate the yaw angle value, pitch angle value, and roll angle value. The formulas are as follows:

[0022]

[0023] Among them, s represents the number of subintervals, p i The probability of the i-th category, μ i represents the value of the i-th category, q i For correction items.

[0024] A further improvement of the present invention is that the specific steps of step 3 are as follows:

[0025] Step 3.1, extracting parameters of the feature extraction network in the multi-classification neural network model based on GhostNet, extracting parameters of the feature extraction network in the neural network model based on classification followed by regression;

[0026] Step 3.2, multiply the parameters of the feature extraction network in the multi-classification neural network model based on GhostNet by a weight of 0.5, and multiply the parameters of the feature extraction network in the classification-then-regression neural network model by a weight of 2;

[0027] In step 3.3, the parameters of the feature extraction network in the GhostNet multi-classification neural network model and the parameters of the feature extraction network in the classification-then-regression neural network model processed in step 3.2 are added one by one and divided by 2.5 to obtain the parameters of the feature extraction network in the final classification-then-regression neural network model.

[0028] The beneficial effects of the present invention are: 1. The present invention adopts a classification-then-regression method in the feature extraction stage of the neural network model to realize human head posture estimation, which can be applied to AI automatic evaluation tasks such as student learning or experimental operations.

[0029] 2. The method of the present invention can be used to split the task, that is, the task of estimating and predicting three angle values ​​of head posture can be split into three subtasks and predict three angles respectively, which not only improves the accuracy of head posture detection, but also reduces the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall flow chart of the present invention.

[0031] Figure 2 It is a specific operation schematic diagram of step 2 of the present invention.

[0032] Figure 3 This is a multi-classification neural network structure diagram based on GhostNet in the present invention.

[0033] Figure 4 It is a structural diagram of a neural network based on classification followed by regression in the present invention. DETAILED DESCRIPTION

[0034] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are only part of the implementation of the present invention, not all of the implementations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1-4 As shown, the present invention is a head posture estimation method based on classification followed by regression, comprising the following steps:

[0036] Step 1, process the data set. First, use the mtcnn face detection framework to crop out the pictures with the face part, then remove the pictures with angles outside the range of [-99,99]. Divide the pitch angle (pitch), yaw angle (yaw), and roll angle (roll) from -99° to 99° into 66 intervals on average. Each interval is regarded as a category. Construct a multi-classification neural network based on GhostNet. The network input is a single RGB picture containing a human head. The network output is the category to which the three angle values ​​(yaw, pitch, roll) belong. Then train the network to obtain a multi-classification network model based on GhostNet.

[0037] Step 2: Construct a neural network model based on classification followed by regression. The neural network based on classification followed by regression includes: feature extraction network, feature enhancement network, attention mechanism module and regression network module. And train the network to obtain a neural network model based on classification followed by regression. The specific steps are as follows:

[0038] Step 2.1: Input the RGB image with a size of 224×224×3 containing a face into the feature extraction network to obtain a 7×7×960-dimensional feature matrix M1 of the image.

[0039] Step 2.2, input the feature matrix M1 into the feature enhancement network. The feature enhancement network contains two convolutional layers. The first convolutional layer consists of 160 convolutional kernels and the second convolutional layer consists of 960 convolutional kernels. After inputting into the first convolutional layer, a feature matrix M with a size of 7×7×160 is obtained.11 , and after being input into the second convolutional layer, a feature enhancement matrix M2 with a size of 7×7×960 is obtained.

[0040] Step 2.3, input the feature enhancement matrix M2 into the attention mechanism module. The attention mechanism module contains three attention branches, each of which is a separate attention mechanism network, denoted as the first attention branch, the second attention branch, and the third attention branch. First, input M2 into the attention mechanism network in branch 1 to obtain the attention enhancement feature matrix M with a size of 7×7×960 representing the yaw angle. y ; Secondly, M2 is input into the attention mechanism network in branch 2 to obtain the attention-enhanced feature matrix M with a size of 7×7×960 representing the pitch angle p ; Finally, M2 is input into the attention mechanism network in branch 3 to obtain the attention-enhanced feature matrix M with a size of 7×7×960 representing the roll angle r .

[0041] Step 2.4, the yaw attention enhanced feature matrix M y , pitch attention enhancement feature matrix M p , roll attention enhanced feature matrix M r Input to the regression network module, the regression network module contains three regression branches, denoted as the first regression branch, the second regression branch and the third regression branch, and these three branches are connected in series behind the first attention branch, the second attention branch and the third attention branch respectively. For the first regression branch, first transform the feature matrix M y The pooling layer input to the first regression branch obtains the yaw pooling feature vector M with a size of 1×960 yp , and then the eigenvector M yp Input to the fully connected layer to obtain the yaw feature vector M of size 1×132 ypf ; For the second regression branch, first transform the feature matrix M p The pooling layer input to the second regression branch obtains the pitch pooling feature vector M with a size of 1×960 pp , and then the eigenvector M pp Input to the fully connected layer to obtain the pitch feature vector M of size 1×132 ppf ; For the third regression branch, first transform the feature matrix M r The pooling layer input to the third regression branch obtains the roll pooling feature vector M with a size of 1×960 rp , and then the eigenvector M rp Input to the fully connected layer to obtain the roll feature vector M of size 1×132 rpf .

[0042] Step 2.5, transform the yaw feature vector M with size 1×132 ypf The first 66 elements of i , the last 66 elements are recorded as q i , calculate the yaw angle value using formula (1);

[0043] The pitch feature vector M with size 1×132 ppf The first 66 elements of i , the last 66 elements are recorded as q i The pitch angle value is calculated using formula (1);

[0044] Roll the feature vector M of size 1×132 rpf The first 66 elements of i , the last 66 elements are recorded as q i The roll angle value is calculated by formula (1); formula (1) is:

[0045]

[0046] Among them, s represents the number of subintervals, p i The probability of the i-th category, μ i represents the value of the i-th category, q i For correction items.

[0047] Step 3: extract the parameters of the feature extraction network in the GhostNet-based multi-classification network model trained in step 1, and weightedly fuse them with the parameters of the feature extraction network in the neural network based on classification and regression trained in step 2; the specific steps are as follows:

[0048] Step 3.1, extract the parameters of the feature extraction network in the multi-classification neural network model based on GhostNet, and extract the parameters of the feature extraction network in the neural network model based on classification followed by regression;

[0049] Step 3.2, multiply the parameters of the feature extraction network in the classification neural network model based on GhostNet by a weight of 0.5, and multiply the parameters of the feature extraction network in the neural network model based on classification followed by regression by a weight of 2;

[0050] Step 3.3, add the parameters of the feature extraction network in the classification neural network model based on GhostNet and the parameters of the feature extraction network in the neural network model based on classification and regression processed in step 3.2 one by one and divide by 2.5 to obtain the final parameters of the feature extraction network in the neural network model based on classification and regression. As shown in the following formula (2):

[0051]

[0052] Among them, ω final represents the network parameters of the fused feature extraction network based on the classification and regression neural network model, ω c represents the parameters of the feature extraction network in the multi-classification neural network model based on GhostNet, ω r Represents the parameters of the feature extraction network in the neural network model based on classification followed by regression.

[0053] Step 4: Use the neural network model based on classification followed by regression after fusion in step 3 to estimate the head posture. This step first inputs a picture of size 224×224×3, and then through forward propagation, the head posture in the picture, namely yaw, pitch, and roll, can be obtained end-to-end.

[0054] The present invention implements the idea of ​​classification followed by regression in the feature extraction stage. First, a multi-classification neural network model based on GhostNet is used to carry out the classification training task of head posture; secondly, a neural network model based on classification followed by regression is trained; finally, the feature extraction networks in the above two models are integrated to obtain a network model based on classification followed by regression in the test stage. The head posture estimation calculation method of the present invention is more stable, has high accuracy, has low hardware requirements, and can effectively reduce costs.

[0055] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A method for estimating human head posture based on classification followed by regression, characterized in that: The steps include: Step 1: Process the data set and build a multi-classification neural network based on GhostNet; Step 2, construct a neural network based on classification followed by regression; Step 2.1, input the RGB image with the face into the feature extraction network to obtain the feature matrix M1 of the RGB image; Step 2.2, input the feature matrix M1 into the feature enhancement network. The feature enhancement network contains two convolutional layers. After inputting into the first convolutional layer, the feature matrix M11 is obtained. After inputting into the second convolutional layer, the feature enhancement matrix M2 is obtained. Step 2.3, input the feature enhancement matrix M2 into the attention mechanism module to obtain the yaw attention enhancement feature matrix My, the pitch attention enhancement feature matrix Mp and the roll attention enhancement feature matrix Mr; Step 2.4, input the yaw attention enhancement feature matrix My, the pitch attention enhancement feature matrix Mp, and the roll attention enhancement feature matrix Mr into the regression network module to obtain the yaw feature vector Mypf, the pitch feature vector Mppf, and the roll feature vector Mrpf; Step 2.5, calculate the yaw angle value through the yaw eigenvector Mypf, calculate the pitch angle value through the pitch eigenvector Mppf, and calculate the roll angle value through the roll eigenvector Mrpf; Step 3: Train a neural network based on classification followed by regression and a classification neural network based on GhostNet, and integrate the two trained network models; Step 3.1, extracting the parameters of the feature extraction network in the multi-classification neural network model based on GhostNet, and extracting the parameters of the feature extraction network in the neural network model based on classification followed by regression; Step 3.2, multiply the parameters of the feature extraction network in the multi-classification neural network model based on GhostNet by a weight of 0.5, and multiply the parameters of the feature extraction network in the classification-then-regression neural network model by a weight of 2; Step 3.3, add the parameters of the feature extraction network in the GhostNet multi-classification neural network model and the parameters of the feature extraction network in the classification-then-regression neural network model processed in step 3.2 one by one and divide by 2.5 to obtain the parameters of the feature extraction network in the final classification-then-regression neural network model; Step 4: Use the network model obtained in step 3 to estimate the head posture.

2. The method for estimating a person's head posture based on classification followed by regression according to claim 1, characterized in that: The attention mechanism module includes three attention branches.

3. The method for estimating a person's head posture based on classification followed by regression according to claim 1, characterized in that: The regression network module includes three regression branches, and the three branches are respectively connected in series behind the three attention branches.

4. The method for estimating a person's head posture based on classification followed by regression according to claim 1, characterized in that: In step 2.5, the size of the yaw feature vector Mypf, the pitch feature vector Mppf, and the roll feature vector Mrpf are all 1×132, and the first 66 elements of each feature vector are denoted as p i , the last 66 elements are recorded as q i , the p of each vector i and q i Substitute the formulas to calculate the yaw angle, pitch angle, and roll angle values. The formulas are as follows: Among them, s represents the number of subintervals, p i The probability of the i-th category, μ i represents the value of the i-th category, q i For correction items.

Images