A head pose estimation method and system

Abstract

The application provides a head posture estimation method and system, comprising: determining an image containing a face; inputting the image into a pre-trained hierarchical prediction network to predict a pitch angle, a yaw angle and a roll angle of a face posture orientation; comprising: a backbone network, a feature pyramid network, a dimension reduction module and a hierarchical prediction module; the backbone network is used for extracting image spatial features of different sizes, the feature pyramid network is used for fusing the image spatial features of different sizes to obtain fused features, the dimension reduction module is used for performing dimension reduction on the fused features in three different dimensions to obtain image spatial features in three dimensions; the hierarchical prediction module comprises: three fully connected layers; the three fully connected layers respectively predict the image spatial features in three dimensions, and each fully connected layer predicts one angle of the face posture orientation, so that the hierarchical prediction network predicts three angles of the face posture orientation, each of which focuses on different image regions, and reduces mutual interference between the three angle predictions.

Description

Technical Field

[0001] This invention belongs to the field of head pose detection, and more specifically, relates to a head pose estimation method and system. Background Technology

[0002] Head pose detection technology has a wide range of applications, such as fatigue detection and autonomous driving. On the one hand, although depth images can achieve very good results in head pose estimation, for RGB images, head pose estimation methods still suffer from discontinuous angle prediction, which limits the application of head pose estimation.

[0003] In real-world environments, people often have extensive facial occlusion and large head turns; the background lighting also varies greatly. Traditional machine learning methods struggle to detect heads in these situations and lack robustness to people of different identities, making them incapable of performing normal head pose estimation tasks.

[0004] Compared to general machine learning methods, deep learning methods have better performance in the image domain and are more suitable for head pose estimation in real-world scenarios. Specifically, they are: (1) robust to people of different identities; (2) insensitive to changes in the background of the person; and (3) capable of performing head pose estimation from a single image, making real-time detection possible.

[0005] Meanwhile, deep learning methods also have certain shortcomings in head pose estimation tasks: (1) In existing head pose estimation methods, the parameter adjustments of the three angles interfere with each other, and it is difficult to balance the prediction effect of the model; (2) The simple addition of the cross-entropy loss function and the mean squared error loss function leads to discontinuity in angle prediction. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a head pose estimation method and system, which aims to solve the problems of mutual interference between the parameter adjustments of the three angles in the existing head pose estimation and the discontinuity of angle prediction.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a head pose estimation method, comprising the following steps:

[0008] Identify images containing human faces;

[0009] The image is input into a pre-trained hierarchical prediction network to predict the pitch, yaw, and roll angles of the face pose, thereby estimating the face head pose. The hierarchical prediction network includes a backbone network, a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module. The backbone network extracts spatial features of different sizes from the image. The feature pyramid network fuses these spatial features to obtain fused features. The dimensionality reduction module performs dimensionality reduction on the fused features in three different dimensions, resulting in three spatial features of the image, each corresponding to a different number of image channels. The hierarchical prediction module includes three fully connected layers. Each fully connected layer predicts one angle of the face pose, ensuring that the hierarchical prediction network focuses on different image regions for each of the three angle predictions, reducing interference between the three angle predictions. The dimensions are in pixels.

[0010] In an optional example, the loss function during the training process of the hierarchical prediction network employs a self-adjusting loss constraint coefficient. This coefficient corrects for the loss magnitude inversion problem caused by the cross-entropy loss term of the prediction angle being greater than the mean squared error loss term when the average absolute error of the prediction angles of the three fully connected layers is less than a threshold. Furthermore, when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold, the error penalty caused by the aforementioned two loss terms is increased, allowing the hierarchical prediction network to converge faster during training. Let the error penalty of the hierarchical prediction network when the average absolute error of the prediction angles of the three fully connected layers is less than the threshold be the first penalty, and let the error penalty of the hierarchical prediction network when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold be the second penalty. Correcting the loss magnitude inversion problem means controlling the first penalty to be less than the second penalty, ensuring that the hierarchical prediction network can train and learn normally.

[0011] In an optional example, the backbone network includes four residual blocks; the face image is processed sequentially through the four residual blocks to obtain four image spatial features of decreasing size.

[0012] The aforementioned four types of image spatial features with decreasing dimensions are fused by the feature pyramid network. The fusion strategy is as follows: first, the first-size spatial feature is fused with the second-size spatial feature; then, the first-size spatial feature is fused with the third-size spatial feature to obtain new second-size spatial features and new third-size spatial features, respectively. Next, the new second-size spatial feature is fused with the new third-size spatial feature; then, the new second-size spatial feature is fused with the fourth-size spatial feature to obtain an updated third-size spatial feature and a new fourth-size spatial feature, respectively. Finally, the updated third-size spatial feature and the new fourth-size spatial feature are fused to obtain the fused feature of the fourth size. The size decreases progressively from the first size to the fourth size.

[0013] The dimensionality reduction module includes three convolutional kernels; the fourth type of fusion feature is processed by the three convolutional kernels in sequence. Each convolutional kernel performs dimensionality reduction on the input image features once, and successively obtains three spatial features with unchanged size and progressively decreasing number of channels.

[0014] In an optional example, the hierarchical prediction network adjusts the three angles of the prediction using the following formula:

[0015]

[0016] in, as well as Γ1, Γ2, and Γ3 represent the predicted values ​​of pitch angle, yaw angle, and roll angle, respectively; K1, K2, and K3 are the weight factors of the three convolutional kernels, respectively; Γ1, Γ2, and Γ3 are the three spatial features of the fused features obtained by the three convolutional kernels of the dimensionality reduction module.

[0017] The relationship between Γ1, Γ2, and Γ3 satisfies the following formula:

[0018] Where W1 is the feedback parameter from the first convolutional kernel to the second convolutional kernel of the dimensionality reduction module, W2 is the feedback parameter from the second convolutional kernel to the third convolutional kernel of the dimensionality reduction module, b4 is the new bias term brought about by dimensionality reduction from the first convolutional kernel to the second convolutional kernel, and b5 is the new bias term brought about by dimensionality reduction from the second convolutional kernel to the third convolutional kernel.

[0019] In an optional example, the loss function of the hierarchical prediction network for:

[0020]

[0021]

[0022] in, This is the value predicted by the hierarchical prediction network for head pose, where y is the true value of the face's head pose in the image, β is the loss constraint coefficient, which is constructed by the relationship between the mean squared error loss and the cross-entropy loss, k is the number of angle categories, σ is the sigmoid function, and L... mse Y represents the mean squared error loss. ic This represents the one-hot encoding formed according to the angle category. It is the category to which the prediction angle belongs.

[0023] In a second aspect, the present invention provides a head pose estimation system, comprising:

[0024] A face image determination unit is used to determine images containing human faces;

[0025] A head pose estimation unit is used to input the image into a pre-trained hierarchical prediction network to predict the pitch, yaw, and roll angles of the face pose orientation, thereby estimating the face head pose. The hierarchical prediction network includes a backbone network, a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module. The backbone network is used to extract image spatial features of different sizes. The feature pyramid network is used to fuse image spatial features of different sizes to obtain fused features. The dimensionality reduction module is used to perform dimensionality reduction on the fused features in three different dimensions to obtain three-dimensional spatial features of the image, with different dimensions corresponding to different numbers of image channels. The hierarchical prediction module includes three fully connected layers. The three fully connected layers predict the three-dimensional spatial features respectively. Each fully connected layer predicts an angle of the face pose orientation, so that the hierarchical prediction network focuses on different image regions for each of the three angles of the face pose orientation, reducing mutual interference between the three angle predictions. The dimensions are in pixels.

[0026] In an optional example, the loss function used by the hierarchical prediction network during training of the head pose estimation unit employs a self-adjusting loss constraint coefficient. This coefficient corrects the loss magnitude distortion caused by the cross-entropy loss term of the prediction angle being greater than the mean squared error loss term of the prediction angle when the average absolute error of the prediction angles of the three fully connected layers is less than a threshold. Conversely, when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold, the error penalty caused by the above two loss terms is increased, allowing the hierarchical prediction network to converge faster during training. Let the error penalty of the hierarchical prediction network when the average absolute error of the prediction angles of the three fully connected layers is less than the threshold be the first penalty, and let the error penalty of the hierarchical prediction network when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold be the second penalty. Correcting the loss magnitude distortion means controlling the first penalty to be less than the second penalty, ensuring that the hierarchical prediction network can train and learn normally.

[0027] In an optional example, the backbone network used by the head pose estimation unit includes four residual blocks; the face image is processed sequentially through the four residual blocks to obtain four image spatial features of decreasing size; these four image spatial features of decreasing size are fused by the feature pyramid network. The fusion strategy is to first fuse the first size spatial feature with the second size spatial feature, then fuse the first size spatial feature with the third size spatial feature to obtain new second size spatial features and new third size spatial features, respectively; then, the new second size spatial feature is fused with the new third size spatial feature, and so on. The second and fourth size spatial features are fused to obtain the updated third and fourth size spatial features, respectively. Finally, the updated third and fourth size spatial features are fused to obtain the fourth size fused feature. The size decreases progressively from the first to the fourth size. The dimensionality reduction module includes three convolutional kernels. The fourth size fused feature is processed by the three convolutional kernels in sequence. Each convolutional kernel performs dimensionality reduction on the input image features once, resulting in three spatial features with the same size and progressively decreasing number of channels.

[0028] In an optional example, the hierarchical prediction network used by the head pose estimation unit adjusts the three predicted angles using the following formula: in, as well as Γ1, Γ2, and Γ3 represent the predicted values ​​of pitch angle, yaw angle, and roll angle, respectively; K1, K2, and K3 are the weight factors of the three convolutional kernels, respectively; Γ1, Γ2, and Γ3 are the three spatial features obtained by the fused features through the three convolutional kernels of the dimensionality reduction module; the relationship between Γ1, Γ2, and Γ3 satisfies the following formula: Where W1 is the feedback parameter from the first convolutional kernel to the second convolutional kernel of the dimensionality reduction module, W2 is the feedback parameter from the second convolutional kernel to the third convolutional kernel of the dimensionality reduction module, b4 is the new bias term brought about by dimensionality reduction from the first convolutional kernel to the second convolutional kernel, and b5 is the new bias term brought about by dimensionality reduction from the second convolutional kernel to the third convolutional kernel.

[0029] In an optional example, the loss function of the hierarchical prediction network used by the head pose estimation unit... for:

[0030]

[0031]

[0032] in, This is the value predicted by the hierarchical prediction network for head pose, where y is the true value of the face's head pose in the image, β is the loss constraint coefficient, which is constructed by the relationship between the mean squared error loss and the cross-entropy loss, k is the number of angle categories, σ is the sigmoid function, and L... mse Y represents the mean squared error loss. ic This represents the one-hot encoding formed according to the angle category. It is the category to which the prediction angle belongs.

[0033] Where k represents the number of angle categories. Specifically, in this embodiment of the invention, -99° to 99° is divided into 66 angle intervals, each interval being 3°, corresponding to 66 angle categories. Furthermore, those skilled in the art can divide the angles into different number of categories according to actual needs, and this invention does not further limit or describe this.

[0034] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0035] This invention provides a head pose estimation method and system that treats head pose estimation as three branches of the same task, incorporating feature pyramids and multi-task convolution concepts. Compared to traditional head pose estimation methods, it reduces the mutual interference between the three angle adjustments, resulting in smaller biases in the head pose estimation results. Traditional methods use a simple addition of cross-entropy and mean squared error loss for training head pose estimation. This invention, based on an analysis of the drawbacks of traditional loss functions, optimizes the loss function, resolving the discontinuous angle estimation loss caused by the inherent discontinuity of the loss function itself, thus further improving the head pose estimation results. This method is compatible with the latest head pose estimation methods based on rotation matrices, which provides a possibility for further improving the accuracy of head pose estimation in the future. Attached Figure Description

[0036] Figure 1 A flowchart of the head pose estimation method provided in an embodiment of the present invention;

[0037] Figure 2 This is a block diagram illustrating the head pose estimation method provided in an embodiment of the present invention.

[0038] Figure 3 This is a framework diagram of the head pose estimation model proposed in an embodiment of the present invention;

[0039] Figure 4 This is an illustration of the angle prediction discontinuity problem provided in an embodiment of the present invention;

[0040] Figure 5 This is a heatmap showing the differences in focus from different angles, as proposed in the embodiments of the present invention.

[0041] Figure 6 Additional attention principle diagram provided for embodiments of the present invention;

[0042] Figure 7 This is a diagram illustrating the head pose estimation effect provided in an embodiment of the present invention.

[0043] Figure 8 This is a diagram illustrating the architecture of a head pose estimation system provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] Figure 1 A flowchart of the head pose estimation method provided in an embodiment of the present invention is shown below. Figure 1 As shown, it includes the following steps:

[0046] S101, Determine an image containing a face;

[0047] S102, the image is input into a pre-trained hierarchical prediction network to predict the pitch angle, yaw angle, and roll angle of the face pose orientation, thereby estimating the face head pose. The hierarchical prediction network includes a backbone network, a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module. The backbone network is used to extract image spatial features of different sizes. The feature pyramid network is used to fuse image spatial features of different sizes to obtain fused features. The dimensionality reduction module is used to perform dimensionality reduction on the fused features in three different dimensions to obtain three-dimensional spatial features of the image, with different dimensions corresponding to different numbers of image channels. The hierarchical prediction module includes three fully connected layers. The three fully connected layers predict the three-dimensional spatial features respectively. Each fully connected layer predicts an angle of the face pose orientation, so that the hierarchical prediction network focuses on different image regions for each of the three angles of the face pose orientation, reducing mutual interference between the three angle predictions. The dimensions are in pixels.

[0048] Specifically, this invention adopts the following technical solution: It provides a head pose estimation method with adaptive loss adjustment under hierarchical prediction, wherein hierarchical prediction refers to using different network layers to separate the prediction tasks for three angles; adaptive loss adjustment adds constraints based on the loss function itself, thereby solving the problem of discontinuity in angle prediction from the perspective of the loss function, specifically including the following steps:

[0049] (i) Cropping faces from single RGB images of people or images in standard datasets. The cropped image size is 224×224. Since data preprocessing methods are common and the principle is simple, they will not be elaborated here.

[0050] (II) The model utilizes a feature pyramid to fuse image features of different scales extracted from different network layers, and employs a dynamic adaptive spatial feature fusion strategy to automatically allocate the weights of the fused features. ResNet 50 is used as the backbone network. Combined with the feature pyramid strategy, features spanning only two layer scales are fused during the fusion process; features exceeding this ratio are not fused. After fusing the extracted image features of different sizes, a fused summary feature is obtained. This feature is then subjected to three dimensionality reduction steps to create three features with the same spatial resolution but different channel numbers, which are used for head pose estimation tasks at three different angles.

[0051] (III) Connect the three features obtained in step (II) to fully connected layers respectively, accumulating three fully connected layers, forming a one-to-one connection between the features and the fully connected layers. At this point, the model has the space to adjust the yaw, pitch, and roll angles individually, reducing the mutual influence between the three angle adjustments. yaw represents the angle of rotation around the y-axis, called the yaw angle; pitch represents the angle of rotation around the x-axis, called the pitch angle; and roll represents the angle of rotation around the z-axis, called the roll angle.

[0052] (iv) Equip the three branches formed in step (iii) with additional attention mechanisms, making the features the model focuses on more concentrated during parameter tuning and possessing identity robustness. External attention makes the model focus on the same features of people with different identities, which makes the predictions from the three perspectives form their own unique feature focus.

[0053] (V) The three features extracted through external attention in step (IV) are then used for training using cross-entropy loss and mean squared error loss. In particular, this invention proposes a dynamically self-adjusting loss constraint term, which solves the problem of discontinuous angle prediction in the traditional training process. Specifically, the mean squared error is used to constrain the cross-entropy loss by utilizing the similar relationship between the mean squared error and the cross-entropy loss, so that the predicted loss of the model during training maintains the same increasing or decreasing trend as the actual angle loss.

[0054] The general approach of each step in this invention is as follows: First, a face image is obtained by cropping, removing interference from irrelevant background factors. The reduced image size also alleviates the computational burden on the model. Second, a feature pyramid fusion strategy is used to ensure that features at different scales all play a role. The fused features possess both detailed and overall characteristics, eliminating the drawback of high-level features focusing on the overall picture due to excessively deep convolutional layers. Furthermore, a step-like head pose prediction method based on feature dimensionality reduction combines the advantages of traditional feature fusion and multi-task prediction, enabling good coordination among the three perspectives within a single model. Simultaneously, an attention mechanism ensures the universality of the features extracted by the model. Finally, an optimized loss function is used to achieve a continuous overall head pose estimation, significantly improving its accuracy.

[0055] After the above five steps, the head pose estimation method proposed in this invention solves the problems of mutual influence of angles and discontinuous angle prediction in the traditional head pose prediction process. The additional computational cost caused by using feature pyramids is also mitigated through dimensionality reduction. Validation on standard datasets shows that the head pose estimation method proposed in this invention is robust to people with different identities or different poses of the same person.

[0056] This invention provides a head pose estimation method with adaptive loss adjustment under hierarchical prediction, and its specific implementation steps are as follows:

[0057] Figure 2 This is a block diagram illustrating the implementation of a head pose estimation method with adaptive loss adjustment under hierarchical prediction according to the present invention. Figure 2 As shown, the head pose estimation method proposed in this invention comprises the following modules: (1) image input module; (2) feature fusion module; (3) hierarchical prediction module; and (4) loss constraint module. The specific operational steps involved in each module are as follows:

[0058] I. Image Input Module: First, acquire images or videos of the person's head pose to be detected, and perform preprocessing operations. Specific operations include, but are not limited to: face cropping to obtain a head image after background removal; resizing the images to 224×224 pixels; arranging images from the video in frame order (fps=60), and arranging the images in the image set sequentially. Note that the image resizing and image sorting steps do not have a specific order requirement. After preprocessing, all images need to be randomly occluded. This is to prevent the model from focusing too much on local features during training, which could reduce the model's generality. The output of the image input module is a batch of processed, standardized head images.

[0059] II. Feature Fusion Module: such as Figure 2 As shown, the feature fusion module is located below the image input module. It receives the standardized image output by the image input module and inputs the fused features into the hierarchical prediction module. A detailed diagram of the feature fusion module can be found here. Figure 3 As shown, the operational steps include: extracting features from the standard image via the ResNet-50 backbone network; for features extracted from different blocks, only a downsampling strategy is used. Regarding feature fusion, for a spatial scale ratio of 2:1, this invention uses a convolutional layer with a stride of 2 and a kernel size of 3×3 to ensure spatial scale consistency; for a spatial scale ratio of 4:1, this invention first uses max pooling with a stride of 2 for max pooling, and then uses a convolutional layer with a stride of 2 and a kernel size of 3×3 to ensure spatial scale consistency. For a spatial scale ratio of 8:1, due to the significant difference in features, this invention does not employ a spatial fusion strategy. Let S represent each stage, then the above feature fusion process can be described as follows:

[0060]

[0061] Where Sj|j=3,4 represents the last two block stages, →j indicates fusion based on the feature space scale of the current block, and γ is the fusion weight. When j=1 or 2, the value corresponding to γ2 or γ3 is 0, that is, only two stages of feature fusion are performed at this time. At the same time, this invention forces γ1+γ2+γ3=1|γ1,γ2,γ3∈[0,1]. To achieve this objective, this invention uses three 1×1 convolutional layers to calculate the weights, as shown in the following formula:

[0062]

[0063] The weights are the values ​​corresponding to the first-scale features. These are the weights corresponding to the second-scale features. γ1 represents the weight corresponding to the third scale feature, and γ2 represents the proportion obtained after a sigmoid-like weighted calculation, ultimately resulting in γ1 + γ2 + γ3 = 1.

[0064] After feature fusion, the model retains some subtle features while also focusing on the overall image. In the next stage, the fused features will be further dimensionality-reduced to form a stepped, hierarchical prediction.

[0065] In one specific embodiment, a 224×224 image is input into a convolutional neural network. After a single convolution operation with a 3×3 kernel, the image's spatial dimensions become 112×112. Then, after passing through four blocks in the ResNet-50 backbone network, image spatial features of sizes 56×56, 28×28, 14×14, and 7×7 are formed. These four-dimensional spatial features are fused using a feature pyramid, maintaining spatial size consistency during fusion. The fusion strategy is as follows: for a 56×56 spatial feature, a max pooling layer is applied to reduce its size to 28×28, followed by a 3×3 convolution to reduce it to 14×14. This is then fused with these two dimensions respectively, and the remaining fusion operations follow the same pattern. Features spanning two scales, such as 112×112 and 7×7 features, are not fused. After passing through the feature pyramid, the fused image feature size is 7×7×2048. At this point, a 1×1 convolutional kernel is used to reduce the number of channels, reducing it to 7×7×1024, which is called dimensionality reduction layer 1 (dw1). dw1 is then further reduced to obtain dimensionality reduction layer 2 (dw2), and then dimensionality reduction again to obtain dimensionality reduction layer 3 (dw3). Now, the three dimensions of features have the same spatial resolution ratio, all being 7×7 in size, but with different numbers of channels.

[0066] III. Hierarchical Prediction Module: such as Figure 2 As shown, the main function of the hierarchical prediction module is to reduce the predictions from the three angles into three branches, with each branch's predictions independent of the others. Before explaining the hierarchical prediction function proposed in this invention, it is necessary to clarify the currently commonly used methods: traditional methods treat the predictions of the three angles of head pose as three branches of the same task, which completely share the same network layers. This increases the burden on the model, such as... Figure 4 As shown, AP stands for Avg pooling. When the model adjusts its parameters based on the loss feedback from other angles, the prediction result of the angle with the smaller prediction loss may become worse because the model must maintain a balance between the adjustments of these three angles. Figure 4 In this context, MAE represents the mean absolute error.

[0067] IV. In short, compared to head pose estimation from a single angle, simultaneously estimating from three angles limits model performance. In traditional head pose estimation tasks, the prediction of the three angles can be described by the following formula:

[0068]

[0069] Where K represents different weights, Γ is the feature extracted by the convolutional layer, and b is the bias factor. θ and ψ represent the predicted values ​​for yaw, pitch, and roll, respectively. Assume the prediction loss for an image is... Since the network layers share gradients during backpropagation, the adjusted prediction loss becomes Although the overall prediction loss is reduced, it is not the optimal model for yaw. After applying the hierarchical prediction structure proposed in this invention, the adjustment formulas for the three angles are changed to:

[0070]

[0071] Wherein, Γ1, Γ2, and Γ3 are the features obtained after dimensionality reduction by three dimensionality reduction layers dw1, dw2, and dw3, resulting from the aforementioned fused features. The correlation between Γ1, Γ2, and Γ3 is as follows, where W1 and W2 are the new convolution parameters, and b4 and b5 are the new bias terms introduced by dimensionality reduction:

[0072]

[0073] This invention treats head pose estimation as three tasks, providing additional adjustment space for model parameter tuning. The prediction order of angles is determined by the distribution of sample size in the dataset. Figure 5 As shown in the figure, (a) represents the feature regions of interest obtained from various angles by traditional methods, and (b) represents the feature regions of interest from three angles obtained by the hierarchical prediction method proposed in this invention. It can be seen that after hierarchical division, the regions of interest for the three angles are no longer the same, which means that the hierarchical prediction strategy has played a role.

[0074] Subsequently, this invention adds an external attention layer to the prediction of each angle. The working principle of the external attention mechanism is as follows: Figure 6 As shown, by continuously extracting common features among the images participating in training, the weights corresponding to these common features are continuously increased, while the weights of other parts are relatively weakened. The hierarchical prediction module takes a batch of features obtained from the backbone network as input, then uses a 1×1 convolutional layer to scale the channels to reduce the computational burden. After passing through an external attention mechanism, it restores the number of channels through a 1×1 convolutional layer, and finally outputs to the loss constraint module.

[0075] IV. Loss Limitation Module: Before explaining the problem solved by the method of this invention, it is necessary to explain the problem of discontinuous loss function prediction caused by traditional methods, such as... Figure 4As shown, when the true head pose is [6.1°, -3.2°, -15°] and the predicted angle is [5.9°, -1.9°, -9.9°], the traditional loss function incorrectly reverses the actual loss relationship between yaw and pitch because the classification loss is greater than the regression loss. Furthermore, the traditional loss function also leads to an imbalance in losses on both sides of the angle classification line; intermittent losses and the incorrectly inverted loss function make it difficult for the model to learn. Let's illustrate the imbalance problem at both ends of the classification line with another simple example. Set the true angles to [0°, 3°, 5°] and the predicted angles to [1°, 3.5°, 7°]. Divide the head angles between (-99°, 99°) into 66 classes with 3° intervals. When the predicted loss is within 1°, there are two cases for angle prediction: inter-class loss and intra-class loss. When the predicted loss is intra-class loss, the cross-entropy loss is small, and the total loss follows the same trend as the true loss. However, when the predicted loss is the between-class loss, since the exponent of the mean squared error is 2, the cross-entropy loss will be greater than the mean squared loss. This will cause the total loss to trend in the opposite direction to the true loss, making it difficult for the model to learn. Traditional loss function methods can be described as:

[0076]

[0077] Where k is the number of categories, Y ic This represents the one-hot encoding formed based on the angle category, and is either 0 or 1, indicating whether the classification is correct. It represents the category to which the predicted angle belongs, σ represents softmax, and L ce L represents the cross-entropy loss. mse This represents the mean squared error loss.

[0078] Considering the synergistic effect between the two types of losses, this invention imposes additional constraints on the classification loss: The updated head pose estimation loss function is as follows:

[0079]

[0080] After loss constraint, a loss term β∈[0,1] is also added to the backpropagation gradient. When the true loss is small, the penalty is smaller. In the example above, β reduces the cross-entropy loss of the pitch angle to 1 / 5 of its original value. This resets the model's total loss to the same trend as the true loss, thus solving the problem of inconsistent angle prediction caused by traditional loss functions. This invention adds an error penalty for angle loss greater than 1° to accelerate the model's convergence speed. The input image passes through four prediction modules to complete one round of training. The model adjusts parameters through the backpropagation mechanism, making the angle prediction continuously more accurate.

[0081] To better explain the combined location information and head posture localization and detection system provided by the present invention, the following detailed description is provided in conjunction with embodiments.

[0082] Figure 7 This is a schematic diagram of head pose estimation in a complex environment provided by an embodiment of the present invention. Figure 7 This includes both cases of large-angle deflection and cases where the head is obstructed. Simultaneously, predictions are performed using the method of this invention and conventional methods, and the results are compared. Figure 7 As shown, when the head pose undergoes a large angle of deflection or is obstructed, the method proposed in this invention reduces the average angle prediction loss by more than 10° compared to traditional head pose estimation methods, and the predictions for each angle are close to the true values. This indicates that, compared with traditional methods, the head pose estimation method proposed in this invention has robustness in complex scenarios, proving that the hierarchical prediction strategy for head pose angles and the self-adjusting strategy for the loss function involved in this invention are effective.

[0083] Figure 7 In this context, Ground truth refers to the real data of head pose angles; HopeNet is an existing method, whose full name is: Fine-grained head pose estimation without keypoints; TPL-net is the head pose estimation method of this invention; and the hierarchical prediction network used in this invention is called: Tiered prediction with loss limit network.

[0084] Figure 8 This is a diagram of the head pose estimation system architecture provided in an embodiment of the present invention, such as... Figure 8 As shown, it includes:

[0085] The face image determination unit 810 is used to determine an image containing a face;

[0086] A head pose estimation unit 820 is used to input the image into a pre-trained hierarchical prediction network to predict the pitch angle, yaw angle, and roll angle of the face pose orientation, thereby estimating the face head pose. The hierarchical prediction network includes a backbone network, a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module. The backbone network is used to extract image spatial features of different sizes. The feature pyramid network is used to fuse image spatial features of different sizes to obtain fused features. The dimensionality reduction module is used to perform dimensionality reduction on the fused features in three different dimensions to obtain three-dimensional spatial features of the image, with different dimensions corresponding to different numbers of image channels. The hierarchical prediction module includes three fully connected layers. The three fully connected layers predict the three-dimensional spatial features respectively. Each fully connected layer predicts an angle of the face pose orientation, so that the hierarchical prediction network focuses on different image regions for each of the three angles of the face pose orientation, reducing mutual interference between the three angle predictions. The dimensions are in pixels.

[0087] Understandable, Figure 8 For detailed implementation of the functions of each unit, please refer to the description in the aforementioned method embodiments, which will not be repeated here.

[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A head pose estimation method, characterized by, The method comprises the following steps: determining an image containing a face; inputting the image into a pre-trained hierarchical prediction network to predict a pitch angle, a yaw angle and a roll angle of a face pose orientation to estimate a face head pose; the hierarchical prediction network comprises a backbone network, a feature pyramid network, a dimension reduction module and a hierarchical prediction module; the backbone network is used to extract image spatial features of different sizes, the feature pyramid network is used to fuse the image spatial features of different sizes to obtain fused features, and the dimension reduction module is used to perform dimension reduction on the fused features in three different dimensions to obtain image spatial features of three dimensions, different dimensions corresponding to different image channel numbers; the hierarchical prediction module comprises three fully connected layers; the three fully connected layers respectively predict the image spatial features of the three dimensions, and each fully connected layer predicts one angle of the face pose orientation, so that the hierarchical prediction network predicts three angles of the face pose orientation, each angle focusing on a different image region, thereby reducing mutual interference between the three angles; the size is in units of pixels; a self-adjusting loss limiting coefficient is used in a loss function in a training process of the hierarchical prediction network to correct a loss size distortion problem caused by a cross-entropy loss term of a predicted angle being greater than a mean square error loss term of the predicted angle when an average absolute error of the three angles predicted by the three fully connected layers is less than a threshold, and to increase error punishment caused by the two loss terms when the average absolute error of the three angles predicted by the three fully connected layers is not less than the threshold, so that the hierarchical prediction network converges at a faster speed in the training process; the error punishment of the hierarchical prediction network is a first punishment when the average absolute error of the three angles predicted by the three fully connected layers is less than the threshold, and the error punishment of the hierarchical prediction network is a second punishment when the average absolute error of the three angles predicted by the three fully connected layers is not less than the threshold, and the correction of the loss size distortion problem refers to controlling the first punishment to be less than the second punishment to ensure that the hierarchical prediction network can be normally trained and learned.

2. The method of claim 1, wherein, the backbone network comprises four residual blocks; a face image is sequentially processed by the four residual blocks to sequentially obtain four image spatial features of decreasing sizes; the four image spatial features of decreasing sizes are fused by the feature pyramid network, and a fusion strategy is that the first size spatial feature is first fused with the second size spatial feature and the first size spatial feature is fused with the third size spatial feature to respectively obtain a new second size spatial feature and a new third size spatial feature, then the new second size spatial feature is fused with the new third size spatial feature and the new second size spatial feature is fused with the fourth size spatial feature to respectively obtain a third size spatial feature updated again and a new fourth size spatial feature, and finally the third size spatial feature updated again and the new fourth size spatial feature are fused to obtain a fourth size fused feature; wherein the sizes of the first size to the fourth size gradually decrease in stages; The dimension reduction module includes three convolution kernels; the fusion features of the fourth size are sequentially processed by the three convolution kernels, and each convolution kernel reduces the dimension of the input image features, and three dimensions of spatial features with invariant size and gradually decreasing channel numbers are sequentially obtained.

3. The method of claim 2, wherein, The adjustment formula of the three angles predicted by the hierarchical prediction network is: ； wherein, , and respectively represent the predicted values of the pitch angle, the yaw angle and the roll angle; , and are weight factors of the three convolution kernels respectively; , and are three kinds of spatial features of the fusion features obtained by the three convolution kernels of the dimension reduction module. , , The relationship between them satisfies the following formula: ; wherein, is a feedback parameter from the first convolutional kernel to the second convolutional kernel of the dimension reduction module, is a feedback parameter from the second convolutional kernel to the third convolutional kernel of the dimension reduction module, is a new bias term resulting from the dimension reduction by the first convolutional kernel to the second convolutional kernel, is a new bias term resulting from the dimension reduction by the second convolutional kernel to the third convolutional kernel.

4. The method according to any one of claims 1 to 3, characterized in that, The loss function of the hierarchical prediction network is: wherein, is the value obtained by the head pose prediction of the hierarchical prediction network, is the true value of the head pose of the face in the image, is a loss limiting coefficient, which is constructed by the same big or small relationship of the mean square error loss and the cross entropy loss, is the number of angle categories, is the sigmod function, represents the mean square error loss, represents the one-hot encoding formed according to the angle category, is the category to which the predicted angle belongs.

5. A head pose estimation system, characterized by, The method comprises the steps of: The face image determination unit is configured to determine an image containing a face. The head pose estimation unit is configured to input the image into a pre-trained hierarchical prediction network to predict a pitch angle, a yaw angle and a roll angle of a face orientation to estimate a face head pose. The hierarchical prediction network includes a backbone network, a feature pyramid network, a dimension reduction module and a hierarchical prediction module. The backbone network is configured to extract image spatial features of different sizes. The feature pyramid network is configured to fuse the image spatial features of different sizes to obtain fusion features. The dimension reduction module is configured to reduce the dimensions of the fusion features to obtain three dimensions of image spatial features, and different dimensions correspond to different image channel numbers. The hierarchical prediction module includes three fully connected layers. Each fully connected layer predicts one angle of the face orientation, so that the hierarchical prediction network predicts three angles of the face orientation, each of which focuses on a different image region, reducing the mutual interference between the three angle predictions. The size is in units of pixels. The loss function in the training process of the hierarchical prediction network used by the head pose estimation unit adopts a self-adjusting loss limiting coefficient. When the average absolute error of the predicted angles of the three fully connected layers is less than a threshold value, the loss size distortion problem caused by the cross-entropy loss term of the predicted angles being greater than the mean square error loss term of the predicted angles is corrected. When the average absolute error of the predicted angles of the three fully connected layers is not less than the threshold value, the error penalty caused by the above two loss terms is increased, so that the hierarchical prediction network converges faster in the training process. When the average absolute error of the predicted angles of the three fully connected layers is less than the threshold value, the error penalty of the hierarchical prediction network is a first penalty. When the average absolute error of the predicted angles of the three fully connected layers is not less than the threshold value, the error penalty of the hierarchical prediction network is a second penalty. The correction of the loss size distortion problem refers to controlling the first penalty to be less than the second penalty to ensure that the hierarchical prediction network can be normally trained and learned.

6. The system of claim 5, wherein, The backbone network used by the head pose estimation unit includes four residual blocks. The face image is sequentially processed by the four residual blocks to sequentially obtain image spatial features of four sizes decreasing in size. The four size-decreasing image space features are fused by the feature pyramid network, and a fusion strategy is that the first size space feature is fused with the second size space feature, and the first size space feature is fused with the third size space feature, to obtain a new second size space feature and a new third size space feature respectively, then the new second size space feature is fused with the new third size space feature, and the new second size space feature is fused with the fourth size space feature, to obtain a third size space feature updated again and a new fourth size space feature respectively, finally, the third size space feature updated again and the new fourth size space feature are fused to obtain a fourth size fusion feature; wherein, the size of the first size to the fourth size decreases gradually; the dimension reduction module includes three convolution kernels; the fourth size fusion feature is processed by the three convolution kernels in turn, and each convolution kernel reduces the dimension of the input image feature once, to obtain three dimensions of space features with unchanged size and gradually decreasing number of channels in turn.

7. The system of claim 6, wherein, The adjustment formula of the hierarchical prediction network used by the head pose estimation unit for predicting three angles is: ; wherein, , and respectively represent the predicted values of the pitch angle, the yaw angle and the roll angle; , and are weight factors of three convolution kernels respectively; , and are three-dimensional spatial features obtained by the fusion features through three convolution kernels of the dimension reduction module; , , The relationship between them satisfies the following formula: ; wherein, is the feedback parameter of the first convolution kernel of the dimension reduction module to the second convolution kernel, is the feedback parameter of the second convolution kernel of the dimension reduction module to the third convolution kernel, is a new bias term brought by the dimension reduction of the first convolution kernel to the second convolution kernel, is a new bias term brought by the dimension reduction of the second convolution kernel to the third convolution kernel.

8. The system of any one of claims 5 to 7, wherein, Loss function of the hierarchical prediction network used by the head pose estimation unit is: wherein, is a value obtained by the head pose prediction of the hierarchical prediction network, is a true value of the head pose of the face in the image, is a loss limiting coefficient, which is constructed by the same large and small relationship of the mean square error loss and the cross entropy loss, is the number of angle categories, is a sigmod function, represents the mean square error loss, represents one-hot encoding formed according to the angle category, is the category to which the predicted angle belongs.

Images