A head pose estimation method and system based on neural implicit field

Abstract

The application provides a head pose estimation method and system based on neural implicit field, belonging to the technical field of image processing. The method discretizes the head pose label, and generates an asymmetric distribution label by jointly modeling Gaussian function and Gumbel function. Sampling points are set at a fixed interval in three-dimensional space, and the global features of the input RGB image are extracted by a RepVgg network. The mapping relationship between image features and spatial sampling points is established by means of neural implicit field, the face information is mapped to point cloud space, and a multilayer perceptron is used to perform binary classification on each sampling point to generate human face point cloud data. The geometric features of the point cloud data are extracted and fused with the global features of the RGB image, and the head pose is predicted by a classification and regression module. The application adopts the above method and system, which can effectively improve the accuracy and robustness of head pose estimation, and provides a reliable solution for practical application.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a head pose estimation method and system based on neural implicit fields. Background Technology

[0002] With the continuous development of computer vision technology, head pose estimation has extremely broad application prospects in many fields such as human-computer interaction, virtual reality, augmented reality, intelligent driving, and security monitoring. For example, in the field of human-computer interaction, accurate head pose estimation enables computers to better understand the user's intentions, thereby achieving a more natural and smooth interactive experience; in intelligent driving scenarios, by monitoring the driver's head pose in real time, it is possible to determine whether the driver's attention is focused, providing an important guarantee for driving safety.

[0003] However, current head pose estimation techniques face numerous challenges. In real-world scenarios, the human head often undergoes significant turning movements, resulting in the loss of some facial information from the visual field. Simultaneously, occlusion caused by external environmental factors frequently occurs. These factors combined render the image information input into the pose estimation model incomplete, leading to substantial errors in head pose estimation and impacting its accuracy and reliability. Furthermore, existing technologies suffer from significant limitations in data source selection: mainstream RGB images lack 3D information, resulting in significant errors at large pose angles or under occlusion; depth maps, while containing spatial dimensions, suffer from low resolution and severe noise interference; and point cloud data relies on expensive equipment and is difficult to widely deploy, failing to meet practical needs.

[0004] Traditional head pose estimation methods can be broadly categorized into two types. The first type is direct regression, which attempts to predict the head pose angle directly from the input image. However, this method is sensitive to noisy samples and struggles to accurately predict the precise pose angle. The second type involves regression with classification constraints. This method introduces a classification layer to model the data distribution, converting the head pose angle into discrete classification labels. Expected values ​​are generated within predefined classification intervals, and the predicted values ​​are weighted according to the predicted angles, and the regression loss is calculated. This method utilizes the data distribution information to some extent, which helps improve the performance of pose estimation. However, head pose is continuous and arbitrarily variable within a specific range. When input samples exhibit high correlation, this method tends to group highly correlated samples into the same class, introducing significant errors and limiting the improvement of pose estimation accuracy.

[0005] In summary, existing head pose estimation techniques are limited in meeting the high-precision pose estimation requirements of application fields when faced with complex realities such as large head rotations and external occlusions. Summary of the Invention

[0006] The purpose of this invention is to provide a head pose estimation method and system based on neural implicit fields, which effectively solves the estimation error problem caused by occlusion and illumination changes under large pose angles, significantly improves the accuracy and robustness of head pose estimation, and provides a highly reliable solution for practical application scenarios such as fatigue driving monitoring and virtual reality interaction.

[0007] To achieve the above objectives, this invention provides a head pose estimation method based on neural implicit fields, comprising the following steps:

[0008] Step S1: Discretize the head pose labels and use Gaussian and Gumbel functions to jointly model and generate asymmetric distribution labels.

[0009] Step S2: Set sampling points at fixed intervals in three-dimensional space, and extract global features of the input RGB image through the backbone network;

[0010] Step S3: Input the global features of the RGB image and the three-dimensional coordinate information of the sampling points into the multilayer perceptron, perform binary classification on each sampling point, and generate human face point cloud data;

[0011] Step S4: Extract the geometric features of the human face point cloud data and fuse them with the global features of the RGB image. Then, use the classification module and regression module to jointly predict the head pose.

[0012] Preferably, in step S1, the head attitude label includes yaw angle, pitch angle, and roll angle.

[0013] Preferably, in step S1, the specific steps of discretization include: dividing the head pose angle range into multiple intervals, and calculating the label value of each interval using a combination of the Gaussian and Gumbel functions.

[0014]

[0015] Among them, β, s l δ and κ represent the scale parameter, position parameter, Gaussian function variance, and head pose label value, respectively.

[0016] Preferably, in step S2, the backbone network is a RepVgg network, which contains multiple RepVggBlock modules. Each RepVggBlock module introduces an Identity branch and a residual branch, and the network layers are converted into 3×3 convolutional layers through a feature fusion strategy.

[0017] Preferably, in step S3, the structure of the multilayer perceptron includes five MLP layers, where MLP1 has an input dimension of 131 and an output dimension of 2048; MLP2 has an input dimension of 2048 and an output dimension of 1024; MLP3 has an input dimension of 1024 and an output dimension of 512; MLP4 has an input dimension of 512 and an output dimension of 131; and MLP5 has an input dimension of 131 and an output dimension of 2. MLP5 is used to determine whether the sampling point belongs to the face surface.

[0018] Preferably, in step S3, the calculation formula for generating human face point cloud data is as follows:

[0019]

[0020] Where, x feature x represents the global features extracted by the image encoder. coord The coordinate information of the point cloud is represented by P, which represents the surface space of the human face, and f(·) represents the mapping relationship of the multi-layer MLP.

[0021] Preferably, the loss function used in step S3 is as follows:

[0022]

[0023] Among them, L c p represents the cross-entropy loss, where N represents the total number of samples. i κ represents the probability that sample i is predicted to be 1. i The label represents sample i.

[0024] Preferably, in step S4, the classification module includes three independent multilayer perceptrons, which predict the classification intervals of yaw angle, pitch angle and roll angle respectively; the regression module sums the products of the representative values ​​of each interval according to the classification results to obtain the final attitude angle.

[0025] Preferably, the loss function used in step S4 is as follows:

[0026]

[0027] Among them, L e Let k represent the total loss, k represent the total number of categories, and n represent the category index. κ represents the distribution of network predictions. k p represents the true label of k. k This represents the representative value of the angular interval corresponding to each category in the network prediction distribution results.

[0028] This invention also provides a head pose estimation system based on neural implicit fields, comprising:

[0029] The data preprocessing module is used to detect and crop the face region of the input RGB image, and to discretize the yaw angle, pitch angle and roll angle labels of the head posture.

[0030] The label modeling module generates asymmetric distributed labels by combining the Gaussian and Gumbel functions.

[0031] The three-dimensional sampling module generates sampling points at fixed intervals within a preset three-dimensional space.

[0032] The feature extraction module uses the RepVgg backbone network to extract global features of the input image and fuses multi-branch convolutions into a single 3×3 convolution through structural reparameterization.

[0033] The point cloud generation module uses a multilayer perceptron to fuse the coordinates of the sampling points and image features, and then uses binary classification to select the three-dimensional point cloud representing the surface of the human face.

[0034] The multimodal fusion module extracts point cloud geometric features and merges them with global image features;

[0035] The attitude prediction module includes a classification module and a regression module, which achieve joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation, respectively.

[0036] Therefore, the present invention employs the aforementioned head pose estimation method and system based on neural implicit fields, and the beneficial technical effects are as follows:

[0037] (1) Improved estimation accuracy: Human face point cloud data is generated by combining neural implicit fields with RGB image features, providing richer information for head pose estimation and making the estimation results more accurate. Compared with methods based solely on RGB images, this method significantly improves the estimation accuracy of yaw, pitch, and roll angles.

[0038] (2) Enhanced robustness: By generating human face point cloud data, information loss caused by large pose angles or occlusion is supplemented, effectively coping with interference factors such as lighting changes and occlusion, and has stronger robustness, and can obtain stable and reliable results on different datasets.

[0039] (3) Optimize feature fusion: Effectively fuse point cloud features with image features, fully explore the complementary information between the two, enable the model to understand head pose features more comprehensively, and thus improve the accuracy and robustness of pose estimation.

[0040] (4) Improve model efficiency: The RepVgg model is used as the backbone network, which reduces the number of network parameters, simplifies the model structure, and ensures performance while meeting the needs of real-time inference. It has good adaptability and flexibility.

[0041] (5) Innovative loss function: Design cross-entropy loss function and comprehensive loss function for different stages, so that the model can learn to generate accurate human face point cloud data and head pose information at the same time, providing a clear target for training and improving model performance. Attached Figure Description

[0042] Figure 1 This is a diagram of the overall structure of the model;

[0043] Figure 2 These are the results of the ablation experiment;

[0044] Figure 3 Comparison of AFLW2000 results;

[0045] Figure 4 The results are from the rolling test; where, Figure 4 (a) in the figure represents the distribution of the comprehensive average absolute error of the head posture angle; Figure 4 (b) in the figure represents the mean absolute error distribution of the yaw angle grouping; Figure 4 (c) in the figure represents the pitch angle error distribution of the pitch angle group; Figure 4 In the figure, (d) represents the pitch angle error distribution of the roll angle group;

[0046] Figure 5 Visualize head pose estimation;

[0047] Figure 6 Sampling points are provided for rectangular and spherical regions, where, Figure 6 (a) in the diagram represents the sampling points in the rectangular region; Figure 6 (b) in the diagram represents the sampling point in the spherical region. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0050] Example 1

[0051] 1. Data preparation and preprocessing.

[0052] Datasets: The datasets used are BIWI, AFLW2000, and 300WLP.

[0053] Face detection and cropping: The MTCNN model is used to detect face regions in the image, and the width and height are expanded by 0.4 times with the face as the center to crop out an RGB image of size 224×224 as input.

[0054] Label alignment: Generate corresponding labels based on the Euler angles (yaw, pitch, roll) of the head pose provided by the dataset, and discretize them.

[0055] 2. Head pose label discretization.

[0056] Discretization method: Combine the Gaussian function and the Gumbel function to generate asymmetric distributed labels.

[0057]

[0058] Among them, β, s l δ and κ represent the scale parameter, position parameter, Gaussian function variance, and head pose label value, respectively.

[0059] The head posture angle range is set to [-100, 100], divided into 66 intervals, each interval being 3°.

[0060] Use the Gumbel function to adjust the asymmetry of the distribution and enhance visual differentiation (e.g., distinguishing labels tilted to the right at 38.27° and to the left at -40.39°).

[0061] Parameter settings: Gaussian function variance δ = 1, Gumbel function scaling parameter β = 3.

[0062] 3. Setting up sampling points in three-dimensional space.

[0063] Sampling region: Sampling points are taken at fixed intervals in the space x, y, z ∈ [-1, 1]. In the experiment of determining the sampling region, the original rectangular sampling space is changed. Within the spherical region x 2 +y 2 +z 2 In the range of 1, sampling is performed at fixed intervals. Points sampled in the regions x,y,z∈[-1,1] and x,y,z∈[-1,1] are as follows: Figure 6 As shown.

[0064] from Figure 6 It can be observed that the same number of sampling points are more densely distributed in the spherical region than in the rectangular region. In the sampling region experiment, 300WLP was used as the training set and AFLW2000 as the test set, and the experimental results are shown in Table 1.

[0065] Table 1. Impact of sampling region on model estimation accuracy

[0066] spatial region Yaw(°) Pitch (°) Roll(°) Mean (°) Rectangular region (4096) 4.32 2.52 2.94 3.26 Spherical region (2048) 4.55 2.61 3.12 3.42 Spherical region (3072) 4.42 3.15 2.93 3.50 Spherical region (4096) 4.64 2.75 3.07 3.48

[0067] As shown in Table 1, the mean absolute error is smaller when using a rectangular region as the sampling area than when using a spherical region. Increasing the number of sampling points in a spherical region does not increase the overall estimation accuracy of the model because a spherical region has a smaller volume than a rectangular region, and the same number of sampling points will appear more densely packed in a spherical region. Therefore, increasing the number of sampling points does not improve the model's estimation accuracy but rather affects the model's inference speed. Thus, this invention chooses a rectangular region.

[0068] Number of samples: A total of 4096 points were sampled in the rectangular area.

[0069] 4. Model building (e.g.) Figure 1 (As shown).

[0070] Backbone network: The RepVgg network is adopted, which contains 5 RepVggBlock modules. Each RepVggBlock module introduces an Identity branch and a Residual branch. During inference, all layers are converted into 3×3 convolutions through a feature fusion strategy.

[0071] Point cloud generation module (Phase 1):

[0072] Input: A 224×224 RGB image with coordinates of 4096 sampling points (3D).

[0073] Structure: 5-layer MLP (parameters are shown in Table 2), the last layer uses binary classification to determine whether the sampling point belongs to the face surface.

[0074] Output: Generate point cloud data for 512 human faces.

[0075] The calculation formula for generating human facial point cloud data is as follows:

[0076]

[0077] Where, x feature x represents the global features extracted by the image encoder. coord The coordinate information of the point cloud is represented by P, which represents the surface space of the human face, and f(·) represents the mapping relationship of the multi-layer MLP.

[0078] Table 2. Parameters of the first-stage multilayer perceptron (MLP)

[0079]

[0080]

[0081] Head pose estimation module (second stage):

[0082] Input: Point cloud data generated in the first stage and global image features.

[0083] Structure: 3-layer MLP (parameters are shown in Table 3), which aggregates point cloud features through MaxPool and stitches them with image features.

[0084] Prediction module: includes a classification module (three independent MLPs predict yaw, pitch, and roll angle ranges respectively) and a regression module (weighted summation to obtain the final angle).

[0085] Table 3. Parameters of the Second-Stage Multilayer Perceptron (MLP)

[0086] enter parameter Output MLP1 3 256 64 MLP2 64 8320 128 MLP3 128 132096 1024

[0087] 5. Model training and parameter setting.

[0088] Training environment: Intel i5-1135G7 processor, GTX3060 GPU, PyTorch 1.71 framework.

[0089] Hyperparameters: initial learning rate 0.001, Adam optimizer, batch_size = 16, training for a total of 80 epochs, with the learning rate decaying to 0.7 times every 20 epochs.

[0090] Loss function:

[0091] Phase 1: Cross-entropy loss

[0092]

[0093] Among them, L c p represents the cross-entropy loss, where N represents the total number of samples. i κ represents the probability that sample i is predicted to be 1. i The label represents sample i.

[0094] Phase Two: Loss Function

[0095]

[0096] Among them, L e Let k represent the total loss, k represent the total number of categories, and n represent the category index. κ represents the distribution of network predictions. k p represents the true label of k. k This represents the representative value of the angular interval corresponding to each category in the network prediction distribution results.

[0097] 6. Test results.

[0098] 6.1 Dataset Testing:

[0099] The BIWI dataset has a mean absolute error of 2.38° (yaw), 2.70° (pitch), and 2.39° (roll).

[0100] AFLW2000 dataset: mean absolute error is 2.91° (yaw), 4.39° (pitch), and 2.83° (roll).

[0101] 6.2 Ablation Experiment:

[0102] To demonstrate that the first-stage task of generating human facial point cloud data is beneficial for the second-stage head pose estimation, the study was divided into three groups, each with different task settings and objectives. In the first group, the model only performed the head pose classification and regression task, without performing the point cloud model generation task; the second group's model performed both point cloud generation and head pose estimation in the first stage; the third group's model performed both the first and second stage tasks, including point cloud generation and head pose estimation. Experimental results are as follows: Figure 2 As shown.

[0103] Method 1 indicates head pose estimation only through the backbone network; Method 2 indicates simultaneous generation of human face point clouds and head pose estimation; Method 3 indicates simultaneous execution of the first-stage human face point cloud generation and the second-stage head pose estimation tasks. Figure 2 The results show that Method3 achieves a lower mean absolute error (MAE) across all pose angles compared to Method2 and Method1, while Method2 achieves a lower MAE than Method1. These experimental results demonstrate that multiple loss can help the network better predict head pose, reduce background interference, and the generated human face point cloud data can further improve the accuracy of head pose estimation.

[0104] 6.3 Comparative Experiment:

[0105] The method proposed in this invention was compared with commonly used head pose estimation methods. The experiment was divided into two groups. The first group used 300WLP as the training set and AFLW2000 as the test set. The results are as follows: Figure 3 The second group used 70% of the BIWI data as the training set and 30% as the test set. By comparing the experimental results of the two groups, the performance of the proposed method on different datasets can be evaluated. The first group of experiments verified the performance of the method on the commonly used 300WLP and AFLW2000 datasets, while the second group of experiments further verified the generalization ability of the method on the BIWI dataset. The experimental results are shown in Table 4.

[0106] from Figure 3As can be seen, the method proposed in this invention achieves optimal results in yaw, pitch, and roll angles on the AFLW2000 test set. Compared with the DADL method, the method proposed in this invention reduces the errors in yaw, pitch, and roll angles by 1.38°, 1.08°, and 1.43°, respectively, significantly improving its accuracy. This is because the method proposed in this invention generates a human face point cloud in the first stage, which can supplement the information lost due to occlusion caused by large attitude angles and external factors. This method is more robust than estimating head pose solely from RGB images.

[0107] Table 4 Model Comparison Experiment

[0108]

[0109]

[0110] As shown in Table 4, the BIWI dataset was collected in a laboratory environment, resulting in less noise compared to the 300WLP and AFLW2000 datasets, leading to a lower mean absolute error in head attitude estimation. Compared to the optimal algorithm in Table 4, the proposed method improves the accuracy of yaw, pitch, and roll angles by 0.91°, 0.44°, and 0.21°, respectively. In comparative experiments, the proposed model achieved optimal results on both the AFLW2000 and BIWI datasets. The overall experimental results demonstrate that the proposed method performs well on various datasets, effectively handling head attitude estimation tasks, improving the accuracy and generalization ability of attitude estimation, and providing a reliable solution for practical applications.

[0111] To conduct more granular testing and better evaluate the model's generalization ability, each of the 24 data groups in the BIWI dataset was used as a separate test set, while the remaining groups were used for training. The training results are as follows: Figure 4 As shown. Figure 4 The results of 24 test experiments were presented, in which the mean absolute error of each head posture angle remained below 6°, and the absolute errors of each head posture angle were even lower, all less than 5°. From Figure 4As shown in (b, c, d), the yaw angle error is mostly below 3°, with an average error of about 4°; the pitch angle error is mostly around 2°, with an average of about 3°; and the roll angle error is mostly below 3°, with an average error of about 4°. However, the second group, used as the test set, has a larger error. This can also be observed from the images. In the second experimental group, the face is obscured by hair, resulting in lower accuracy of head pose estimation. However, the overall average pose estimation error still remains below 4.5°, which is within an acceptable error range. These results indicate that the proposed method surpasses the current mainstream BIWI methods in head pose recognition, confirming the reliability and robustness of the method and providing support for its reliability in practical applications.

[0112] Figure 5 The test results of the model of this invention on AFLW2000 are visualized. The first row is an RGB image; the second row shows the point cloud of the face generated by the model in the first stage with white color and the sampling points in space with blue color; the third row is a visualization of the head pose estimation results of the model in the second stage on the RGB image.

[0113] Example 2

[0114] A head pose estimation system based on neural implicit fields includes:

[0115] The data preprocessing module is used to detect and crop the face region of the input RGB image, and to discretize the yaw angle, pitch angle and roll angle labels of the head posture.

[0116] The label modeling module generates asymmetric distributed labels by combining the Gaussian and Gumbel functions.

[0117] The three-dimensional sampling module generates sampling points at fixed intervals within a preset three-dimensional space.

[0118] The feature extraction module uses the RepVgg backbone network to extract global features of the input image and fuses multi-branch convolutions into a single 3×3 convolution through structural reparameterization.

[0119] The point cloud generation module uses a multilayer perceptron to fuse the coordinates of the sampling points and image features, and then uses binary classification to select the three-dimensional point cloud representing the surface of the human face.

[0120] The multimodal fusion module extracts point cloud geometric features and merges them with global image features;

[0121] The attitude prediction module includes a classification module and a regression module, which achieve joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation, respectively.

[0122] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0123] Therefore, the present invention employs the above-mentioned head pose estimation method and system based on neural implicit fields, which can effectively improve the accuracy and robustness of head pose estimation and provide a reliable solution for practical applications.

[0124] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A head pose estimation method based on neural implicit fields, characterized in that, Includes the following steps: Step S1: Discretize the head pose labels and use Gaussian and Gumbel functions to jointly model and generate asymmetric distribution labels. Step S2: Set sampling points at fixed intervals in three-dimensional space, and extract global features of the input RGB image through the backbone network; Step S3: Input the global features of the RGB image and the three-dimensional coordinate information of the sampling points into the multilayer perceptron, perform binary classification on each sampling point, and generate human face point cloud data; Step S4: Extract the geometric features of the human face point cloud data and fuse them with the global features of the RGB image. Then, use the classification module and the regression module to jointly predict the head pose. In step S2, the backbone network is a RepVgg network, which contains multiple RepVggBlock modules. Each RepVggBlock module introduces an Identity branch and a residual branch, and the network layer is converted into a 3×3 convolutional layer through a feature fusion strategy. In step S3, the structure of the multilayer perceptron includes five MLP layers, where MLP1 has an input dimension of 131 and an output dimension of 2048; MLP2 has an input dimension of 2048 and an output dimension of 1024; MLP3 has an input dimension of 1024 and an output dimension of 512; MLP4 has an input dimension of 512 and an output dimension of 131; and MLP5 has an input dimension of 131 and an output dimension of 2. MLP5 is used to determine whether the sampling point belongs to the face surface. In step S4, the classification module contains three independent multilayer perceptrons, which predict the classification intervals for yaw angle, pitch angle and roll angle respectively; The regression module sums the products of the representative values ​​of each interval based on the classification results to obtain the final attitude angle.

2. The head pose estimation method based on neural implicit fields according to claim 1, characterized in that, In step S1, the head attitude labels include yaw angle, pitch angle, and roll angle.

3. The head pose estimation method based on neural implicit fields according to claim 1, characterized in that, In step S1, the specific steps of discretization include: dividing the head pose angle range into multiple intervals, and calculating the label value of each interval using the Gaussian function and the Gumbel function. ； in, , , and These represent the scale parameter, position parameter, Gaussian function variance, and head pose label value, respectively.

4. The head pose estimation method based on neural implicit fields according to claim 1, characterized in that, In step S3, the calculation formula for generating human face point cloud data is as follows: ； in, This represents the global features extracted by the image encoder. Represents point cloud coordinate information. Represents the surface space of the human face. This represents the mapping relationship of a multi-level MLP.

5. The head pose estimation method based on neural implicit fields according to claim 1, characterized in that, In step S3, the loss function used is as follows: ； in, Represents cross-entropy loss, Represents the total number of samples. Indicates sample The probability of predicting a value of 1. Indicates sample The tag.

6. The head pose estimation method based on neural implicit fields according to claim 1, characterized in that, In step S4, the loss function used is as follows: ； in, Indicates the total loss. Indicates the total number of categories. Indicates category index, This represents the distribution of network predictions. express The true label, This represents the representative value of the angular interval corresponding to each category in the network prediction distribution results.

7. A head pose estimation system based on neural implicit fields, characterized in that, A method for performing head pose estimation based on neural implicit fields as described in any one of claims 1-6, comprising: The data preprocessing module is used to detect and crop the face region of the input RGB image, and to discretize the yaw angle, pitch angle and roll angle labels of the head posture. The label modeling module generates asymmetric distributed labels by combining the Gaussian and Gumbel functions. The three-dimensional sampling module generates sampling points at fixed intervals within a preset three-dimensional space. The feature extraction module uses the RepVgg backbone network to extract global features of the input image and fuses multi-branch convolutions into a single 3×3 convolution through structural reparameterization. The point cloud generation module uses a multilayer perceptron to fuse the coordinates of the sampling points and image features, and then uses binary classification to select the three-dimensional point cloud representing the surface of the human face. The multimodal fusion module extracts point cloud geometric features and merges them with global image features; The attitude prediction module includes a classification module and a regression module, which achieve joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation, respectively.

Images