Head posture estimation method and system based on nerve implicit field

Abstract

The invention provides a head posture estimation method and system based on a neural implicit field, and belongs to the technical field of image processing. According to the method, discretization processing is carried out on a head posture label, and a Gaussian function and a Gumbel function are used for joint modeling to generate an asymmetrically distributed label. Sampling points are arranged at fixed intervals in a three-dimensional space, and global features of an input RGB image are extracted through a RepVang network. And establishing a mapping relationship between image features and spatial sampling points by means of a neural implicit field, mapping face information to a point cloud space, and performing binary classification on each sampling point by using a multi-layer perceptron to generate human face point cloud data. Geometric features of the point cloud data are extracted and fused with global features of the RGB image, and the head posture is predicted through combination of a classification module and a regression module. By adopting the method and the system, the precision and the robustness of head posture estimation can be effectively improved, and a reliable solution is provided for practical application.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a head posture estimation method and system based on neural implicit field. Background Art

[0002] With the continuous development of computer vision technology, head pose estimation has extremely broad application prospects in many fields, including human-computer interaction, virtual reality, augmented reality, intelligent driving, and security monitoring. For example, in the field of human-computer interaction, accurate head pose estimation can enable computers to better understand user intentions, thereby achieving a more natural and smooth interactive experience. In intelligent driving scenarios, real-time monitoring of the driver's head pose can promptly determine whether the driver is paying attention, providing important protection for driving safety.

[0003] However, current head pose estimation technology faces many challenges. In real-world scenarios, the human head often experiences significant deflection movements, which can cause some facial information to be lost from the visual range. At the same time, occlusion caused by external environmental factors often occurs. These factors combined make the image information input into the pose estimation model incomplete, which in turn causes the model to produce large errors when estimating head pose, affecting the accuracy and reliability of pose estimation. Furthermore, existing technologies have significant limitations in their choice of data sources: mainstream RGB images lack three-dimensional information, resulting in significant errors at large pose angles or occlusions; depth maps contain spatial dimensions but have low resolution and severe noise interference; point cloud data relies on expensive equipment and is difficult to popularize, making it difficult to meet actual needs.

[0004] Traditional head pose estimation methods can be divided into two main categories. The first category involves direct regression, which attempts to predict head pose angles directly from the input image. However, these methods are sensitive to samples affected by noise and struggle to accurately predict the precise pose angles. The second category involves regression with classification constraints. These methods introduce a classification layer to model the data distribution, converting head pose angles into discrete classification labels. These labels generate expected values ​​within a predefined classification interval, weight the predicted values ​​based on the predicted angles, and calculate a regression loss. This method leverages the data distribution to some extent, helping to improve pose estimation performance. However, head pose varies continuously and arbitrarily within a specific range. When input samples exhibit high correlation, this method tends to classify highly correlated samples into the same category, introducing significant errors and limiting the accuracy of pose estimation.

[0005] In summary, the existing head posture estimation technology has certain limitations and is unable to meet the application field's demand for high-precision posture estimation when faced with complex actual situations such as large head deflection and external occlusion. Summary of the Invention

[0006] The purpose of the present invention is to provide a head posture estimation method and system based on neural implicit fields, which effectively solves the estimation error problem caused by occlusion and lighting changes at large posture angles, significantly improves the accuracy and robustness of head posture estimation, and provides a highly reliable solution for practical application scenarios such as fatigue driving monitoring and virtual reality interaction.

[0007] To achieve the above object, the present invention provides a head posture estimation method based on neural implicit field, comprising the following steps:

[0008] Step S1: discretize the head posture labels and use Gaussian function and Gumbel function to jointly model and generate asymmetric distribution labels;

[0009] Step S2: setting sampling points at fixed intervals in three-dimensional space and extracting global features of the input RGB image through the backbone network;

[0010] Step S3: input the global features of the RGB image and the three-dimensional coordinate information of the sampling points into a multi-layer perceptron, perform binary classification on each sampling point, and generate human face point cloud data;

[0011] Step S4: extract the geometric features of the human face point cloud data, fuse them with the global features of the RGB image, and jointly predict the head posture through the classification module and regression module.

[0012] Preferably, in step S1, the head posture label includes yaw angle, pitch angle and roll angle.

[0013] Preferably, in step S1, the specific steps of the discretization process include: dividing the head posture angle range into multiple intervals, and obtaining the label value of each interval by jointly calculating the Gaussian function and the Gumbel function:

[0014]

[0015] Among them, β, s l , δ and κ represent the scale parameter, location parameter, Gaussian function variance and head pose label value respectively.

[0016] Preferably, in step S2, the backbone network is a RepVgg network, which includes multiple RepVggBlock modules, an Identity branch and a residual branch are introduced into each RepVggBlock module, and the network layer is converted into a 3×3 convolutional layer through a feature fusion strategy.

[0017] Preferably, in step S3, the structure of the multilayer perceptron includes five layers of MLP, wherein the input dimension of MLP1 is 131 and the output dimension is 2048; the input dimension of MLP2 is 2048 and the output dimension is 1024; the input dimension of MLP3 is 1024 and the output dimension is 512; the input dimension of MLP4 is 512 and the output dimension is 131; the input dimension of MLP5 is 131 and the output dimension is 2, and MLP5 is used to determine whether the sampling point belongs to the face surface.

[0018] Preferably, in step S3, the calculation formula for generating human face point cloud data is as follows:

[0019]

[0020] Among them, x feature represents the global features extracted by the image encoder, x coord represents the point cloud coordinate information, P represents the human face surface space, and f(·) represents the mapping relationship of the multi-layer MLP.

[0021] Preferably, in step S3, the loss function used is as follows:

[0022]

[0023] Among them, L c represents the cross entropy loss, N represents the total number of samples, and p i represents the probability when sample i is predicted to be 1, κ i represents the label of sample i.

[0024] Preferably, in step S4, the classification module includes three independent multi-layer perceptrons, which respectively predict the classification intervals of the yaw angle, pitch angle and roll angle; the regression module sums the products of the representative values ​​of each interval according to the classification results to obtain the final attitude angle.

[0025] Preferably, in step S4, the loss function used is as follows:

[0026]

[0027] Among them, L e represents the total loss, k represents the total number of categories, and n represents the category index. represents the distribution of network predictions, κ k represents the true label of k, p k Indicates the representative value of the angle interval corresponding to each category in the network prediction distribution result.

[0028] The present invention also provides a head posture estimation system based on neural implicit field, comprising:

[0029] The data preprocessing module is used to detect and crop the face area of ​​the input RGB image and discretize the yaw, pitch, and roll angle labels of the head posture;

[0030] The label modeling module generates asymmetric distribution labels by combining Gaussian function and Gumbel function;

[0031] The 3D sampling module generates sampling points at fixed intervals within a preset 3D space.

[0032] The feature extraction module uses the RepVgg backbone network to extract the global features of the input image and fuses the multi-branch convolution into a single 3×3 convolution through structural reparameterization;

[0033] The point cloud generation module uses a multi-layer perceptron to fuse the coordinates of the sampling points and image features, and selects a three-dimensional point cloud representing the face surface through binary classification;

[0034] Multimodal fusion module, extracting point cloud geometric features and fusing them with global image features;

[0035] The attitude prediction module includes a classification module and a regression module, which realizes the joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation respectively.

[0036] Therefore, the present invention adopts the above-mentioned head posture estimation method and system based on neural implicit field, and the beneficial technical effects are as follows:

[0037] (1) Improved estimation accuracy: Using neural implicit fields combined with RGB image features to generate facial point cloud data provides richer information for head pose estimation, making the estimation results more accurate. Compared with methods based solely on RGB images, this method significantly improves the estimation accuracy of yaw, pitch, and roll angles.

[0038] (2) Enhanced robustness: By generating human facial point cloud data to supplement the information missing due to large posture angles or occlusion, it can effectively cope with interference factors such as lighting changes and occlusion, and has stronger robustness, and can obtain stable and reliable results on different data sets.

[0039] (3) Optimize feature fusion: effectively integrate point cloud features with image features, fully tap into their complementary information, and enable the model to more comprehensively understand the head posture features, thereby improving the accuracy and robustness of posture estimation.

[0040] (4) Improve model efficiency: Use the RepVgg model as the backbone network to reduce the number of network parameters, simplify the model structure while ensuring performance, meet real-time reasoning requirements, and have good adaptability and flexibility.

[0041] (5) Innovative loss function: Cross-entropy loss function and comprehensive loss function are designed for different stages, so that the model can simultaneously learn to generate accurate human facial point cloud data and head posture information, providing a clear goal for training and improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is the overall structure diagram of the model;

[0043] Figure 2 This is the result of ablation experiment;

[0044] Figure 3 For comparison of AFLW2000 results;

[0045] Figure 4 is the rolling test result; among them, Figure 4 (a) is the comprehensive mean absolute error distribution of head posture angle; Figure 4 (b) is the mean absolute error distribution of yaw angle groups; Figure 4 (c) in the figure is the pitch angle error distribution of the pitch angle group; Figure 4 (d) in the figure is the pitch angle error distribution of the roll angle group;

[0046] Figure 5 Visualize head pose estimation;

[0047] Figure 6 are the sampling points of rectangular area and spherical area, where Figure 6 (a) in the figure is the sampling point of the rectangular area; Figure 6 (b) in the figure is the sampling point of the spherical area. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0050] Example 1

[0051] 1. Data preparation and preprocessing.

[0052] Dataset: Three datasets are used: BIWI, AFLW2000 and 300WLP.

[0053] Face detection and cropping: Use the MTCNN model to detect the face area in the image, expand the width and height by 0.4 times with the face as the center, and crop the RGB image of size 224×224 as input.

[0054] Label alignment: Generate corresponding labels based on the Euler angles (yaw, pitch, and roll) of the head posture provided by the dataset and discretize them.

[0055] 2. Discretization of head pose labels.

[0056] Discretization method: Combine Gaussian function and Gumbel function to generate asymmetric distribution labels.

[0057]

[0058] Among them, β, s l , δ and κ represent the scale parameter, location parameter, Gaussian function variance and head pose label value respectively.

[0059] The head posture angle range is set to [-100, 100], divided into 66 intervals, each interval is 3°.

[0060] The Gumbel function is used to adjust the asymmetry of the distribution and enhance visual differentiation (e.g., the labels with a 38.27° rightward tilt and a -40.39° leftward tilt are distinguished).

[0061] Parameter settings: Gaussian function variance δ = 1, Gumbel function scale parameter β = 3.

[0062] 3. Setting of three-dimensional space sampling points.

[0063] Sampling area: Sampling points are taken at fixed intervals in the x, y, z∈[-1,1] space. In the experiment of determining the sampling area, the original rectangular sampling space is changed. 2 +y 2 +z 2 = 1, sampling is performed at a fixed interval. The points sampled in the regions x,y,z∈[-1,1] and x,y,z∈[-1,1] are as follows Figure 6 shown.

[0064] from Figure 6 It can be observed that the same number of sampling points in a spherical area is denser than in a rectangular area. In the sampling area experiment, 300WLP was used as the training set and AFLW2000 was used as the test set. The experimental results are shown in Table 1.

[0065] Table 1 The impact of sampling area on model estimation accuracy

[0066] Space area Yaw(°) Pitch(°) Roll(°) Mean(°) Rectangular area (4096) 4.32 2.52 2.94 3.26 Spherical area (2048) 4.55 2.61 3.12 3.42 Spherical area (3072) 4.42 3.15 2.93 3.50 Spherical area (4096) 4.64 2.75 3.07 3.48

[0067] As can be seen from Table 1, the mean absolute error is smaller when using a rectangular area as the sampling point area than when using a spherical area. Increasing the number of sampling points in a spherical area does not increase the overall estimation accuracy of the model. This is because a spherical area is smaller than a rectangular area, and the same number of sampling points will appear denser in a spherical area. Increasing the number of sampling points does not improve the estimation accuracy of the model, but instead reduces the model's inference speed. Therefore, the present invention selects a rectangular area.

[0068] Number of samples: A total of 4096 points are sampled in the rectangular area.

[0069] 4. Model construction (such as Figure 1 shown).

[0070] Backbone network: uses the RepVgg network, which contains 5 RepVggBlock modules. Each RepVggBlock module introduces an identity branch and a residual branch. During inference, all layers are converted to 3×3 convolutions through the feature fusion strategy.

[0071] Point cloud generation module (first stage):

[0072] Input: 224×224 RGB image and 4096 sample point coordinates (3D).

[0073] Structure: 5-layer MLP (parameters are shown in Table 2), the last layer uses binary classification to determine whether the sampling point belongs to the face surface.

[0074] Output: Generate 512 human facial point cloud data.

[0075] The calculation formula for generating human face point cloud data is as follows:

[0076]

[0077] Among them, x feature represents the global features extracted by the image encoder, x coord represents the point cloud coordinate information, P represents the human face surface space, and f(·) represents the mapping relationship of the multi-layer MLP.

[0078] Table 2 Parameters of the first stage multi-layer perceptron (MLP)

[0079]

[0080]

[0081] Head pose estimation module (second stage):

[0082] Input: Point cloud data and image global features generated in the first stage.

[0083] Structure: 3-layer MLP (parameters are shown in Table 3), aggregating point cloud features through MaxPool and concatenating them with image features.

[0084] Prediction module: Contains a classification module (three independent MLPs predict the yaw, pitch, and roll angle intervals respectively) and a regression module (weighted summation to obtain the final angle).

[0085] Table 3 Parameters of the second stage multi-layer perceptron (MLP)

[0086] enter parameter Output MLP1 3 256 64 MLP2 64 8320 128 MLP3 128 132096 1024

[0087] 5. Model training and parameter setting.

[0088] Training environment: Intel i5-1135G7 processor, GTX3060 GPU, PyTorch1.71 framework.

[0089] Hyperparameters: initial learning rate 0.001, Adam optimizer, batch_size = 16, training for 80 epochs, and learning rate decay of 0.7 times every 20 epochs.

[0090] Loss function:

[0091] First stage: cross entropy loss:

[0092]

[0093] Among them, L c represents the cross entropy loss, N represents the total number of samples, and p i represents the probability when sample i is predicted to be 1, κ i represents the label of sample i.

[0094] Second stage: loss function:

[0095]

[0096] Among them, L e represents the total loss, k represents the total number of categories, and n represents the category index. represents the distribution of network predictions, κ k represents the true label of k, p k Indicates the representative value of the angle interval corresponding to each category in the network prediction distribution result.

[0097] 6. Test results.

[0098] 6.1、Dataset test:

[0099] BIWI dataset: The mean absolute error is 2.38° (yaw), 2.70° (pitch), and 2.39° (roll).

[0100] AFLW2000 dataset: The mean absolute error is 2.91° (yaw), 4.39° (pitch), and 2.83° (roll).

[0101] 6.2 Ablation Experiment

[0102] To prove that the task of generating a human face point cloud data model in the first stage is beneficial to the head pose estimation in the second stage. In the experiment, the research was divided into three groups, each with different task settings and goals. In the first group of experiments, the model only performed the head pose classification and regression task, and did not perform the point cloud model generation task; the second group of models simultaneously performed the point cloud generation task and performed head pose estimation in the first stage; the third group of models simultaneously performed the first and second stage tasks, including point cloud generation and head pose estimation. The experimental results are as follows Figure 2 shown.

[0103] Method 1 means that only the backbone network is used to estimate the head pose, Method 2 means that both the generation of human face point cloud and head pose estimation are performed simultaneously, and Method 3 means that both the first stage generation of human face point cloud and the second stage head pose estimation are performed simultaneously. Figure 2 We observed that Method 3 achieved lower mean absolute errors at all pose angles than Methods 2 and 1, while Method 2 achieved lower mean absolute errors than Method 1. These experimental results demonstrate that multiple losses can help the network better predict head pose, reduce background interference, and generate facial point cloud data that can further improve head pose estimation accuracy.

[0104] 6.3. Comparative experiment:

[0105] The proposed method is compared with the currently commonly used head pose estimation method. The experiments are divided into two groups. The first group uses 300WLP as the training set and AFLW2000 as the test set. The results are as follows: Figure 3 The second set used 70% of the BIWI data as the training set and 30% as the test set. By comparing the two sets of experimental results, we can evaluate the performance of the proposed method on different datasets. The first set of experiments validated the performance of the method on the commonly used 300WLP and AFLW2000 datasets, while the second set of experiments further verified the generalization ability of the method on the BIWI dataset. The experimental results are shown in Table 4.

[0106] from Figure 3It can be seen that the proposed method achieves optimal results for yaw, pitch, and roll angles on the AFLW2000 test set. Compared with the DADL method, the proposed method reduces the errors in yaw, pitch, and roll angles by 1.38°, 1.08°, and 1.43°, respectively, and its accuracy is significantly improved. This is because the proposed method generates a facial point cloud in the first stage, which can supplement the information missing at large posture angles and due to occlusion caused by external factors. This method is more robust than head posture estimation based solely on RGB images.

[0107] Table 4 Model comparison experiment

[0108]

[0109]

[0110] As can be seen from Table 4, the BIWI dataset was collected in a laboratory environment and therefore has less noise than the 300WLP and AFLW2000 datasets, resulting in a lower mean absolute error in head pose estimation. In terms of yaw angle, pitch angle, and roll angle, the proposed method improves the accuracy by 0.91°, 0.44°, and 0.21°, respectively, compared to the optimal algorithm in Table 4. In the comparative experiments, the proposed model achieved the best results on both the AFLW2000 and BIWI datasets. Comprehensive experimental results show that the proposed method performs well on different datasets, can effectively handle head pose estimation tasks, improves the accuracy and generalization ability of pose estimation, and provides a reliable solution for practical applications.

[0111] In order to conduct more fine-grained testing and thus better evaluate the generalization of the model, each of the 24 data groups in the BIWI dataset is used as a test set, while the remaining groups are used for training. The training results are shown in Figure 2. Figure 4 shown. Figure 4 The results of 24 test experiments were presented, in which the average absolute error of each head posture angle was kept below 6°, and the absolute error of each head posture angle was even lower, less than 5°. Figure 4As can be seen in (b, c, d), the yaw angle error is mostly below 3°, with an average error of approximately 4°; the pitch angle error is mostly around 2°, with an average of approximately 3°; and the roll angle error is mostly below 3°, with an average error of approximately 4°. However, the error in the second test set is larger. The images also show that the head pose estimation accuracy is lower in the second experimental group due to hair occluding the face. However, the overall pose estimation error remains below 4.5°, within an acceptable error range. These results demonstrate that the proposed method surpasses current mainstream BIWI methods in head pose recognition, confirming its reliability and robustness and supporting its reliability in practical applications.

[0112] Figure 5 This is a visualization of the test results of the model of the present invention on AFLW2000. The first row is the RGB image; the white color in the second row is the point cloud of the face generated by the model in the first stage, and the blue color is the sampling points in space; the third row is the visualization of the head posture estimation results of the second stage model on the RGB image.

[0113] Example 2

[0114] A head pose estimation system based on neural implicit field, comprising:

[0115] The data preprocessing module is used to detect and crop the face area of ​​the input RGB image and discretize the yaw, pitch, and roll angle labels of the head posture;

[0116] The label modeling module generates asymmetric distribution labels by combining Gaussian function and Gumbel function;

[0117] The 3D sampling module generates sampling points at fixed intervals within a preset 3D space.

[0118] The feature extraction module uses the RepVgg backbone network to extract the global features of the input image and fuses the multi-branch convolution into a single 3×3 convolution through structural reparameterization;

[0119] The point cloud generation module uses a multi-layer perceptron to fuse the coordinates of the sampling points and image features, and selects a three-dimensional point cloud representing the face surface through binary classification;

[0120] Multimodal fusion module, extracting point cloud geometric features and fusing them with global image features;

[0121] The attitude prediction module includes a classification module and a regression module, which realizes the joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation respectively.

[0122] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0123] Therefore, the present invention adopts the above-mentioned head posture estimation method and system based on neural implicit field, which can effectively improve the accuracy and robustness of head posture estimation and provide a reliable solution for practical applications.

[0124] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A head posture estimation method based on neural implicit field, characterized in that: The following steps are involved: Step S1: discretize the head posture labels and use Gaussian function and Gumbel function to jointly model and generate asymmetric distribution labels; Step S2: setting sampling points at fixed intervals in three-dimensional space and extracting global features of the input RGB image through the backbone network; Step S3: input the global features of the RGB image and the three-dimensional coordinate information of the sampling points into a multi-layer perceptron, perform binary classification on each sampling point, and generate human face point cloud data; Step S4: extract the geometric features of the human face point cloud data, fuse them with the global features of the RGB image, and jointly predict the head posture through the classification module and regression module.

2. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S1, the head posture label includes yaw angle, pitch angle and roll angle.

3. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S1, the specific steps of the discretization process include: dividing the head posture angle range into multiple intervals, and the label value of each interval is obtained by jointly calculating the Gaussian function and the Gumbel function: Among them, β, s l , δ and κ represent the scale parameter, location parameter, Gaussian function variance and head pose label value respectively.

4. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S2, the backbone network is a RepVgg network, which contains multiple RepVggBlock modules. Identity branches and residual branches are introduced into each RepVggBlock module, and the network layer is converted into a 3×3 convolutional layer through a feature fusion strategy.

5. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S3, the structure of the multilayer perceptron includes five layers of MLP, where the input dimension of MLP1 is 131 and the output dimension is 2048; the input dimension of MLP2 is 2048 and the output dimension is 1024; the input dimension of MLP3 is 1024 and the output dimension is 512; the input dimension of MLP4 is 512 and the output dimension is 131; the input dimension of MLP5 is 131 and the output dimension is 2. MLP5 is used to determine whether the sampling point belongs to the face surface.

6. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S3, the calculation formula for generating human face point cloud data is as follows: Among them, x feature represents the global features extracted by the image encoder, x coord represents the point cloud coordinate information, P represents the human face surface space, and f(·) represents the mapping relationship of the multi-layer MLP.

7. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S3, the loss function used is as follows: Among them, L c represents the cross entropy loss, N represents the total number of samples, and p i represents the probability when sample i is predicted to be 1, κ i represents the label of sample i.

8. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S4, the classification module contains three independent multi-layer perceptrons, which respectively predict the classification intervals of the yaw angle, pitch angle, and roll angle; the regression module sums the products of the representative values ​​of each interval according to the classification results to obtain the final attitude angle.

9. The head posture estimation method based on neural implicit field according to claim 1, characterized in that: In step S4, the loss function used is as follows: Among them, L e represents the total loss, k represents the total number of categories, and n represents the category index. represents the distribution of network predictions, κ k represents the true label of k, p k Indicates the representative value of the angle interval corresponding to each category in the network prediction distribution result.

10. A head posture estimation system based on neural implicit field, characterized in that: include: The data preprocessing module is used to detect and crop the face area of ​​the input RGB image and discretize the yaw, pitch, and roll angle labels of the head posture; The label modeling module generates asymmetric distribution labels by combining Gaussian function and Gumbel function; The 3D sampling module generates sampling points at fixed intervals within a preset 3D space. The feature extraction module uses the RepVgg backbone network to extract the global features of the input image and fuses the multi-branch convolution into a single 3×3 convolution through structural reparameterization; The point cloud generation module uses a multi-layer perceptron to fuse the coordinates of the sampling points and image features, and selects a three-dimensional point cloud representing the face surface through binary classification; Multimodal fusion module, extracting point cloud geometric features and fusing them with global image features; The attitude prediction module includes a classification module and a regression module, which realizes the joint estimation of yaw angle, pitch angle and roll angle through classification interval prediction and weighted summation respectively.

Images