A method for optimizing three-dimensional human head reconstruction based on a single image

Abstract

The application is a kind of three-dimensional human head reconstruction method based on a single image, belonging to the field of computer vision. The single image is input, preprocessed first, then input into the network model for analysis, and the required parameters of the FLAME parameterized human head model are output, the three-dimensional human head is constructed by using the FLAME model, then the three-dimensional face key points on the initial human head grid are calculated, and the three-dimensional key points of the image face are obtained to apply constraints, the loss is calculated by using the two groups of key points to optimize the model parameters, and a more accurate three-dimensional human head model is obtained. The network also outputs a displacement map with wrinkle detail information, which is used to add wrinkle detail effect to the reconstructed human head model. The optimization method of the application still has constraint optimization for three-dimensional reconstruction in the application stage, and the three-dimensional human head reconstruction effect of the single image is obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and is an optimization method for 3D head reconstruction based on a single image. Background Technology

[0002] 3D head reconstruction is an important research area in computer vision, used to recover 3D heads from images or videos. The ideal result is a realistic reproduction of the head on a computer, including its shape, pose, expression, and detailed features. When using multiple images or videos of the same person for head reconstruction, more information is available, usually resulting in good results. However, in real-world scenarios, it is sometimes difficult to obtain multiple images or videos of the same person, and only a single image is available. Currently, there are several problems with reconstructing heads using a single image. Due to the limited information contained in a single image, auxiliary models are often needed to reconstruct the head's shape, pose, and expression. Using parametric models can transform the 3D head reconstruction problem into a problem of obtaining and optimizing model parameters. However, in unconstrained situations, reconstructing a head from a single image using parametric models often results in a reconstruction that is biased towards an average head or poor reconstruction of certain parts. Furthermore, due to the lack of facial details, such as wrinkles, the reconstruction result may lack realism.

[0003] In 3D head reconstruction based on a single image, because a single image contains less information than multiple images, it is necessary to eliminate the influence of interference information and extract as much useful information as possible during the image preprocessing stage to achieve better reconstruction results. Compared with using 2D keypoint constraints, using 3D keypoint constraints can better constrain the reconstruction of the 3D head model. Therefore, this invention uses 3D keypoint constraints to optimize model parameters and improve the accuracy of the reconstructed head mesh. Furthermore, it predicts head detail information through the network and adds wrinkle details to the head model, making the head reconstruction effect more realistic. Most reconstruction methods only have constraints during the training phase; the optimization method of this invention still has a constrained optimization scheme for 3D reconstruction during the application phase, significantly improving the 3D head reconstruction effect from a single image. Summary of the Invention

[0004] Purpose of the invention: This invention is an optimization method for 3D head reconstruction based on a single image, which aims to improve the accuracy of 3D head reconstruction using parametric models and make the reconstructed head more realistic.

[0005] Technical solution: An optimization method for 3D head reconstruction based on a single image, characterized by the following steps: S1: Input a human head RGB image, detect the face in the image, locate the human head, locate the 3D facial key points, and blur the background of the human head image; S2: Input the detected and cropped head image into the network model for analysis. Predict the parameters required for the FLAME parameterized model from the image. Furthermore, use the network model to predict facial detail parameters from the image, outputting a feature map with wrinkle detail information; S3: Construct a 3D human head using the FLAME parametric model and calculate the 3D facial key points on the initial human head mesh; S4: Compare the detected 3D facial landmarks with those calculated from the initial head mesh, optimize the parameters using a loss function, and reconstruct the 3D head using the optimized model parameters; S5: Use feature maps with wrinkle details to enhance the wrinkle effect on the human head, process the human head mesh texture, and complete the 3D human head reconstruction.

[0006] In step S1, the head position is located from the image. A multi-task cascaded convolutional neural network model is used to detect the face, outputting the coordinates of the two opposite corners of the face's square bounding box. The top left corner is marked as (x1, y1), and the bottom right corner is marked as (x2, y2). The bounding box coordinates and a custom extended height and width are used to expand the face rectangle to cover the entire head position. The head is located based on the detected face, and the calculation formula is introduced:

[0007] The height distance of the expansion is h, the width distance of the expansion is d, the coordinates of the upper left corner after the selection are represented as (x1-d, y1-h), and the coordinates of the lower right corner are represented as (x1+d, y1+h). In order to ensure that the image size is uniform and neat, integer division operators are used. Then, the head is cropped out, and the background of the cropped head image is blurred using a mask to reduce the influence of interference factors on the head reconstruction.

[0008] In step S2, the detected and cropped head image is input into the network model for analysis. The network consists of two parts: one part predicts the parameters required for the FLAME parameterized model, and the other part predicts facial detail information. The network outputs a 256×256×1 feature map, which is a displacement map containing wrinkle detail information. Since a single image is used to reconstruct the 3D head, in order to improve the network's feature extraction capability and extract as much useful information as possible from the image, the network part of this invention uses a channel attention mechanism based on adaptive receptive fields, which can use convolutional kernels of different sizes to perform convolution operations according to different inputs. The network model uses multiple combinations of convolutional kernels of different sizes. 7×7 and 3×3 convolutional kernels are used to obtain pose parameters, and 7×7, 5×5, and 3×3 convolutional kernels are used to obtain facial detail parameters. Other parameters are obtained using 5×5 and 3×3 convolutional kernels. Feature extraction is performed on the head region using convolutional kernels of different sizes, yielding feature maps with varying information. These feature maps represent different aspects of the head, providing richer features than those extracted by a single convolutional kernel. Different weights are then assigned to these feature maps to determine their contribution to the head representation, thereby effectively enhancing the head feature extraction results.

[0009] In step S4, the network model used in this method not only uses ground truth to constrain the reconstruction effect during training, but also constrains the head reconstruction in the practical application stage. During head reconstruction in the practical application stage, a relatively accurate 3D facial landmark localization method is used to locate 68 3D facial landmarks, which are considered as ground truth and used to constrain the 68 3D facial landmarks calculated after the initial head reconstruction. The loss is calculated using the detected landmarks and the landmarks calculated from the head mesh, and the following formula is introduced to calculate the loss:

[0010] Among the two sets of parameters x i and y i These represent the coordinates detected by the network and the coordinates calculated from the initial head mesh, respectively, with l being the array length. A threshold k is set during reconstruction; when the loss falls below the threshold, the expected result is considered achieved, and the final 3D head model is reconstructed using the parameters at this point. Based on the training results, k=0.03 was tested, which improves both the reconstruction effect and the reconstruction speed.

[0011] Beneficial effects: This invention discloses a method for optimizing 3D head reconstruction based on a single image. It uses a single RGB image as input, feeds the preprocessed image into a network model for analysis, outputs the parameters required for a FLAME parametric head model, constructs a 3D head model using the FLAME model, calculates the 3D facial keypoints on the initial head mesh, and calculates the loss using the calculated 3D facial keypoints and those detected in the preprocessing. The network outputs optimized model parameters, resulting in a more accurate 3D head model. Furthermore, the network outputs a displacement map with wrinkle detail information to add wrinkle detail effects to the reconstructed head model. This invention improves the accuracy of 3D head reconstruction using parametric models, making the reconstructed 3D head model more realistic. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the reconstruction of a three-dimensional human head according to the present invention; Figure 2 This is a schematic diagram illustrating the principle of calculating facial key points from a human head mesh triangular facet in this invention. Figure 3 This is a rendering of a three-dimensional human face with key points annotated according to the present invention. Figure 4 This is a comparison chart of the training effects of using three-dimensional keypoint constraints and two-dimensional keypoint constraints in this invention; Figure 5 This is a comparison of the reconstruction effects of using three-dimensional keypoint constraints and two-dimensional keypoint constraints in this invention. Detailed Implementation

[0013] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0014] An optimization method for 3D head reconstruction based on a single image, characterized by the following steps: S1: Input a human head RGB image, detect the face in the image, locate the human head, locate the 3D facial key points, and blur the background of the human head image; S2: Input the detected and cropped head image into the network model for analysis. Predict the parameters required for the FLAME parameterized model from the image. Furthermore, use the network model to predict facial detail parameters from the image, outputting a feature map with wrinkle detail information; S3: Construct a 3D human head using the FLAME parametric model and calculate the 3D facial key points on the initial human head mesh; S4: Compare the detected 3D facial landmarks with those calculated from the initial head mesh, optimize the parameters using a loss function, and reconstruct the 3D head using the optimized model parameters; S5: Utilizing feature maps containing wrinkle details to enhance the wrinkle effect on the head, the head mesh texture is processed to complete the 3D head reconstruction. The head reconstruction process is as follows: Figure 1 As shown.

[0015] First, the image is preprocessed. A multi-task cascaded convolutional network is used to detect faces. A pre-trained face detection model and a 68-point facial landmark model are used to detect faces and 68 3D facial landmarks. After detecting a face, the network needs to locate the head. The network outputs the coordinates of the two opposite corners of the face's square bounding box, with the top-left corner marked as (x1, y1) and the bottom-right corner marked as (x2, y2). The bounding box coordinates and a custom-defined height and width are used to expand the face rectangle to cover the entire head area. Based on the detected face, the head is located using the following formula:

[0016] The height distance of the expansion is h, the width distance of the expansion is d, the coordinates of the upper left corner after the selection are represented as (x1-d, y1-h), and the coordinates of the lower right corner are represented as (x1+d, y1+h). In order to ensure that the image size is uniform and neat, integer division operators are used. Then, the head is cropped out, and the background of the cropped head image is blurred using a mask to reduce the influence of interference factors on the head reconstruction.

[0017] The detected and cropped head image is input into the network model of this invention for analysis. The network consists of two parts: one part predicts the parameters required for the FLAME parameterized model. Using the FLAME model, a complete 3D head model controlled by these parameters is constructed. The other part of the network predicts facial detail information to construct the facial wrinkle detail effect. The network outputs a 256×256×1 feature map, i.e., a displacement map with wrinkle detail information. Since a single image is used to reconstruct the 3D head, to improve the network's feature extraction capability and extract as much useful information as possible, the network part of this invention uses a channel attention mechanism based on adaptive receptive fields, which can use convolutional kernels of different sizes to perform convolution operations according to different inputs. The network model uses multiple combinations of convolutional kernels of different sizes. A combination of 7×7 and 3×3 convolutional kernels is used to obtain pose parameters, and a combination of 7×7, 5×5, and 3×3 convolutional kernels is used to obtain facial detail parameters. Other parameters are obtained using a combination of 5×5 and 3×3 convolutional kernels. Feature extraction is performed on the head region using convolutional kernels of different sizes, yielding feature maps with varying information. These feature maps represent different aspects of the head, providing richer features than those extracted by a single convolutional kernel. Different weights are assigned to different feature maps to determine their contribution to the head representation, thus effectively enhancing the head feature extraction results.

[0018] Then, the 3D facial keypoints on the initial head mesh are calculated. Since the FLAME parametric model divides the head surface into many small triangles, the coordinates P of the head mesh vertex represented by the three vertices of each triangle are calculated. A formula similar to the centroid calculation formula for triangles is introduced to calculate the coordinates of the keypoints, as shown in the formula:

[0019] Where A, B, and C are the three vertices of the triangle, and d1, d2, and d3 are the distances from point P to the three vertices, respectively. The schematic diagram is shown below. Figure 2 .

[0020] After calculating the coordinates of all vertices, the indices of the vertices of each triangle face in the head mesh are extracted to find the indices of the triangle faces corresponding to the 3D facial keypoints. Then, the triangle faces of all 3D keypoints are extracted, the coordinates of the 3D keypoints on the head mesh are calculated, and stored in an array. Finally, the positions of the 3D facial keypoints are found from the 3D head mesh, and the 3D facial keypoints are calculated and labeled. The result is as follows: Figure 3 As shown.

[0021] The network model used in this method not only uses ground truth constraints to limit the reconstruction effect during training, but also continues to constrain head reconstruction during practical application. In the practical application stage, during head reconstruction, a relatively accurate 3D facial landmark localization method is used to locate 68 3D facial landmarks, which are considered ground truth and used to constrain the 68 3D facial landmarks calculated after the initial head reconstruction. The loss is calculated using the detected landmarks and the landmarks calculated from the head mesh, and the following formula is used to calculate the loss:

[0022] Among the two sets of parameters x i and y i These represent the coordinates detected by the network and the coordinates calculated from the initial head mesh, respectively, with l being the array length. A threshold k is set during the reconstruction process; when the loss falls below the threshold, the desired effect is considered achieved, and the final 3D head model is reconstructed using the parameters at this point.

[0023] A comparative experiment was conducted using 3D and 2D keypoint supervision, and the results were analyzed across different training epochs. The 3D keypoint constraint method showed that it effectively reduced the loss. Based on the training performance, k=0.03 was set, which improved both the reconstruction quality and speed. The training results are as follows: Figure 4 As shown.

[0024] Comparing the reconstruction results using 3D keypoint constraints and those using only 2D keypoint constraints, testing revealed that reconstructing a human head using only 2D keypoint constraints sometimes failed to achieve the average head shape, resulting in a significantly distorted mesh. Figure 5 As shown in the left figure. When the reconstruction effect of 2D keypoint constraints is poor, using 3D keypoint constraints can achieve a better reconstruction effect, such as... Figure 5 As shown in the right figure, this demonstrates that using 3D keypoint constraints can improve the accuracy of head mesh reconstruction.

[0025] Finally, a displacement map with wrinkle detail information is used to deform the head mesh model generated by the parametric model. The displacement map with wrinkle detail information is applied to the head mesh, and the vertices of the head mesh are deformed according to the displacement values ​​stored in the displacement map. By embedding height information on the 3D mesh surface, the surface geometry is changed to represent details, increasing its bumpiness and thus adding a wrinkle effect to the reconstructed head model. Then, a texture is applied to the head mesh to complete the 3D head reconstruction.

[0026] This invention improves the accuracy of human head mesh reconstruction based on a single image by using the constraints of three-dimensional key points to reconstruct a more realistic human head model. It also adds wrinkle details to the reconstructed human head model, making the reconstructed human head more realistic and has broad application prospects.

Claims

1. A method for three-dimensional head reconstruction based on a single image, characterized in that: This method is performed according to the following steps: S1: Input a human head RGB image, detect the face in the image, locate the human head, locate the 3D facial key points, and blur the background of the human head image; S2: Input the detected and cropped head image into the network model, predict the parameters required for the FLAME parameterization model from the image, and use the network model to predict the facial detail parameters from the image, and output a feature map with wrinkle detail information. S3: Construct a 3D human head using the FLAME parametric model and calculate the 3D facial key points on the initial human head mesh; S4: Compare the detected 3D facial landmarks with those calculated from the initial head mesh, optimize the parameters using a loss function, and reconstruct the 3D head using the optimized model parameters; S5: Use feature maps with wrinkle detail information to enhance the wrinkle effect on the human head, process the human head mesh texture, and complete the 3D human head reconstruction; In step S1, the head position is located from the image. A multi-task cascaded convolutional neural network model is used to detect the face, outputting the coordinates of the two opposite corners of the face's square bounding box. The top left corner is marked as (x1, y1), and the bottom right corner is marked as (x2, y2). The bounding box coordinates and a custom extended height and width are used to expand the face rectangle to cover the entire head position. The head is located based on the detected face, and the calculation formula is introduced: The height distance of the expansion is h, the width distance of the expansion is d, the coordinates of the upper left corner after the selection are (x1-d, y1-h), and the coordinates of the lower right corner are (x1+d, y1+h). In order to ensure that the image size is uniform and neat, the integer division operator is used. Then the head is cropped out, and the background of the cropped head image is blurred by using a mask to reduce the influence of interference factors on the head reconstruction. In step S2, the detected and cropped head image is input into the network model for analysis. The network consists of two parts: one part predicts the parameters required for the FLAME parameterized model, and the other part predicts facial detail information. The network outputs a 256×256×1 feature map, which is a displacement map with wrinkle detail information. All parts of the network use a channel attention mechanism based on adaptive receptive fields, which can use convolutional kernels of different sizes to perform convolution operations according to different inputs. The network model uses multiple combinations of convolutional kernels of different sizes. 7×7 and 3×3 convolutional kernels are used to obtain pose parameters, and 7×7, 5×5, and 3×3 convolutional kernels are used to obtain facial detail parameters. Other parameters are obtained using 5×5 and 3×3 convolutional kernels. Different sizes of convolutional kernels are used to extract features from the head region to obtain feature maps with different information. Different weights are then assigned to different feature maps to determine the contribution of different feature maps to the head representation, thereby effectively enhancing the head feature extraction effect. In step S4, during the head reconstruction phase in practical applications, the network model used locates 68 3D facial key points using a 3D facial key point localization method. These are considered ground truth and used to constrain the 68 3D facial key points calculated after the initial head reconstruction. The loss is calculated using the detected key points and the key points calculated from the head mesh, and the following formula is introduced to calculate the loss: Among the two sets of parameters x i and y i These are the coordinates detected by the network and the coordinates calculated from the initial head mesh, respectively, with l being the array length. A threshold k is set during the reconstruction process; when the loss is below the threshold, it is considered to have achieved the expected effect, and the final 3D head model is reconstructed using the parameters at this time. Based on the effect test during training, k=0.03 is used, which can improve the reconstruction effect while ensuring the reconstruction speed.

Images