Three-dimensional head model reconstruction method, device and terminal

Abstract

The application provides a three-dimensional head model reconstruction method and device and a terminal, and belongs to the technical field of three-dimensional modeling. The method comprises the following steps: obtaining target video data, which is obtained by shooting around a human head; obtaining three-dimensional point cloud and camera pose information corresponding to a video frame image in the target video data; determining position information of a three-dimensional head model according to the three-dimensional point cloud and the camera pose information based on an initial three-dimensional head model, wherein the position information comprises position information of facial feature points; adjusting the three-dimensional head model according to the three-dimensional point cloud, the camera pose information and the position information of the three-dimensional head model to obtain a target three-dimensional head model; and performing texture rendering on the target three-dimensional head model according to the camera pose information and a target texture generation model. In the embodiment of the application, the modeling process is relatively simple without the need for complex hardware devices or special environment sites, and the cost of three-dimensional head model reconstruction can be reduced.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method, apparatus and terminal for reconstructing a 3D head model. Background Technology

[0002] Existing methods for reconstructing 3D face models mainly include 3D scanning based on lasers or structured light or 3D reconstruction based on multi-view images. However, these methods require complex hardware equipment and strictly controlled environments to achieve high accuracy. They are time-consuming, slow, and extremely costly, making them difficult to use for applications aimed at ordinary users. Summary of the Invention

[0003] This invention provides a method, apparatus, and terminal for reconstructing a three-dimensional head model, which solves the problem of high cost in the existing technology for reconstructing three-dimensional face models.

[0004] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for reconstructing a three-dimensional head model, comprising:

[0005] Acquire target video data, wherein the target video data is a video captured around the head of a person;

[0006] For video frame images in the target video data, obtain the 3D point cloud and camera pose information corresponding to the video frame images;

[0007] Based on the initial 3D head model, the position information of the 3D head model is determined according to the 3D point cloud and the camera pose information, and the position information includes the position information of facial feature points;

[0008] Based on the 3D point cloud, the camera pose information, and the position information of the 3D head model, the 3D head model is adjusted to obtain the target 3D head model.

[0009] Based on the camera pose information and the target texture generation model, the target 3D head model is texture rendered.

[0010] Optionally, obtaining the 3D point cloud and camera pose information corresponding to the video frame image in the target video data includes:

[0011] The SURF (Speeded Up Robust Features) algorithm is used to detect the feature points of the video frame image.

[0012] The feature points of the video frame image are corrected using the KLT (Kanade-Lucas-Tomasi Tracking) algorithm;

[0013] Based on the feature point matching relationship between two adjacent frames, estimate the three-dimensional position of the feature points and the three-dimensional camera pose information corresponding to the video frame image;

[0014] Based on the 3D position of the feature points and the 3D camera pose information, the 3D point cloud and the camera pose information corresponding to the video frame image are obtained.

[0015] Optionally, the step of using the corner tracking KLT algorithm to correct the feature points of the video frame image includes:

[0016] Using the KLT algorithm, a first feature point that matches the video frame image and the second target image is obtained, wherein the second target image is the next frame image of the video frame image;

[0017] Based on the feature points in the second target image other than the first feature point, the feature points of the video frame image are corrected.

[0018] Optionally, determining the position information of the 3D head model based on the 3D point cloud and the camera pose information includes:

[0019] Based on the three-dimensional point cloud and the camera pose information, determine the two-dimensional face feature points obtained by projecting the face feature points on the three-dimensional head model onto the camera.

[0020] The two-dimensional facial feature points are matched with the target two-dimensional facial feature points to obtain the position information of the three-dimensional head model;

[0021] The target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image.

[0022] Optionally, adjusting the 3D head model based on the 3D point cloud, the camera pose information, and the position information of the 3D head model to obtain the target 3D head model includes:

[0023] Based on the 3D point cloud, the camera pose information, and the position information of the 3D head model, the shape and expression of the 3D head model are adjusted to obtain the target shape and target expression of the target 3D head model.

[0024] Using a geometric optimization algorithm, the vertices of the 3D head model and the 3D point cloud are matched, and the position information of the 3D head model is adjusted to minimize the point-to-surface distance between the vertices of the 3D head model and the matched 3D point cloud, thereby obtaining the target position information of the target 3D head model.

[0025] Optionally, the step of performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model includes:

[0026] Generate latent space vectors based on the 3D point cloud;

[0027] The latent space vector is input into the target texture generation model to obtain the first texture image;

[0028] Based on the camera pose information, the target texture image after the first texture image is projected by the camera is obtained;

[0029] The target texture image is used to perform texture rendering on the target 3D head model.

[0030] Optionally, before performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model, the method further includes:

[0031] Obtain a training dataset, which includes face images;

[0032] The texture generation network model is trained using the training dataset to obtain the target texture generation model.

[0033] Optionally, training the texture generation network model using the training dataset includes:

[0034] Based on the 3D point cloud corresponding to the first face image in the training dataset, a latent space vector satisfying a Gaussian distribution is generated.

[0035] The latent space vector is input into the generator of the texture generation network model to obtain a two-dimensional image;

[0036] The two-dimensional image is used as a texture and rendered on a three-dimensional face model to obtain a second face image;

[0037] The first face image and the second face image are respectively input into the discriminator of the texture generation network model to obtain the discrimination result;

[0038] Based on the discrimination result, the parameters of the generator and the discriminator are adjusted;

[0039] Repeat the above steps until the parameters of the generator and the discriminator converge.

[0040] Secondly, the present invention also provides a three-dimensional head model reconstruction device, comprising:

[0041] The data acquisition module is used to acquire target video data, which is a video shot around a person's head;

[0042] The first processing module is used to obtain the three-dimensional point cloud and camera pose information corresponding to the video frame image in the target video data.

[0043] The second processing module is used to determine the position information of the three-dimensional head model based on the initial three-dimensional head model, according to the three-dimensional point cloud and the camera pose information, wherein the position information includes the position information of facial feature points;

[0044] The third processing module is used to adjust the three-dimensional head model according to the three-dimensional point cloud, the camera pose information and the position information of the three-dimensional head model to obtain the target three-dimensional head model.

[0045] The rendering module is used to perform texture rendering on the target 3D head model based on the camera pose information and the target texture generation model.

[0046] Thirdly, the present invention also provides a terminal, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0047] Fourthly, the present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the first aspect or the method described in the first aspect.

[0048] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0049] In an embodiment of the present invention, a video is captured around a person's head. By analyzing and processing the images in the video, information such as the corresponding 3D point cloud and camera pose can be obtained. This information is then used to adjust the initial 3D head model to reconstruct the 3D head model of the person. This solution does not require complex hardware or special environments, the modeling process is relatively simple, it can reduce the modeling cost of the 3D head model, and effectively improve the realism of face rendering. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the three-dimensional head model reconstruction method provided in the embodiments of the present invention;

[0052] Figure 2 This is a structural block diagram of the three-dimensional head model reconstruction device provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the hardware structure of the terminal provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the technical problems, technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this invention. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this invention. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0055] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0056] In the various embodiments of the present invention, it should be understood that the sequence numbers of the following processes do not imply a specific order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. Furthermore, the terms "system" and "network" are often used interchangeably herein.

[0057] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0058] In this embodiment of the invention, the user terminal may be a mobile phone (or cell phone), or other devices capable of sending or receiving wireless signals, including user equipment, personal digital assistant (PDA), wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop (WLL) station, CPE (Customer Premise Equipment) or mobile smart hotspot capable of converting mobile signals into WiFi signals, smart home appliance, or other devices that can spontaneously communicate with mobile communication networks without human operation.

[0059] Typically, consumer-facing applications (such as virtual humans and game modeling) require low-cost, high-fidelity 3D face reconstruction methods. However, existing 3D face model reconstruction methods are time-consuming, slow, and costly. Therefore, with the booming development of 5G user applications, there is an urgent need for a low-cost, fast, and easy-to-operate 3D head model reconstruction solution.

[0060] Please see Figure 1 , Figure 1 This is a flowchart illustrating a three-dimensional head model reconstruction method provided in an embodiment of the present invention. The method includes the following steps:

[0061] Step 11: Obtain target video data, which is a video captured around the head of a person.

[0062] Here, a camera can be used for shooting, which can be a mobile phone camera, a tablet camera, or any other camera capable of shooting video. For example, a mobile phone can be used to shoot around the head of a stationary person to obtain target video data (e.g., RGB video).

[0063] Step 12: For the video frame images in the target video data, obtain the 3D point cloud and camera pose information corresponding to the video frame images.

[0064] It should be noted that the target video data can be viewed as a series of continuous images, with each video frame being any single frame within that series. Understandably, since the target video data is filmed around a person's head, it can provide RGB images from multiple perspectives.

[0065] Step 13: Based on the initial 3D head model, determine the position information of the 3D head model according to the 3D point cloud and the camera pose information. The position information includes the position information of facial feature points.

[0066] It should be noted that the initial 3D head model here can be the average 3D head model of a human.

[0067] Step 14: Adjust the three-dimensional head model according to the three-dimensional point cloud, the camera pose information, and the position information of the three-dimensional head model to obtain the target three-dimensional head model.

[0068] Here, based on an average human 3D head model, adjustments are made to it according to information such as 3D point clouds obtained from the target video data, so that the shape and other features of the target 3D head model are closer to the human head in the target video data.

[0069] It should be noted that existing face reconstruction solutions are slow and extremely costly. In contrast, this invention, through the steps described above, reconstructs a complete 3D head based solely on video captured by a regular camera (e.g., a smartphone camera), without requiring expensive equipment or facilities, thus reducing the cost of 3D head modeling.

[0070] Step 15: Based on the camera pose information and the target texture generation model, perform texture rendering on the target 3D head model.

[0071] In this step, a target texture generation model is used to perform texture rendering on the target 3D head model, which effectively improves the realism of face rendering.

[0072] In this embodiment, a video is captured around the person's head. By analyzing and processing the images in the video, information such as the corresponding 3D point cloud and camera pose can be obtained. This information is then used to adjust the initial 3D head model to reconstruct the person's 3D head model. This solution does not require complex hardware or special environments, and the modeling process is relatively simple. It can reduce the modeling cost of the 3D head model and effectively improve the realism of face rendering. It can be applied to scenarios such as virtual humans (e.g., virtual customer service, virtual anchors), game production, and film production.

[0073] As an optional embodiment of the present invention, the three-dimensional head model reconstruction method provided by the present invention can be executed by a system, which mainly includes: a motion structure estimation module, a global pose estimation module, a shape and expression estimation module, and a texture estimation module. The target video data can be input into the system, and the system can output a three-dimensional head model with realistic texture, i.e., the target three-dimensional head model.

[0074] It should be noted that the four modules in this system can be understood as four models with data interaction relationships (for example, the output data of the motion structure estimation module can be used as part or all of the input data of the global pose estimation module). Each model can correspond to a learning algorithm, and the models can be trained separately until they converge.

[0075] As an optional embodiment of the present invention, step 12 may specifically include:

[0076] Step 1201: Use the Accelerated Robust Feature SURF algorithm to detect the feature points of the video frame image.

[0077] In this step, the SURF algorithm can be used to detect feature points of video frame images in the target video data.

[0078] Step 1202: Use the corner tracking KLT algorithm to correct the feature points of the video frame image.

[0079] Specifically, step 1202 may include:

[0080] Using the KLT algorithm, a first feature point that matches the video frame image and the second target image is obtained, wherein the second target image is the next frame image of the video frame image; based on the feature points in the second target image other than the first feature point, the feature points of the video frame image are corrected.

[0081] For example, newly emerging feature points in the second target image (i.e., feature points in the second target image other than the first feature points) can be added to the feature points of the video frame image to correct the feature points of the video frame image.

[0082] Step 1203: Based on the feature point matching relationship between two adjacent frames, estimate the three-dimensional position of the feature points and the three-dimensional camera pose information corresponding to the video frame image.

[0083] In this step, since there is a certain motion transformation relationship between two adjacent frames and they have many identical feature points, the triangular projection relationship can be used to roughly estimate the 3D camera pose motion and the 3D position of feature points between frames, so as to obtain the 3D position of feature points and 3D camera pose information corresponding to the video frame image.

[0084] Step 1204: Based on the three-dimensional position of the feature points and the three-dimensional camera pose information, obtain the three-dimensional point cloud and the camera pose information corresponding to the video frame image.

[0085] In this embodiment, feature matching is used to calculate the camera pose (i.e., camera pose information) and the 3D point cloud of the entire scene in each frame. This ensures that the 3D point cloud, after being projected onto the camera, can match the coordinates of the 2D feature points detected in the image (i.e., the feature points in the video frame image). In other words, based on a rough estimate of the 3D camera pose and the 3D positions of the feature points, joint optimization is performed to ensure that the coordinates of all 3D feature points (i.e., the 3D positions of the feature points) in the camera projection satisfy the positions of the 2D feature points in the preceding SURF algorithm (i.e., the positions of the feature points detected from the video frame image, denoted by v). i express).

[0086] It should be noted that the specific process in step 12 above can be executed by the motion structure estimation module in the system. The main function of this motion structure estimation module is to calculate the camera position (i.e., 3D camera pose information) and to make a rough estimate of the geometric structure of the scene (i.e., 3D point cloud). The input to this motion structure estimation module is RGB video frames (i.e., video frame images, denoted by I), and the output of this module is the 3D point cloud and the camera pose of each frame (i.e., the camera pose information corresponding to the video frame image, denoted by P). t (This is represented as an example). The motion structure estimation module can be optimized according to the following formula:

[0087]

[0088] Where L1 represents the optimization function of the motion structure estimation module; P t Indicates the 3D camera pose; t represents the current frame number; w i Indicates the 3D position of the feature point; i represents the feature point index; v i V represents the position of a two-dimensional feature point; t T represents the set of two-dimensional feature points appearing in frame t; λ represents the coefficient; T i I represents the set of all frame indices containing feature point i; t I is the vector representation of the video frame image corresponding to frame t; τ P is the vector representation of the image of the next frame after frame t; τ This represents the 3D camera pose corresponding to the next frame image of frame t.

[0089] In this embodiment of the invention, the coordinates of all three-dimensional feature points (i.e., the three-dimensional positions of feature points) can be matched with the coordinates of two-dimensional feature points detected in the image (i.e., the video frame image) after being projected by the camera, thereby improving accuracy.

[0090] It should be noted that currently common industry solutions typically only use feature point coordinate matching. However, due to limitations in the accuracy of feature point detection and tracking algorithms in videos, this can easily lead to large errors in camera pose calculation. The joint optimization process in this embodiment of the invention, however, utilizes both the 3D camera pose and the 3D position of feature points for joint optimization, thus improving accuracy.

[0091] Optionally, step 13 may specifically include:

[0092] Step 1301: Based on the three-dimensional point cloud and the camera pose information, determine the two-dimensional face feature points obtained by projecting the face feature points on the three-dimensional head model onto the camera.

[0093] Here, the definition of the location of facial feature points follows the industry-standard definition of 68 facial key points.

[0094] Step 1302: Match the two-dimensional facial feature points with the target two-dimensional facial feature points to obtain the position information of the three-dimensional head model; wherein, the target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image.

[0095] In this step, by calculating the position of the 3D head model (i.e., the position information of the 3D head model), the 68 facial feature points corresponding to the 3D head model can be matched with the 68 2D facial feature points detected from the video frame image after being projected by the camera.

[0096] In this embodiment, a 3D model of an average human head is used, whose shape and expression can be modified using two one-dimensional vectors respectively. Based on the initial 3D head model, a series of adjustments are made according to the target video data (such as expression and shape adjustments, skin problem rendering, etc.) to finally obtain the target 3D head model.

[0097] The steps in this embodiment can be executed by the global pose estimation module in the system. The main function of this global pose estimation module is to calculate the position of the 3D head model. The input to this global pose estimation module is RGB video frames and the camera pose of each frame. The output of this global pose estimation module is the position information of the 3D head model (e.g., a 4x4 matrix, denoted by T). This global pose estimation module can be optimized according to the following formula:

[0098]

[0099] Where L2 represents the optimization function of the global pose estimation module; T represents the position of the 3D head model; t is the current frame number; P tL(F) represents the 3D camera pose; L(F) represents the 68 facial feature points corresponding to the 3D head model; F represents the 3D face model; I t Let t be the vector representation of the video frame image corresponding to frame t.

[0100] It should be noted that existing face reconstruction solutions mainly focus on reconstructing the facial region, neglecting the entire head (such as hair). Since most applications (such as virtual anchors and game avatars) require complete models, this limits the practical application scenarios. In contrast, this invention uses multi-view geometric projection and 3D point cloud constraints to recover the geometry of the entire head.

[0101] Optionally, step 14 may specifically include:

[0102] Step 1401: Based on the 3D point cloud, the camera pose information, and the position information of the 3D head model, adjust the shape and expression of the 3D head model to obtain the target shape and target expression of the target 3D head model.

[0103] In this step, 3D point clouds and facial feature points can be used to make a fine estimate of the shape of the 3D head model. Specifically, the position information of the 3D head model can be determined first based on the facial feature points, and then the shape of the 3D head model can be adjusted based on the position information of the 3D head model.

[0104] Step 1402: Using a geometric optimization algorithm, the vertices of the 3D head model and the 3D point cloud are matched, and the position information of the 3D head model is adjusted to minimize the point-to-surface distance between the vertices of the 3D head model and the matched 3D point cloud, thereby obtaining the target position information of the target 3D head model.

[0105] It should be noted that the order in which steps 1401 and 1402 are performed is not limited. For example, while adjusting and optimizing the shape and expression of the 3D head model, high-precision matching of the vertices of the 3D head model and the 3D point cloud can be performed, that is, minimizing the point-to-surface distance between the vertices of the 3D head model and the matched 3D point cloud.

[0106] It should be noted that the steps in this embodiment can be performed by the shape and expression estimation module in the system. The main function of this shape and expression estimation module is to accurately calculate the shape, expression, and position of the 3D head (i.e., the positional information of the 3D head model). In other words, after the global pose estimation module roughly aligns the 3D face model and facial feature points, this shape and expression estimation module can adjust the shape and expression of the 3D head model to more accurately match the feature points. The input to this shape and expression estimation module is: RGB video frames, camera pose (using P...).t (representation), 3D point cloud (using w) i The shape and expression estimation module outputs the following: the shape of the 3D head model (denoted by α), the expression of the 3D head model (denoted by β), and the optimized position of the 3D head model. The optimization of this module can be performed using the following formula:

[0107]

[0108] Where L3 represents the optimization function of the shape and expression estimation module; α represents shape; β represents expression; T represents the position of the adjusted 3D head model; t is the current frame number; P t Indicates the pose of a 3D camera; w i Indicates the 3D position of the feature point; i represents the feature point index; v i V represents the position of a two-dimensional feature point; t φ(F) represents the set of two-dimensional feature points appearing in frame t; λ represents the coefficient; φ(F) represents the set of two-dimensional feature points appearing in frame t. α,β ) represents the point in the 3D point cloud closest to the vertex of the 3D head model (denoted as "point one"); F α,β Represents the estimated value of a 3D point cloud; I t n is the vector representation of the video frame image corresponding to frame t; i Let be the plane normal vector estimated at point 1.

[0109] The algorithm in the above formula can accurately match 3D feature points and 2D feature points, as well as the shape of a 3D head model and the shape of a 3D point cloud.

[0110] It should be noted that in currently commonly used methods, the 68 facial feature points are too sparse and all concentrated on the face, thus ignoring other parts of the head, making it difficult to match the top of the head, ears, and other areas. However, in this embodiment of the invention, by applying 3D point clouds as additional geometric constraints, the geometry of the top of the head, ears, and the entire head can be matched, resulting in a more complete 3D head model with realistic effects, improved algorithm robustness, and enhanced accuracy and realism of the face model reconstruction.

[0111] Optionally, the step of performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model includes:

[0112] Generate latent space vectors based on the 3D point cloud;

[0113] The latent space vector is input into the target texture generation model to obtain the first texture image;

[0114] Based on the camera pose information, the target texture image after the first texture image is projected by the camera is obtained;

[0115] The target texture image is used to perform texture rendering on the target 3D head model.

[0116] It should be noted that existing technologies suffer from poor facial texture detail quality, resulting in low visual realism. However, in this embodiment of the invention, the texture of the entire head can be directly estimated based on RGB video frames, achieving a realistic rendering result.

[0117] It should also be noted that existing technologies for reconstructing 3D face models based on computer vision can only recover the geometric features of the facial region, but struggle to recover realistic facial textures. In contrast, this invention utilizes a trained target texture generation model to render the texture of a target 3D head model, thereby obtaining a 3D head model with realistic facial textures.

[0118] The steps in this embodiment can be executed by the texture estimation module in the system. The main function of this texture estimation module is to calculate the texture of the 3D head model, which can improve the realism of the target 3D head model's appearance. That is, after the system's motion structure estimation module, global pose estimation module, and shape and expression estimation module have calculated and processed the data, the geometry of the target 3D head model can be determined. The texture estimation module can then be used to estimate the head texture map for rendering, thereby performing texture rendering on the target 3D head model. The inputs to the texture estimation module are: RGB video frames, camera pose, the shape of the 3D head model, the expression of the 3D head model, and the adjusted position of the 3D head model. The output of the texture estimation module is: the target texture image of the 3D head model.

[0119] In this texture estimation module, the input vector (i.e., the latent space vector) of the pre-trained texture generation network is optimized so that the output texture of the texture generation network matches the video frame image (i.e., the multi-view RGB image captured by the camera) in the target video data, which is the image rendered after projection based on the camera pose (i.e., the target texture image). This texture estimation module can be optimized according to the following formula:

[0120]

[0121] Where L4 represents the optimization function of the texture estimation module; Z represents the latent space vector; t is the current frame number; and G represents the target texture generation model. This represents the image obtained after rendering the target 3D head model using z as input to G; P tα represents the 3D camera pose; β represents the shape; T represents the expression; and I represents the adjusted position of the 3D head model. t Let t be the vector representation of the video frame image corresponding to frame t.

[0122] It should be noted that currently, commonly used methods for facial texture reconstruction typically involve using sophisticated measurement equipment to precisely acquire facial images from different angles and then fusing these images. However, this approach requires precisely controlled experimental settings and involves a large amount of computation. Furthermore, inaccurate camera pose calibration can lead to ghosting issues in the fused facial texture.

[0123] To address this deficiency in the existing technology, the embodiments of the present invention employ a method for facial texture reconstruction based on a pre-trained network. This method can still effectively match the captured facial video even when the pose estimation of the device (e.g., a mobile phone) camera is inaccurate, thereby ensuring that the generated facial texture does not contain ghosting or other undesirable effects.

[0124] Optionally, before performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model, the method further includes:

[0125] Obtain a training dataset, which includes face images;

[0126] The texture generation network model is trained using the training dataset to obtain the target texture generation model.

[0127] Optionally, training the texture generation network model using the training dataset includes:

[0128] (i) Generate a latent space vector that satisfies a Gaussian distribution based on the three-dimensional point cloud corresponding to the first face image in the training dataset.

[0129] For example, the training dataset could consist of 90,000 face images (photos) of a preset size, such as 1024x1024 pixels. This would generate a latent space vector with a Gaussian distribution and a length of 512 pixels.

[0130] (ii) Input the latent space vector into the generator of the texture generation network model to obtain a two-dimensional image.

[0131] In this step, the generated random latent space vector can be used as the input to the generator network, and the output is a 1024x1024 pixel two-dimensional image.

[0132] (iii) The two-dimensional image is used as a texture and rendered on a three-dimensional face model to obtain a second face image.

[0133] It should be noted that the 3D face model in this step can be randomly selected, and its shape and expression do not need to be specially set.

[0134] In this step, the 1024x1024 two-dimensional image generated in the previous step is used as a texture and rendered on a three-dimensional face (i.e., a three-dimensional face model) with random shapes and expressions to obtain a rendered 1024x1024 face image, i.e., the second face image. It can be understood that the generation of this second face image is related to the first face image.

[0135] (iv) Input the first face image and the second face image into the discriminator of the texture generation network model respectively to obtain the discrimination result;

[0136] (v) Based on the discrimination result, modify the parameters of the generator and the discriminator.

[0137] In other words, the real face images (i.e., the first face image) in the training dataset and the face images generated by the generator (i.e., the second face image) are respectively input into the discriminator for discrimination, thereby optimizing the parameters of the generator and the discriminator.

[0138] (vi) Repeat the above steps until the parameters of the generator and the discriminator converge.

[0139] It should be noted that the above training process for the texture generation network model can generate complex face textures (e.g., 1024x1024 resolution 2D face textures) from extremely low-dimensional input parameters (e.g., a 1D vector of length 512), which can greatly improve the realism of the final output face and the robustness of the algorithm.

[0140] In this embodiment of the invention, a 3D head model of a person can be reconstructed from a video captured by a common device (such as a camera, mobile phone camera, etc.). Moreover, this 3D head modeling method based on multiple perspectives and 3D point clouds results in a relatively complete 3D head model, which can include details such as face, ears, hair, and skin spots. It can achieve better accuracy than existing face reconstruction schemes, enhance the accuracy and realism of face model reconstruction, produce realistic results, reduce the cost of 3D face modeling, and improve the robustness of the algorithm.

[0141] like Figure 2 As shown, this embodiment of the invention provides a three-dimensional head model reconstruction device 200, comprising:

[0142] Data acquisition module 201 is used to acquire target video data, which is a video shot around a person's head;

[0143] The first processing module 202 is used to obtain the three-dimensional point cloud and camera pose information corresponding to the video frame image in the target video data.

[0144] The second processing module 203 is used to determine the position information of the three-dimensional head model based on the initial three-dimensional head model, according to the three-dimensional point cloud and the camera pose information, wherein the position information includes the position information of facial feature points;

[0145] The third processing module 204 is used to adjust the three-dimensional head model according to the three-dimensional point cloud, the camera pose information and the position information of the three-dimensional head model to obtain the target three-dimensional head model.

[0146] The rendering module 205 is used to perform texture rendering on the target 3D head model based on the camera pose information and the target texture generation model.

[0147] In this embodiment, a video is taken around the person's head. By analyzing and processing the images in the video, information such as the corresponding 3D point cloud and camera pose can be obtained. This information can then be used to adjust the initial 3D head model and reconstruct the 3D head model of the person. This solution does not require complex hardware or special environments, and the modeling process is relatively simple, which can reduce the modeling cost of the 3D head model.

[0148] Optionally, the first processing module 202 includes:

[0149] The first processing submodule is used to detect the video frame image using the Accelerated Robust Feature SURF algorithm to obtain the feature points of the video frame image;

[0150] The feature correction submodule is used to correct the feature points of the video frame image using the corner tracking KLT algorithm;

[0151] The second processing submodule is used to estimate the three-dimensional position of the feature points and the three-dimensional camera pose information corresponding to the video frame image based on the feature point matching relationship between two adjacent frames;

[0152] The third processing submodule is used to obtain the three-dimensional point cloud and the camera pose information corresponding to the video frame image based on the three-dimensional position of the feature points and the three-dimensional camera pose information.

[0153] Optionally, the feature correction submodule includes:

[0154] The first processing unit is used to obtain a first feature point that matches the video frame image and the second target image using the KLT algorithm, wherein the second target image is the next frame image of the video frame image;

[0155] The feature correction unit is used to correct the feature points of the video frame image based on the feature points in the second target image other than the first feature point.

[0156] Optionally, the second processing module 203 includes:

[0157] The fourth processing submodule is used to determine the two-dimensional face feature points obtained by projecting the face feature points on the three-dimensional head model onto the camera, based on the three-dimensional point cloud and the camera pose information.

[0158] The fifth processing submodule is used to match the two-dimensional facial feature points with the target two-dimensional facial feature points to obtain the position information of the three-dimensional head model;

[0159] The target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image.

[0160] Optionally, the third processing module 204 includes:

[0161] The sixth processing submodule is used to adjust the shape and expression of the three-dimensional head model based on the three-dimensional point cloud, the camera pose information and the position information of the three-dimensional head model, so as to obtain the target shape and target expression of the target three-dimensional head model.

[0162] The seventh processing submodule is used to use a geometric optimization algorithm to match the vertices of the 3D head model with the 3D point cloud, adjust the position information of the 3D head model, so that the point-to-surface distance between the vertices of the 3D head model and the matched 3D point cloud is minimized, and obtain the target position information of the target 3D head model.

[0163] Optionally, the rendering module includes:

[0164] The eighth processing submodule is used to generate latent space vectors based on the 3D point cloud.

[0165] The ninth processing submodule is used to input the latent space vector into the target texture generation model to obtain the first texture image;

[0166] The tenth processing submodule is used to obtain the target texture image after the first texture image is projected by the camera based on the camera pose information;

[0167] The rendering submodule is used to perform texture rendering on the target 3D head model using the target texture image.

[0168] Optionally, the device 200 further includes:

[0169] The first training module is used to acquire a training dataset, which includes face images;

[0170] The second training module is used to train the texture generation network model using the training dataset to obtain the target texture generation model.

[0171] Optionally, the second training module includes:

[0172] The first training submodule is used to generate a latent space vector that satisfies a Gaussian distribution based on the three-dimensional point cloud corresponding to the first face image in the training dataset.

[0173] The second training submodule is used to input the latent space vector into the generator of the texture generation network model to obtain a two-dimensional image;

[0174] The third training submodule is used to render the two-dimensional image as a texture on a three-dimensional face model to obtain a second face image;

[0175] The fourth training submodule is used to input the first face image and the second face image into the discriminator of the texture generation network model to obtain the discrimination result;

[0176] The fifth training submodule is used to correct the parameters of the generator and the discriminator based on the discrimination result;

[0177] The sixth processing submodule is used to repeatedly execute the above steps until the parameters of the generator and the discriminator converge.

[0178] In this embodiment of the invention, a 3D head model of a person can be reconstructed from a video captured by a common device (such as a camera, mobile phone camera, etc.). Moreover, this 3D head modeling method based on multiple perspectives and 3D point clouds results in a relatively complete 3D head model that can include details such as face, ears, hair, and skin spots. It can achieve better accuracy than existing face reconstruction schemes, with realistic effects, reduce the cost of 3D face modeling, improve algorithm robustness, and enhance the accuracy and realism of face model reconstruction.

[0179] like Figure 3 As shown, the terminal of this embodiment of the invention includes: a processor 300; and a memory 320 connected to the processor 300 via a bus interface. The memory 320 is used to store programs and data used by the processor 300 when performing operations, and the processor 300 calls and executes the programs and data stored in the memory 320.

[0180] Processor 300 reads the program from memory 320 and executes the following procedures:

[0181] Acquire target video data, wherein the target video data is a video captured around the head of a person;

[0182] For video frame images in the target video data, obtain the 3D point cloud and camera pose information corresponding to the video frame images;

[0183] Based on the initial 3D head model, the position information of the 3D head model is determined according to the 3D point cloud and the camera pose information, and the position information includes the position information of facial feature points;

[0184] Based on the 3D point cloud, the camera pose information, and the position information of the 3D head model, the 3D head model is adjusted to obtain the target 3D head model.

[0185] Based on the camera pose information and the target texture generation model, the target 3D head model is texture rendered.

[0186] Transceiver 310 is used to receive and send data under the control of processor 300.

[0187] Among them, Figure 3 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 300 and memory represented by memory 320 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 310 can be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 330 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0188] The processor 300 is responsible for managing the bus architecture and general processing, while the memory 320 can store the data used by the processor 300 when performing operations.

[0189] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0190] The video frame image is detected using the Accelerated Robust Feature SURF algorithm to obtain the feature points of the video frame image;

[0191] The feature points of the video frame image are corrected using the corner tracking KLT algorithm;

[0192] Based on the feature point matching relationship between two adjacent frames, estimate the three-dimensional position of the feature points and the three-dimensional camera pose information corresponding to the video frame image;

[0193] Based on the 3D position of the feature points and the 3D camera pose information, the 3D point cloud and the camera pose information corresponding to the video frame image are obtained.

[0194] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0195] Using the KLT algorithm, a first feature point that matches the video frame image and the second target image is obtained, wherein the second target image is the next frame image of the video frame image;

[0196] Based on the feature points in the second target image other than the first feature point, the feature points of the video frame image are corrected.

[0197] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0198] Based on the three-dimensional point cloud and the camera pose information, determine the two-dimensional face feature points obtained by projecting the face feature points on the three-dimensional head model onto the camera.

[0199] The two-dimensional facial feature points are matched with the target two-dimensional facial feature points to obtain the position information of the three-dimensional head model;

[0200] The target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image.

[0201] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0202] Based on the 3D point cloud, the camera pose information, and the position information of the 3D head model, the shape and expression of the 3D head model are adjusted to obtain the target shape and target expression of the target 3D head model.

[0203] Using a geometric optimization algorithm, the vertices of the 3D head model and the 3D point cloud are matched, and the position information of the 3D head model is adjusted to minimize the point-to-surface distance between the vertices of the 3D head model and the matched 3D point cloud, thereby obtaining the target position information of the target 3D head model.

[0204] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0205] Generate latent space vectors based on the 3D point cloud;

[0206] The latent space vector is input into the target texture generation model to obtain the first texture image;

[0207] Based on the camera pose information, the target texture image after the first texture image is projected by the camera is obtained;

[0208] The target texture image is used to perform texture rendering on the target 3D head model.

[0209] Optionally, the processor 300 is further configured to read the computer program and perform the following steps:

[0210] Obtain a training dataset, which includes face images;

[0211] The texture generation network model is trained using the training dataset to obtain the target texture generation model.

[0212] Optionally, training the texture generation network model using the training dataset includes:

[0213] Based on the 3D point cloud corresponding to the first face image in the training dataset, a latent space vector satisfying a Gaussian distribution is generated.

[0214] The latent space vector is input into the generator of the texture generation network model to obtain a two-dimensional image;

[0215] The two-dimensional image is used as a texture and rendered on a three-dimensional face model to obtain a second face image;

[0216] The first face image and the second face image are respectively input into the discriminator of the texture generation network model to obtain the discrimination result;

[0217] Based on the discrimination result, the parameters of the generator and the discriminator are adjusted;

[0218] Repeat the above steps until the parameters of the generator and the discriminator converge.

[0219] The terminal provided in this embodiment of the invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0220] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a computer program instructing the relevant hardware to implement them. The computer program includes instructions to perform some or all of the steps of the above methods; and the computer program can be stored in a readable storage medium, which can be any form of storage medium.

[0221] Furthermore, specific embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the aforementioned three-dimensional head model reconstruction method. This achieves the same technical effect, and to avoid repetition, will not be described further here.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0223] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0224] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The above describes the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also within the scope of protection of the present invention.

Claims

1. A method for reconstructing a three-dimensional head model, characterized in that, include: Acquire target video data, wherein the target video data is a video captured around the head of a person; For video frame images in the target video data, obtain the 3D point cloud and camera pose information corresponding to the video frame images; Based on the initial 3D head model, the position information of the target 3D head model is determined according to the 3D point cloud and the camera pose information, and the position information includes the position information of facial feature points; Based on the 3D point cloud, the camera pose information, and the position information of the target 3D head model, the initial 3D head model is adjusted to obtain the target 3D head model; Based on the camera pose information and the target texture generation model, the target 3D head model is texture rendered. The step of determining the position information of the target 3D head model based on the 3D point cloud and the camera pose information includes: Based on the three-dimensional point cloud and the camera pose information, determine the two-dimensional face feature points obtained by projecting the face feature points on the target three-dimensional head model onto the camera. The two-dimensional facial feature points are matched with the target two-dimensional facial feature points to obtain the position information of the target three-dimensional head model; Wherein, the target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image; The step of adjusting the initial 3D head model based on the 3D point cloud, the camera pose information, and the position information of the target 3D head model to obtain the target 3D head model includes: Based on the 3D point cloud, the camera pose information, and the position information of the target 3D head model, the shape and expression of the initial 3D head model are adjusted to obtain the target shape and target expression of the target 3D head model; The step of performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model includes: Based on the three-dimensional point cloud, generate a latent space vector; The latent space vector is input into the target texture generation model to obtain the first texture image; Based on the camera pose information, the target texture image after the first texture image is projected by the camera is obtained; The target texture image is used to perform texture rendering on the target 3D head model.

2. The method according to claim 1, characterized in that, The step of obtaining the 3D point cloud and camera pose information corresponding to the video frame images in the target video data includes: The video frame image is detected using the Accelerated Robust Feature SURF algorithm to obtain the feature points of the video frame image; The feature points of the video frame image are corrected using the corner tracking KLT algorithm; Based on the feature point matching relationship between two adjacent frames, estimate the three-dimensional position of the feature points and the three-dimensional camera pose information corresponding to the video frame image; Based on the 3D position of the feature points and the 3D camera pose information, the 3D point cloud and the camera pose information corresponding to the video frame image are obtained.

3. The method according to claim 2, characterized in that, The method of using the corner tracking KLT algorithm to correct feature points of the video frame image includes: Using the KLT algorithm, a first feature point that matches the video frame image and the second target image is obtained, wherein the second target image is the next frame image of the video frame image; Based on the feature points in the second target image other than the first feature point, the feature points of the video frame image are corrected.

4. The method according to claim 1, characterized in that, Before performing texture rendering on the target 3D head model based on the camera pose information and the target texture generation model, the method further includes: Obtain a training dataset, which includes face images; The texture generation network model is trained using the training dataset to obtain the target texture generation model.

5. The method according to claim 4, characterized in that, The step of training the texture generation network model using the training dataset includes: Based on the 3D point cloud corresponding to the first face image in the training dataset, a latent space vector satisfying a Gaussian distribution is generated. The latent space vector is input into the generator of the texture generation network model to obtain a two-dimensional image; The two-dimensional image is used as a texture and rendered on a three-dimensional face model to obtain a second face image; The first face image and the second face image are respectively input into the discriminator of the texture generation network model to obtain the discrimination result; Based on the discrimination result, the parameters of the generator and the discriminator are adjusted; Repeat the above steps until the parameters of the generator and the discriminator converge.

6. A three-dimensional head model reconstruction device, characterized in that, include: The data acquisition module is used to acquire target video data, which is a video shot around a person's head; The first processing module is used to obtain the three-dimensional point cloud and camera pose information corresponding to the video frame image in the target video data. The second processing module is used to determine the position information of the target three-dimensional head model based on the initial three-dimensional head model, according to the three-dimensional point cloud and the camera pose information, wherein the position information includes the position information of facial feature points; The third processing module is used to adjust the initial three-dimensional head model according to the three-dimensional point cloud, the camera pose information and the position information of the target three-dimensional head model to obtain the target three-dimensional head model. The rendering module is used to perform texture rendering on the target 3D head model based on the camera pose information and the target texture generation model. The second processing module includes: The fourth processing submodule is used to determine the two-dimensional face feature points obtained by projecting the face feature points on the target three-dimensional head model onto the camera, based on the three-dimensional point cloud and the camera pose information. The fifth processing submodule is used to match the two-dimensional facial feature points with the target two-dimensional facial feature points to obtain the position information of the target three-dimensional head model; Wherein, the target two-dimensional facial feature points are two-dimensional facial feature points detected from the video frame image; The third processing module includes: The sixth processing submodule is used to adjust the shape and expression of the initial three-dimensional head model based on the three-dimensional point cloud, the camera pose information and the position information of the target three-dimensional head model, so as to obtain the target shape and target expression of the target three-dimensional head model. The rendering module includes: The eighth processing submodule is used to generate latent space vectors based on the three-dimensional point cloud; The ninth processing submodule is used to input the latent space vector into the target texture generation model to obtain the first texture image; The tenth processing submodule is used to obtain the target texture image after the first texture image is projected by the camera based on the camera pose information; The rendering submodule is used to perform texture rendering on the target 3D head model using the target texture image.

7. A terminal, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the three-dimensional head model reconstruction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the three-dimensional head model reconstruction method as described in any one of claims 1 to 5.

Images