Method for optimizing three-dimensional human body reconstruction model, method for three-dimensional human body reconstruction, optimization device, reconstruction device, equipment, storage medium and product

Abstract

The application discloses a three-dimensional human body reconstruction model optimization method, a three-dimensional human body reconstruction method, an optimization device, a reconstruction device, equipment, a storage medium and a product, and relates to the technical field of three-dimensional reconstruction. The method comprises the following steps: constructing a prediction parameter set and a reference parameter set by respectively passing an original portrait image and a reference portrait image through an initial three-dimensional human body reconstruction model; generating a human body three-dimensional grid by a expressive human body model according to the prediction parameter set and the reference parameter set; constructing a reconstruction image by the initial three-dimensional human body reconstruction model based on the human body three-dimensional grid and the reference portrait image; and optimizing the initial three-dimensional human body reconstruction model according to the original portrait image, the prediction parameter set, the human body three-dimensional grid and the reconstruction image to obtain a target three-dimensional human body reconstruction model. The application can realize efficient and high-fidelity three-dimensional human body reconstruction.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to optimization methods for three-dimensional human body reconstruction models, three-dimensional human body reconstruction methods, optimization devices, reconstruction devices, equipment, storage media, and products. Background Technology

[0002] Three-dimensional human body reconstruction is one of the core tasks in the field of computer vision and graphics, and it has broad application prospects in virtual reality, film and television production, human-computer interaction and other fields.

[0003] Currently, mainstream 3D human reconstruction technologies are generally based on parametric human models. These methods reconstruct the human body in 3D by regressing low-dimensional parameters (such as pose and shape) from the input image. However, existing 3D human reconstruction technologies focus on regressing the overall shape of the human body, while their ability to model local details geometrically is significantly insufficient. This leads to serious distortion in the final 3D human reconstruction results, thus affecting their practicality and realism in high-end applications such as film and television special effects and digital humans.

[0004] The above content is only used to assist in understanding the technical solution of this application, and therefore may contain information that does not constitute prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an optimization method, a three-dimensional human body reconstruction method, an optimization device, a reconstruction device, equipment, storage medium, and product for a three-dimensional human body reconstruction model, aiming to solve problems such as low reconstruction accuracy of the face and hand regions, poor local pixel alignment, and lack of unified parametric modeling of the human body in existing three-dimensional human body reconstruction.

[0006] To achieve the above objectives, this application proposes an optimization method for a three-dimensional human body reconstruction model, the optimization method comprising:

[0007] The original human portrait image and the reference human portrait image are used to construct prediction parameter sets and reference parameter sets respectively through the initial 3D human body reconstruction model;

[0008] Based on the predicted parameter set and the reference parameter set, a three-dimensional human body mesh is generated using an expressive human body model.

[0009] Based on the human body 3D mesh and the reference human image, a reconstructed image is constructed using the initial 3D human body reconstruction model;

[0010] The initial 3D human body reconstruction model is optimized based on the original human portrait image, the prediction parameter set, the 3D human body mesh, and the reconstructed image to obtain the target 3D human body reconstruction model.

[0011] In one embodiment, the initial 3D human body reconstruction model includes a visual estimator, which comprises a feature extraction layer, a multi-head prediction layer, and a parameter decoding layer; wherein, the step of constructing a prediction parameter set from the original human portrait image using the initial 3D human body reconstruction model includes:

[0012] The original portrait image is encoded through the feature extraction layer to obtain semantic encoding;

[0013] The semantic codes are integrated by the multi-head prediction layer to obtain the target semantic code;

[0014] The target semantic encoding is decoded by the parameter decoding layer to construct a prediction parameter set.

[0015] In one embodiment, the parameter decoding layer includes a first output header, a second output header, and a third output header; wherein, the step of decoding the target semantic code through the parameter decoding layer to construct a prediction parameter set includes:

[0016] The target semantic encoding is decoded using the first output head to obtain the original body parameters;

[0017] The target semantic code is decoded using the second output header to obtain the original header parameters;

[0018] The target semantic code is decoded by the third output head to obtain the original camera parameters;

[0019] Based on the original body parameters, the original head parameters, and the original camera parameters, a set of prediction parameters is constructed.

[0020] In one embodiment, the step of generating a three-dimensional human body mesh based on the predicted parameter set and the reference parameter set using an expressive human body model includes:

[0021] A combined parameter set is determined based on the predicted parameter set and the reference parameter set;

[0022] The combined parameter set is used to generate a three-dimensional human body mesh through the expressive human body model.

[0023] In one embodiment, the step of determining the combined parameter set based on the predicted parameter set and the reference parameter set includes:

[0024] Obtain the original body pose parameters, original head pose parameters, original facial expression parameters, and original camera parameters from the prediction parameter set;

[0025] Obtain the reference body shape parameters and reference head shape parameters from the reference parameter set;

[0026] A set of combined parameters is determined based on the original body posture parameters, the original head posture parameters, the original facial expression parameters, the original camera parameters, the reference body shape parameters, and the reference head shape parameters.

[0027] In one embodiment, the expressive human body model includes a 3D face reconstruction model and a human body model; wherein, the step of generating a 3D human body mesh by combining the parameter set through the expressive human body model includes:

[0028] Based on the original head pose parameters, original facial expression parameters, and reference head shape parameters in the combined parameter set, a three-dimensional head mesh is obtained through the three-dimensional face reconstruction model. The three-dimensional head mesh includes a three-dimensional head mesh point cloud and the connection relationship of the three-dimensional head mesh point cloud.

[0029] Based on the original body posture parameters and reference body shape parameters in the combined parameter set, a three-dimensional body mesh is obtained through the human body model. The three-dimensional body mesh includes a three-dimensional body mesh point cloud and the connection relationship of the three-dimensional body mesh point cloud.

[0030] The head 3D mesh and the body 3D mesh are fused to generate a human body 3D mesh.

[0031] In one embodiment, the initial 3D human body reconstruction model includes a neural renderer, which includes an image representation extractor, an action mapping module, a Gaussian decoder, and a Gaussian renderer; wherein, the step of constructing a reconstructed image based on the 3D human body mesh and the reference human portrait image using the initial 3D human body reconstruction model includes:

[0032] The deep neural representation of the reference portrait image is extracted using the image representation extractor.

[0033] The motion mapping module processes the human body 3D mesh and the deep neural representation to obtain the target deep neural representation.

[0034] The Gaussian attribute set is obtained by decoding the target deep neural representation using the Gaussian decoder.

[0035] The Gaussian property set is rendered into a reconstructed image using the Gaussian renderer.

[0036] In one embodiment, the initial 3D human body reconstruction model includes a visual estimator and a neural renderer; wherein, the step of optimizing the initial 3D human body reconstruction model based on the original human portrait image, the prediction parameter set, the 3D human body mesh, and the reconstructed image to obtain the target 3D human body reconstruction model includes:

[0037] The visual estimator is initially optimized based on the true parameter set, the prediction parameter set, the original human portrait image, and the human body 3D mesh to obtain the initial visual estimator;

[0038] The initial visual estimator and the neural renderer are optimized based on the original human portrait image and the reconstructed image to obtain the target visual estimator and the target neural renderer for the target 3D human body reconstruction model.

[0039] In one embodiment, the step of performing initial optimization of the visual estimator based on the truth parameter set, the prediction parameter set, the original portrait image, and the human body 3D mesh to obtain an initial visual estimator includes:

[0040] Obtain the set of truth parameters;

[0041] The parametric regression loss is determined based on the predicted parameter set and the true parameter set;

[0042] The key point loss is determined based on the original human portrait image, the original camera parameters, and the human body 3D mesh.

[0043] The visual estimator is initially optimized based on the parameter regression loss and the keypoint loss to obtain the initial visual estimator.

[0044] In one embodiment, the step of determining the parameter regression loss based on the predicted parameter set and the true parameter set includes:

[0045] Calculate the first loss value between the true body parameters in the true parameter set and the original body parameters in the predicted parameter set;

[0046] Calculate a second loss value between the truth head parameters in the truth parameter set and the original head parameters in the prediction parameter set;

[0047] Based on the first loss value and the second loss value, the parameter regression loss is determined.

[0048] In one embodiment, the step of determining the keypoint loss based on the original portrait image and the human body 3D mesh includes:

[0049] Key point detection is performed on the original human portrait image to obtain three-dimensional human key points;

[0050] The three-dimensional human body mesh is sampled to obtain predicted three-dimensional human body key points;

[0051] Two-dimensional human key point detection is performed on the original human portrait image to obtain two-dimensional human key points;

[0052] Based on the original camera parameters of the predicted parameter set and the predicted 3D human key points, the predicted 2D human key points are obtained.

[0053] Calculate the third loss value between the two-dimensional human key points and the predicted two-dimensional human key points;

[0054] Calculate the fourth loss value between the three-dimensional human key points and the predicted three-dimensional human key points;

[0055] Based on the third and fourth loss values, the critical point loss is determined.

[0056] In one embodiment, the step of optimizing the initial visual estimator and the neural renderer based on the original human portrait image and the reconstructed image to obtain a target visual estimator and a target neural renderer for the target 3D human body reconstruction model includes:

[0057] Calculate the fifth loss value between the original portrait image and the reconstructed image;

[0058] Multi-scale features are extracted from the reconstructed image and the original portrait image respectively to obtain reconstructed multi-scale features and original multi-scale features;

[0059] Calculate the sixth loss value of the reconstructed multi-scale features and the original multi-scale features;

[0060] The initial visual estimator and the neural renderer are optimized based on the fifth loss value and the sixth loss value, respectively, to obtain the target visual estimator and the target neural renderer.

[0061] Based on the target visual estimator and target neural renderer, a target 3D human reconstruction model is obtained.

[0062] Furthermore, to achieve the above objectives, this application also proposes a three-dimensional human body reconstruction method, which determines the target three-dimensional human body reconstruction model and an expressive human body model based on the aforementioned optimization method for the target three-dimensional human body reconstruction model. The three-dimensional human body reconstruction method includes:

[0063] The target human image is used to construct a target 3D human body model to obtain a target parameter set;

[0064] The target parameter set is used to obtain a three-dimensional mesh of the target human body through the expressive human body model;

[0065] Based on the target human body 3D mesh and the target human image, a target 3D human body reconstruction image is generated using the target 3D human body reconstruction model.

[0066] Furthermore, to achieve the above objectives, this application also proposes an optimization device for a three-dimensional human body reconstruction model, the optimization device comprising:

[0067] The determination module is used to construct a prediction parameter set and a reference parameter set from the original human portrait image and the reference human portrait image respectively through the initial 3D human body reconstruction model;

[0068] The determining module is further configured to generate a three-dimensional human body mesh based on the predicted parameter set and the reference parameter set using an expressive human body model;

[0069] The construction module is used to construct a reconstructed image based on the human body 3D mesh and the reference human portrait image through the initial 3D human body reconstruction model;

[0070] The optimization module is used to optimize the initial three-dimensional human body reconstruction model based on the original human portrait image, the prediction parameter set, the human body three-dimensional mesh, and the reconstructed image to obtain the target three-dimensional human body reconstruction model.

[0071] Furthermore, to achieve the above objectives, this application also proposes a three-dimensional human body reconstruction device, which determines a target three-dimensional human body reconstruction model and an expressive human body model based on the aforementioned optimization method for three-dimensional human body reconstruction models. The three-dimensional human body reconstruction device includes:

[0072] The determination module is used to obtain a target parameter set by using the target human image through the target three-dimensional human body reconstruction model;

[0073] The determining module is further configured to obtain a three-dimensional mesh of the target human body by passing the target parameter set through the expressive human body model;

[0074] The generation module is used to generate a target three-dimensional human reconstruction image based on the target human three-dimensional mesh and the target human image through the target three-dimensional human reconstruction model.

[0075] In addition, to achieve the above objectives, this application also proposes an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the optimization method for the three-dimensional human body reconstruction model or the three-dimensional human body reconstruction method as described above.

[0076] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the optimization method for the three-dimensional human body reconstruction model or the three-dimensional human body reconstruction method described above.

[0077] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the optimization method for the three-dimensional human body reconstruction model or the three-dimensional human body reconstruction method described above.

[0078] One or more technical solutions proposed in this application have at least the following technical effects:

[0079] The three-dimensional human body reconstruction model optimization method, three-dimensional human body reconstruction method, optimization device, reconstruction device, equipment, storage medium and product proposed in this application are as follows: An initial three-dimensional human body reconstruction model is used to construct a prediction parameter set and a reference parameter set from an original human image and a reference human image, respectively; a three-dimensional human body mesh is generated based on the prediction parameter set and the reference parameter set using an expressive human body model; a reconstructed image is constructed based on the three-dimensional human body mesh and the reference human image using the initial three-dimensional human body reconstruction model; and the initial three-dimensional human body reconstruction model is optimized based on the original human image, the prediction parameter set, the three-dimensional human body mesh and the reconstructed image to obtain a target three-dimensional human body reconstruction model. Compared to existing technologies, this application designs an expressive human body model that allows for the simultaneous prediction of parameters related to the body, hands, and head (face) from a single image, thereby achieving parametric modeling that is consistent with facial expressions and the whole body. At the same time, by optimizing the initial 3D human body reconstruction model and fusing optimized parameter priors from the original human portrait image and the reference human portrait image, the reconstruction accuracy of local details is significantly improved. The pixel-level difference between the reconstructed image and the original human portrait image is used as a supervision signal to drive the model to learn a more accurate geometric and appearance representation. Finally, efficient and high-fidelity 3D human body reconstruction is achieved without the need for frame-by-frame optimization. Attached Figure Description

[0080] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0081] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a flowchart illustrating an embodiment of the optimization method for the three-dimensional human body reconstruction model of this application.

[0083] Figure 2 A schematic diagram of the visual estimator structure provided in Embodiment 1 of the optimization method for the three-dimensional human body reconstruction model of this application;

[0084] Figure 3 The flowchart for constructing the reconstructed image is provided in Embodiment 1 of the optimization method for the three-dimensional human body reconstruction model of this application;

[0085] Figure 4 The overall reconstruction flowchart provided in Embodiment 1 of the optimization method for the three-dimensional human body reconstruction model of this application;

[0086] Figure 5 This is a flowchart illustrating the second embodiment of the optimization method for the three-dimensional human body reconstruction model in this application.

[0087] Figure 6 This is a flowchart illustrating an embodiment of the three-dimensional human body reconstruction method of this application.

[0088] Figure 7 This is an application example diagram provided for Embodiment 1 of the three-dimensional human body reconstruction method of this application;

[0089] Figure 8 This is a schematic diagram of the module structure of the optimization device for the three-dimensional human body reconstruction model according to an embodiment of this application;

[0090] Figure 9 This is a schematic diagram of the modular structure of the three-dimensional human body reconstruction device according to an embodiment of this application;

[0091] Figure 10 This is a schematic diagram of the device structure of the three-dimensional human body reconstruction method or the hardware operating environment involved in the three-dimensional human body reconstruction method in the embodiments of this application.

[0092] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0093] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0094] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0095] To address the problems of low reconstruction accuracy of facial and hand regions, poor local pixel alignment, and lack of unified parametric human body modeling in existing 3D human body reconstruction technologies, the main solution of this application is as follows: First, construct a prediction parameter set and a reference parameter set by using an initial 3D human body reconstruction model with the original portrait image and a reference portrait image, respectively. Second, generate a 3D human body mesh using an expressive human body model based on the prediction parameter set and the reference parameter set. Third, construct a reconstructed image using the initial 3D human body reconstruction model based on the 3D human body mesh and the reference portrait image. Finally, optimize the initial 3D human body reconstruction model based on the original portrait image, the prediction parameter set, the 3D human body mesh, and the reconstructed image to obtain the target 3D human body reconstruction model.

[0096] As can be seen from the above, compared with the prior art, the present application designs an expressive human body model that enables the simultaneous prediction of parameters related to the body, hands, and head (face) from a single image, thereby achieving parametric modeling of facial expressions consistent with the whole body. At the same time, by optimizing the initial 3D human body reconstruction model and by fusing the optimized parameter priors of the original human portrait image and the reference human portrait image, the reconstruction accuracy of local details is significantly improved. The pixel-level difference between the reconstructed image and the original human portrait image is used as a supervision signal to drive the model to learn a more accurate geometric and appearance representation. Finally, efficient and high-fidelity 3D human body reconstruction is achieved without the need for frame-by-frame optimization.

[0097] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or three-dimensional human body reconstruction device capable of performing the above functions. The following description uses a three-dimensional human body reconstruction device as an example to illustrate this embodiment and the subsequent embodiments.

[0098] Based on this, embodiments of this application provide an optimization method for a three-dimensional human body reconstruction model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the optimization method for the three-dimensional human body reconstruction model of this application.

[0099] In this embodiment, the optimization method for the three-dimensional human body reconstruction model includes steps S10 to S40:

[0100] Step S10: The original human portrait image and the reference human portrait image are respectively constructed using the initial three-dimensional human body reconstruction model to construct a prediction parameter set and a reference parameter set;

[0101] It should be noted that the original human image refers to a single target image to be reconstructed in 3D. This can be a single input human image or a video frame containing a human image from a video frame sequence. The prediction parameter set can be obtained from the original human image and includes original body parameters, original head parameters, and original camera parameters. The original body parameters include original body pose parameters and original body shape parameters, while the original head parameters include original head pose parameters, original head shape parameters, and original facial expression parameters. The prediction parameter set can be represented as follows: ,in, These are body parameters, including the original body posture parameters. and original body shape parameters , Represents the rotation angles of various joints of the human body in the original portrait image; These are head parameters, which include the original head pose parameters. Original head shape parameters and original facial expression parameters , Indicates the angle of rotation of the head, eyes, etc. Indicates facial expression parameters; These are the original camera parameters, representing the camera parameters.

[0102] It should be noted that the reference portrait image This refers to a portrait image depicting the same subject as the original portrait image, and the pose of the reference portrait image is usually more standardized compared to the original portrait image. The reference parameter set can be obtained from the reference portrait image and includes body parameters, head parameters, and camera parameters. Reference body parameters include reference body pose parameters and reference body shape parameters, while reference head parameters include reference head pose parameters, reference head shape parameters, and reference expression parameters. The reference parameter set can be represented as... ,in, Body parameters, including reference body posture parameters and reference body shape parameters , Represents the rotation angle of each joint of the human body in the reference portrait image; For head parameters, which include reference head pose parameters Reference head shape parameters and reference facial expression parameters , Indicates the rotation angle of the head, eyes, etc. Indicates facial expression parameters; , indicating camera parameters.

[0103] It is understandable that the initial 3D human reconstruction model refers to the unoptimized 3D human reconstruction model, which includes a visual estimator (Visual Transformer estimator, Fvit) and a neural renderer. The visual estimator is mainly used to obtain a parameter set from the human portrait image, while the neural renderer is mainly used to construct the reconstructed image based on the 3D human body mesh and the human portrait image.

[0104] In one feasible implementation, the initial 3D human body reconstruction model includes a visual estimator, which comprises a feature extraction layer, a multi-head prediction layer, and a parameter decoding layer. The step of constructing a prediction parameter set from the original human image using the initial 3D human body reconstruction model includes: encoding the original human image through the feature extraction layer to obtain semantic encoding; integrating the semantic encoding through the multi-head prediction layer to obtain target semantic encoding; and decoding the target semantic encoding through the parameter decoding layer to construct the prediction parameter set.

[0105] It should be noted that the visual estimator uses a ViT (Vision Transformer) based backbone network, directly taking RGB portrait images as input, without the need to crop or segment the hand or face regions; the image resolution of the input portrait image can be set to 256×192, ensuring a balance between real-time performance and spatial feature representation. This design can simultaneously predict high-dimensional parameters of the whole body and face during the forward propagation of a single image, achieving end-to-end parametric human reconstruction.

[0106] like Figure 2 As shown, the visual estimator includes a feature extraction layer, a multi-head prediction layer, and a parameter decoding layer, wherein the feature extraction layer, the multi-head prediction layer, and the parameter decoding layer are connected in sequence, and the visual encoder ViT is composed of the feature extraction layer and the multi-head prediction layer.

[0107] It should be noted that the feature extraction layer can be composed of multiple Transformer encoders, which can be used to encode the original portrait image to obtain the global and local features (i.e., semantic encoding) of the original portrait image; the multi-head prediction layer can include multiple Transformer heads, which can integrate the semantic encoding; the parameter decoding layer can decode the integrated semantic encoding (i.e., target semantic encoding) to construct the prediction parameter set.

[0108] In one feasible implementation, the parameter decoding layer includes a first output head, a second output head, and a third output head; wherein, the step of decoding the target semantic code through the parameter decoding layer to construct a prediction parameter set includes: decoding the target semantic code through the first output head to obtain original body parameters; decoding the target semantic code through the second output head to obtain original head parameters; decoding the target semantic code through the third output head to obtain original camera parameters; and constructing a prediction parameter set based on the original body parameters, the original head parameters, and the original camera parameters.

[0109] It should be noted that the integrated semantic code (i.e., the target semantic code) is simultaneously input into the first, second, and third output heads for decoding, yielding the original body parameters, original head parameters, and original camera parameters. The first output head of the parameter decoding layer can be an SMSLX Head, the second output head can be a FLAME Head, and the third output head can be a Camera Head. Therefore, the original body parameters in the prediction parameter set can be obtained using existing technologies such as SMSLX, the original head parameters can be obtained using existing technologies such as FLAME, and the original camera parameters can be obtained using technologies such as CameraHead. The step of inputting the reference human image into the initial 3D human reconstruction model to obtain the reference parameter set is the same as the process described above for inputting the original human image into the initial 3D human reconstruction model to construct the prediction parameter set.

[0110] Step S20: Generate a three-dimensional human body mesh using an expressive human body model based on the predicted parameter set and the reference parameter set;

[0111] In one feasible implementation, the step of generating a three-dimensional human body mesh using an expressive human body model based on the predicted parameter set and the reference parameter set includes: determining a combined parameter set based on the predicted parameter set and the reference parameter set; and generating a three-dimensional human body mesh using the combined parameter set through the expressive human body model.

[0112] It should be noted that the predicted parameter set and the reference parameter set can be selectively fused to generate a combined parameter set. For example, the combined parameter set inherits the pose, expression and camera parameters of the original portrait image, while adopting the shape parameters of the reference portrait image, thus forming the optimal parameter combination.

[0113] In one feasible implementation, the step of determining the combined parameter set based on the predicted parameter set and the reference parameter set includes: obtaining the original body pose parameters, original head pose parameters, original facial expression parameters, and original camera parameters from the predicted parameter set; obtaining the reference body shape parameters and reference head shape parameters from the reference parameter set; and determining the combined parameter set based on the original body pose parameters, the original head pose parameters, the original facial expression parameters, the original camera parameters, the reference body shape parameters, and the reference head shape parameters.

[0114] It should be noted that the original body pose parameters, original head pose parameters, original facial expression parameters, and original camera parameters can be selected from the prediction parameter set and compared with those from... The selected reference body shape parameters and reference head shape parameters are combined to obtain the combined parameters. .

[0115] Understandably, reference portrait images typically have standardized poses, making them more reliable for predicting the inherent body and head shapes of the individuals. However, the specific poses and expressions of the reference portrait images are not the targets we want to reconstruct. Therefore, in the parameter combination process, we adopt the original body pose, original head pose, original facial expression, and original camera parameters predicted from the original portrait image, and integrate the reference body shape parameters and reference head shape parameters provided by the reference portrait image. This allows the model to clearly distinguish and process pose changes and identity shapes separately during training and inference, effectively overfitting to specific training poses and enhancing the generalization ability to accurately reconstruct challenging unseen poses and expressions.

[0116] In one feasible implementation, the expressive human body model includes a 3D face reconstruction model and a human body model; wherein, the step of generating a 3D human body mesh by combining the parameter set through the expressive human body model includes: converting the original head pose parameters, original facial expression parameters, and reference head shape parameters in the combined parameter set into a head 3D mesh through the 3D face reconstruction model, wherein the head 3D mesh includes a head 3D mesh point cloud and the connection relationships of the head 3D mesh point cloud; converting the original body pose parameters and reference body shape parameters in the combined parameter set into a body 3D mesh through the human body model, wherein the body 3D mesh includes a body 3D mesh point cloud and the connection relationships of the body 3D mesh point cloud; and fusing the head 3D mesh and the body 3D mesh to generate a 3D human body mesh.

[0117] It should be noted that expressive human models This includes 3D face reconstruction models and human body models, where the 3D face reconstruction model can be achieved through models such as the FLAME model. Once this is achieved, human body models can be created using models such as the SMPLX model. Waiting to be realized.

[0118] It should be noted that the head 3D mesh, or head 3D mesh, is generated by using a 3D face reconstruction model to convert the original head pose parameters, original facial expression parameters, and reference head shape parameters from the original image into a head 3D mesh. Among them, the head 3D mesh ,in, The point cloud representing the 3D mesh of the head. This represents the connectivity of the point cloud in the head 3D mesh. The body 3D mesh is created by converting the body parameters of the original and reference human images into a 3D body mesh using a human model. ,Right now ;in Point cloud representing a 3D mesh of the body, This represents the connection relationships of the point cloud in the three-dimensional mesh of the body.

[0119] In practice, the point cloud of the head 3D mesh can be fused with the point cloud of the body 3D mesh to generate a human 3D mesh. (i.e., 3D human body mesh), in which , The point cloud of the human body 3D mesh. This relates to the connection relationships of the point cloud in the 3D mesh of the human body. Specifically, it relates to the point cloud of the 3D mesh of the head. Point cloud of the body's 3D mesh The points are then fused to obtain a fused point cloud. The fusion method could be, for example, fusing the point cloud of a 3D body mesh. Replace the head point cloud in the image with the point cloud of the head 3D mesh. This yields the fused point cloud; projecting the fused point cloud using the original camera parameters creates the 3D human body mesh V. The above steps can be expressed by the following formula:

[0120]

[0121] in, This represents a 3D point cloud fusion function.

[0122] The calculation process is as follows:

[0123] ,

[0124] in, This represents a 3D point cloud fusion function. This represents the dot product operation of matrices. For FLAME 3D face reconstruction model, This is an SMPLX human model.

[0125] Step S30: Based on the three-dimensional human body mesh and the reference human image, construct a reconstructed image using the initial three-dimensional human body reconstruction model;

[0126] In one feasible implementation, the initial 3D human body reconstruction model includes a neural renderer, which comprises an image representation extractor, an action mapping module, a Gaussian decoder, and a Gaussian renderer. The step of constructing a reconstructed image based on the 3D human body mesh and the reference human image using the initial 3D human body reconstruction model includes: extracting deep neural representations of the reference human image using the image representation extractor; processing the 3D human body mesh and the deep neural representations using the action mapping module to obtain target deep neural representations; decoding the target deep neural representations using the Gaussian decoder to obtain a Gaussian attribute set; and rendering the Gaussian attribute set into a reconstructed image using the Gaussian renderer.

[0127] It should be noted that the neural renderer Image representation extractor Action mapping module Gaussian decoder and Gaussian renderer composition.

[0128] It is understandable that, such as Figure 3 As shown, the reconstructed image is a two-dimensional image synthesized by the neural renderer based on the visual representation of the human body 3D mesh and the reference human image. This image is consistent with the original human image in terms of viewpoint, and its purpose is to provide a differentiable, pixel-level supervision signal.

[0129] In the specific implementation, firstly, for the input reference human image, the deep neural representation Y (i.e., deep neural representation) of the reference image is extracted through an image representation extractor, which serves as the basis for subsequent high-fidelity human reconstruction. Based on deep neural representation and human 3D mesh, the human representation is reconstructed through an action mapping module to obtain the human deep neural representation in the target space (i.e., the target deep neural representation). Next, the deep neural representation of the target is decoded using a Gaussian decoder to obtain the Gaussian attribute set G of the human body. The Gaussian property set includes position, 3D rotation, scale, transparency, and color attributes, and can be represented as... Where y represents the position of the Gaussian sphere in space, r represents 3D rotation, and s represents scale. 'c' represents transparency, and 'c' represents color; finally, the Gaussian property set G is rendered into a reconstructed image using a Gaussian renderer. , can be represented as The calculation process for reconstructing the image above can be represented as follows:

[0130] .

[0131] Step S40: Optimize the initial three-dimensional human reconstruction model based on the original human portrait image, the prediction parameter set, the three-dimensional human body mesh, and the reconstructed image to obtain the target three-dimensional human reconstruction model.

[0132] In practical implementation, a two-stage training approach can be used to optimize the visual estimator and the neural renderer. Specifically, in the first stage, the visual estimator predicts the parameters of the human model (a 3D human mesh), achieving coarse-level estimation of human pose, shape, and expression. In the second stage, the neural renderer is introduced for joint training, using photometric consistency loss as a supervision signal to further improve local details and pixel alignment accuracy. The optimized target visual estimator can estimate the pose, shape, and expression of the 3D human from the image, while the target neural renderer is used to reproject the 3D human model into a realistic image. In other words, the target visual estimator and the target neural renderer can jointly achieve end-to-end 3D human reconstruction.

[0133] It should be noted that, as Figure 4 As shown, it illustrates the optimized architecture of a 3D human body reconstruction model, which mainly includes: a visual estimator, an expressive human body model, a neural renderer, and a two-stage training and optimization. Through this architecture, an optimized target visual estimator and an optimized target neural renderer can be obtained, thereby enabling the image data to sequentially achieve an end-to-end processing chain from feature extraction, parameter regression, mesh generation to rendering optimization.

[0134] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Step S40 also includes steps S401 to S402:

[0135] Step S401: The visual estimator is initially optimized based on the true parameter set, the prediction parameter set, the original human portrait image, and the human body 3D mesh to obtain the initial visual estimator;

[0136] It should be noted that in the first stage (i.e., the initial optimization stage), the visual estimator can be initially optimized using the prediction parameter set, the original human portrait image, and the human body 3D mesh, thereby obtaining the initial visual estimator.

[0137] In one feasible implementation, the step of performing initial optimization of the visual estimator based on the ground truth parameter set, the predicted parameter set, the original portrait image, and the human body 3D mesh to obtain an initial visual estimator includes: obtaining the ground truth parameter set; determining a parameter regression loss based on the predicted parameter set and the ground truth parameter set; determining a keypoint loss based on the original portrait image, original camera parameters, and the human body 3D mesh; and performing initial optimization of the visual estimator based on the parameter regression loss and the keypoint loss to obtain the initial visual estimator.

[0138] It should be noted that the ground truth parameter set can be obtained from existing public datasets (such as Ubody), and it includes ground truth body parameters, ground truth head parameters, and ground truth camera parameters. The ground truth body parameters include ground truth body pose parameters and ground truth body shape parameters, and the ground truth head parameters include ground truth head pose parameters, ground truth head shape parameters, and ground truth facial expression parameters. The ground truth parameter set can be represented as: ,in, These are body parameters, including true body pose parameters. And true body shape parameters , Represents the rotation angles of various limb joints in a ground truth portrait image. These are head parameters, which include ground truth head pose parameters. Truth value head shape parameters And true facial expression parameters , Indicates the rotation angle of the head, such as the eyes. Indicates facial expression parameters; Here, represents the ground truth camera parameters, and represents the camera parameters themselves. The vision estimator is optimized using the ground truth parameter set and the predicted parameter set, primarily through parametric regression loss and keypoint loss.

[0139] In one feasible implementation, the step of determining the parameter regression loss based on the predicted parameter set and the true parameter set includes: calculating a first loss value between the true body parameter in the true parameter set and the original body parameter in the predicted parameter set; calculating a second loss value between the true head parameter in the true parameter set and the original body parameter in the predicted parameter set; and determining the parameter regression loss based on the first loss value and the second loss value.

[0140] In the specific implementation, the L2 norm loss (i.e., the first loss value) of the ground truth body parameters in the ground truth parameter set and the original body parameters in the prediction parameter set are calculated separately; the L1 norm loss (i.e., the second loss value) of the ground truth head parameters in the ground truth parameter set and the original head parameters in the prediction parameter set are calculated separately; and then the parametric regression loss is calculated based on the first loss value and the second loss value. This step can be represented as:

[0141]

[0142] Where ||2 represents the L2 norm loss and || represents the L1 norm loss; the L2 norm loss can be such as MSE, and the L1 norm loss can be such as MAE.

[0143] In one feasible implementation, key point detection is performed on the original human portrait image to obtain three-dimensional human key points; the three-dimensional human body mesh is sampled to obtain predicted three-dimensional human body key points.

[0144] Two-dimensional human keypoint detection is performed on the original human image to obtain two-dimensional human keypoints; based on the original camera parameters of the prediction parameter set and the predicted three-dimensional human keypoints, predicted two-dimensional human keypoints are obtained; a third loss value is calculated between the two-dimensional human keypoints and the predicted two-dimensional human keypoints; a fourth loss value is calculated between the three-dimensional human keypoints and the predicted three-dimensional human keypoints; and the keypoint loss is determined based on the third loss value and the fourth loss value.

[0145] In practical implementation, a 3D human key point detector can be used. Keypoint detection is performed on the original portrait image to obtain the 3D human keypoints of the original portrait image. Then, through the sampling function The original human portrait image's 3D human body mesh is sampled to extract the corresponding predicted 3D human body key points. Then, through a two-dimensional human key point detector Two-dimensional human keypoint detection is performed on the original portrait image to obtain the two-dimensional human keypoints of the original portrait image. Finally, using the original camera parameters of the original portrait image. Predicting 3D human key points Projecting the data onto the original human image space yields predicted 2D human keypoints. The L1 norm loss (third loss value) between the predicted 2D human keypoints and the 2D human keypoints in the original human image is calculated. Similarly, the L1 norm loss (fourth loss value) between the predicted 3D human keypoints and the 3D human keypoints in the original human image is calculated. The keypoint loss is then determined based on the sum of the third and fourth loss values. This process can be represented as:

[0146] ;

[0147] in, ,here It is an existing 3D human keypoint detector, such as MMPose; This is the sampling function, which samples and extracts the corresponding 3D human body key points from the human body mesh V; = (I), here It is an existing 2D human keypoint detector, such as DWPose.

[0148] In practical implementation, the sum of the parametric regression loss and the keypoint loss can be used as the objective optimization function to optimize the visual estimator, resulting in the initially optimized visual estimator (i.e., the initial visual estimator). The optimization process can be expressed as:

[0149] .

[0150] It should be noted that the optimizer used in the optimization process can be AdamW, the model learning rate of the optimizer can be set to 0.001, and the optimization stopping condition is to train on the entire training set for M1 epochs, where M1 can be 200 in the experimental settings.

[0151] Step S402: Optimize the initial visual estimator and the neural renderer based on the original human portrait image and the reconstructed image to obtain the target visual estimator and the target neural renderer for the target 3D human reconstruction model.

[0152] In one feasible implementation, the step of optimizing the initial visual estimator and the neural renderer based on the original portrait image and the reconstructed image to obtain a target visual estimator and a target neural renderer for the target 3D human body reconstruction model includes: calculating a fifth loss value between the original portrait image and the reconstructed image; extracting multi-scale features from the reconstructed image and the original portrait image respectively using a multi-scale image feature extractor to obtain reconstructed multi-scale features and original multi-scale features; calculating a sixth loss value for the reconstructed multi-scale features and the original multi-scale features; optimizing the initial visual estimator and the neural renderer based on the fifth loss value and the sixth loss value respectively to obtain a target visual estimator and a target neural renderer; and obtaining a target 3D human body reconstruction model based on the target visual estimator and the target neural renderer.

[0153] In the specific implementation, the L1 norm loss (the fifth loss value) between the original portrait image and the reconstructed image is calculated; then, a multi-scale image feature extractor is used to extract features from the reconstructed image. and original human portrait images Multiscale features ( and ), and calculate the L1 norm loss between the multi-scale features of the original portrait image (i.e., the original multi-scale features) and the multi-scale features of the reconstructed image (i.e., the reconstructed multi-scale features). The sum of the fifth and sixth loss values ​​is used as the objective optimization function to optimize the initial visual Transformer estimator. Neural renderer Optimization is performed to obtain the target visual Transformer estimator. Target Neural Renderer The optimization process can be represented as:

[0154] ;

[0155] in, , , For an existing multi-scale image feature extractor, For original multi-scale features, To reconstruct multi-scale features.

[0156] It should be noted that the AdamW optimizer can be used simultaneously during the optimization process. and The optimizer's model learning rate can be set to 0.001, and the stopping condition is to train for M2 epochs on the entire training set, where M2 can be 100 in the experimental settings.

[0157] In this embodiment, the visual estimator is initially optimized based on the prediction parameter set, the original human portrait image, and the human body 3D mesh to obtain an initial visual estimator; the initial visual estimator and the neural renderer are then optimized based on the original human portrait image, the prediction parameter set, the human body 3D mesh, and the reconstructed image to obtain a target visual estimator and a target neural renderer for the target 3D human body reconstruction model.

[0158] The above approach constructs a phased optimization process: the initial optimization focuses on the geometric and pose estimation capabilities of the visual estimator; the secondary optimization, based on this, combines the neural renderer for end-to-end fine-tuning. This design not only improves the geometric accuracy of 3D human reconstruction but also enhances texture fitting and detail restoration capabilities through the secondary optimization stage, ultimately achieving high-fidelity and highly realistic 3D human modeling results.

[0159] Based on this, the embodiments of this application also provide a three-dimensional human body reconstruction method, referring to... Figure 6 , Figure 6 This is a flowchart illustrating the first embodiment of the three-dimensional human body reconstruction method of this application.

[0160] In this embodiment, the target three-dimensional human body reconstruction model and the expressive human body model are determined based on the optimization method of the three-dimensional human body reconstruction model described above. The three-dimensional human body reconstruction method includes steps S10'~S30':

[0161] Step S10': Obtain the target parameter set by using the target human image through the target three-dimensional human body reconstruction model;

[0162] It should be noted that the target 3D human body reconstruction model is the optimized 3D human body reconstruction model, which can be used for 3D human body reconstruction.

[0163] Step S20': The target parameter set is processed by the expressive human body model to obtain a three-dimensional mesh of the target human body;

[0164] It should be noted that the target parameter set can be input into an expressive human body model to generate a target human body 3D mesh with fine geometric structure and accurate posture expression, providing an accurate geometric basis for subsequent high-quality 3D reconstruction.

[0165] Step S30': Generate a target three-dimensional human reconstruction image based on the target human body three-dimensional mesh and the target human image through the target three-dimensional human body reconstruction model.

[0166] It should be noted that the optimized target 3D human body reconstruction model can be used to deeply fuse and render the precise geometric structure (target human body 3D mesh) with the realistic appearance information (target human portrait image), and finally output a 3D human body reconstruction image with high-fidelity details and realistic texture.

[0167] In the implementation, the Human3.6M, COCO, MPII, and UBody datasets were used for training and evaluation. Experimental results show that the LVE index is reduced to 2.55×10⁻⁶ for head reconstruction. -5 It significantly outperforms other contemporary methods such as SMIRRK and TEASER; the hand PA-PVE error is 8.8, which is better than similar methods based on the SMPLEX model; the PCK@0.05 of the whole body key points reaches 0.87, showing pixel-level alignment effect;

[0168] With an inference speed of 0.05s / frame, it achieves real-time portrait reconstruction and animation-driven performance, with driving effects as shown below. Figure 7 As shown.

[0169] This embodiment obtains a target parameter set by passing the target human image through the target 3D human body reconstruction model; then, it obtains a target human body 3D mesh by passing the target parameter set through the expressive human body model; finally, it generates a target 3D human body reconstruction image based on the target human body 3D mesh and the target human image through the target 3D human body reconstruction model. This method achieves rapid generation of a high-precision 3D human body model from a single image, significantly improving reconstruction efficiency and model realism, effectively preserving detailed human body features, and providing reliable technical support for applications such as digital humans and virtual reality.

[0170] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the three-dimensional human body reconstruction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0171] This application also provides an optimization device for a three-dimensional human body reconstruction model, please refer to... Figure 8 The optimization device for the three-dimensional human body reconstruction model includes:

[0172] The determination module 10 is used to construct a prediction parameter set and a reference parameter set by using the original human portrait image and the reference human portrait image respectively through the initial three-dimensional human body reconstruction model;

[0173] The determining module 10 is further configured to generate a three-dimensional human body mesh based on the predicted parameter set and the reference parameter set using an expressive human body model.

[0174] Construction module 20 is used to construct a reconstructed image based on the human body 3D mesh and the reference human image through the initial 3D human body reconstruction model;

[0175] The optimization module 30 is used to optimize the initial three-dimensional human body reconstruction model based on the original human portrait image, the prediction parameter set, the human body three-dimensional mesh and the reconstructed image to obtain the target three-dimensional human body reconstruction model.

[0176] The optimization device for the three-dimensional human body reconstruction model provided in this application adopts the optimization method for the three-dimensional human body reconstruction model in the above embodiments, which can solve the problems of low reconstruction accuracy of the face and hand regions, poor local pixel alignment, and lack of unified parametric modeling of the human body in existing three-dimensional human body reconstruction. Compared with the prior art, the beneficial effects of the optimization device for the three-dimensional human body reconstruction model provided in this application are the same as the beneficial effects of the optimization method for the three-dimensional human body reconstruction model provided in the above embodiments, and other technical features in the optimization device for the three-dimensional human body reconstruction model are the same as the features disclosed in the method of the above embodiments, and will not be repeated here.

[0177] This application also provides a three-dimensional human body reconstruction device, please refer to... Figure 9The target 3D human body reconstruction model and the expressive human body model are determined by the optimization method based on any one of the above-mentioned methods. The 3D human body reconstruction device includes:

[0178] The determination module 10' is used to obtain a target parameter set by passing the target human image through the target three-dimensional human body reconstruction model;

[0179] The determining module 10' is further configured to obtain a three-dimensional mesh of the target human body by passing the target parameter set through the expressive human body model;

[0180] The generation module 20' is used to generate a target three-dimensional human reconstruction image based on the target human three-dimensional mesh and the target human image through the target three-dimensional human reconstruction model.

[0181] The three-dimensional human body reconstruction device provided in this application, employing the three-dimensional human body reconstruction method described in the above embodiments, can solve the technical problem that the current parametric modeling method easily leads to severe distortion in the three-dimensional reconstruction results, affecting the visual realism of the model. Compared with the prior art, the beneficial effects of the three-dimensional human body reconstruction device provided in this application are the same as those of the three-dimensional human body reconstruction method provided in the above embodiments, and other technical features in the three-dimensional human body reconstruction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0182] This application provides an apparatus comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the optimization method for the three-dimensional human body reconstruction model or the three-dimensional human body reconstruction method described in Embodiment 1 above.

[0183] The following is for reference. Figure 10 The diagram illustrates a structural schematic of a device suitable for implementing embodiments of this application. The devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0184] like Figure 10As shown, the device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for device operation. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the 3D human reconstruction device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagram shows devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0185] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0186] The device provided in this application, employing the optimization method or 3D human body reconstruction method of the above embodiments, can solve problems such as low reconstruction accuracy of facial and hand regions, poor local pixel alignment, and lack of unified parametric human body modeling in existing 3D human body reconstruction. Compared with the prior art, the beneficial effects of the device provided in this application are the same as those of the optimization method or 3D human body reconstruction method of the above embodiments, and other technical features of this 3D human body reconstruction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0187] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0188] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0189] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the optimization method or the three-dimensional human body reconstruction method of the above embodiments.

[0190] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0191] The aforementioned computer-readable storage medium may be included in the device or may exist independently and not assembled into the device.

[0192] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a device, enable the device to construct a prediction parameter set and a reference parameter set from an original human portrait image and a reference human portrait image respectively through an initial 3D human body reconstruction model; generate a 3D human body mesh based on the prediction parameter set and the reference parameter set through an expressive human body model; construct a reconstructed image based on the 3D human body mesh and the reference human portrait image through the initial 3D human body reconstruction model; and optimize the initial 3D human body reconstruction model based on the original human portrait image, the prediction parameter set, the 3D human body mesh, and the reconstructed image to obtain a target 3D human body reconstruction model.

[0193] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0195] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0196] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the optimization method or 3D human body reconstruction method of the above-mentioned 3D human body reconstruction model. This can solve problems such as low reconstruction accuracy of the face and hand regions, poor local pixel alignment, and lack of unified parametric human body modeling in existing 3D human body reconstruction methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the optimization method or 3D human body reconstruction method of the above-mentioned embodiments, and will not be elaborated upon here.

[0197] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the optimization method for the three-dimensional human body reconstruction model or the three-dimensional human body reconstruction method described above.

[0198] The computer program product provided in this application can solve the problems of low reconstruction accuracy of facial and hand regions, poor local pixel alignment, and lack of unified parametric human body modeling in existing 3D human body reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the optimization method or 3D human body reconstruction method of the above embodiments, and will not be repeated here.

[0199] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An optimization method for a three-dimensional human body reconstruction model, characterized in that, The optimization method for the three-dimensional human body reconstruction model includes: The original human portrait image and the reference human portrait image are used to construct prediction parameter sets and reference parameter sets respectively through the initial 3D human body reconstruction model; Based on the predicted parameter set and the reference parameter set, a three-dimensional human body mesh is generated using an expressive human body model. The step of generating a 3D human body mesh using an expressive human body model based on the predicted parameter set and the reference parameter set includes: A combined parameter set is determined based on the predicted parameter set and the reference parameter set; The step of determining the combined parameter set based on the predicted parameter set and the reference parameter set includes: Obtain the original body parameters, original head parameters, original facial expression parameters, and original camera parameters from the prediction parameter set; Obtain the reference body shape parameters and reference head shape parameters from the reference parameter set; A set of combined parameters is determined based on the original body parameters, the original head parameters, the original facial expression parameters, the original camera parameters, the reference body shape parameters, and the reference head shape parameters; The combined parameter set is used to generate a three-dimensional human body mesh through the expressive human body model; The expressive human body model includes a 3D face reconstruction model and a human body model. The step of generating a 3D human body mesh by combining the parameter set through the expressive human body model includes: Based on the original head parameters, original facial expression parameters, and reference head shape parameters in the combined parameter set, a three-dimensional head mesh is obtained through the three-dimensional face reconstruction model. The three-dimensional head mesh includes a three-dimensional head mesh point cloud and the connection relationship of the three-dimensional head mesh point cloud. Based on the original body parameters and reference body shape parameters in the combined parameter set, a three-dimensional body mesh is obtained through the human body model. The three-dimensional body mesh includes a three-dimensional body mesh point cloud and the connection relationship of the three-dimensional body mesh point cloud. The head 3D mesh and the body 3D mesh are fused to generate a human 3D mesh; Based on the human body 3D mesh and the reference human image, a reconstructed image is constructed using the initial 3D human body reconstruction model; The initial 3D human body reconstruction model is optimized based on the original human portrait image, the prediction parameter set, the 3D human body mesh, and the reconstructed image to obtain the target 3D human body reconstruction model.

2. The optimization method for the three-dimensional human body reconstruction model as described in claim 1, characterized in that, The initial 3D human body reconstruction model includes a visual estimator, which comprises a feature extraction layer, a multi-head prediction layer, and a parameter decoding layer; wherein, the step of constructing a prediction parameter set from the original human portrait image using the initial 3D human body reconstruction model includes: The original portrait image is encoded through the feature extraction layer to obtain semantic encoding; The semantic codes are integrated by the multi-head prediction layer to obtain the target semantic code; The target semantic encoding is decoded by the parameter decoding layer to construct a prediction parameter set.

3. The optimization method for the three-dimensional human body reconstruction model as described in claim 2, characterized in that, The parameter decoding layer includes a first output head, a second output head, and a third output head; wherein, the step of decoding the target semantic code through the parameter decoding layer to construct a prediction parameter set includes: The target semantic encoding is decoded using the first output head to obtain the original body parameters; The target semantic code is decoded using the second output header to obtain the original header parameters; The target semantic code is decoded by the third output head to obtain the original camera parameters; Based on the original body parameters, the original head parameters, and the original camera parameters, a set of prediction parameters is constructed.

4. The optimization method for the three-dimensional human body reconstruction model as described in claim 1, characterized in that, The initial 3D human body reconstruction model includes a neural renderer, which comprises an image representation extractor, an action mapping module, a Gaussian decoder, and a Gaussian renderer; wherein, the step of constructing a reconstructed image based on the 3D human body mesh and the reference human portrait image using the initial 3D human body reconstruction model includes: The deep neural representation of the reference portrait image is extracted using the image representation extractor. The motion mapping module processes the human body 3D mesh and the deep neural representation to obtain the target deep neural representation. The Gaussian attribute set is obtained by decoding the target deep neural representation using the Gaussian decoder. The Gaussian property set is rendered into a reconstructed image using the Gaussian renderer.

5. The optimization method for a three-dimensional human body reconstruction model as described in any one of claims 1-4, characterized in that, The initial 3D human body reconstruction model includes a visual estimator and a neural renderer; wherein, the step of optimizing the initial 3D human body reconstruction model based on the original human portrait image, the prediction parameter set, the human body 3D mesh, and the reconstructed image to obtain the target 3D human body reconstruction model includes: The visual estimator is initially optimized based on the true parameter set, the prediction parameter set, the original human portrait image, and the human body 3D mesh to obtain the initial visual estimator; The initial visual estimator and the neural renderer are optimized based on the original human portrait image and the reconstructed image to obtain the target visual estimator and the target neural renderer for the target 3D human body reconstruction model.

6. The optimization method for the three-dimensional human body reconstruction model as described in claim 5, characterized in that, The step of performing initial optimization of the visual estimator based on the true parameter set, the predicted parameter set, the original human portrait image, and the human body 3D mesh to obtain an initial visual estimator includes: Obtain the set of truth parameters; The parametric regression loss is determined based on the predicted parameter set and the true parameter set; The key point loss is determined based on the original human portrait image, the original camera parameters, and the human body 3D mesh. The visual estimator is initially optimized based on the parameter regression loss and the keypoint loss to obtain the initial visual estimator.

7. The optimization method for the three-dimensional human body reconstruction model as described in claim 6, characterized in that, The step of determining the parameter regression loss based on the predicted parameter set and the true parameter set includes: Calculate the first loss value between the true body parameters in the true parameter set and the original body parameters in the predicted parameter set; Calculate a second loss value between the truth head parameters in the truth parameter set and the original head parameters in the prediction parameter set; Based on the first loss value and the second loss value, the parameter regression loss is determined.

8. The optimization method for the three-dimensional human body reconstruction model as described in claim 6, characterized in that, The step of determining the keypoint loss based on the original human portrait image, the original camera parameters, and the human body 3D mesh includes: Key point detection is performed on the original human portrait image to obtain three-dimensional human key points; The three-dimensional human body mesh is sampled to obtain predicted three-dimensional human body key points; Two-dimensional human key point detection is performed on the original human portrait image to obtain two-dimensional human key points; Based on the original camera parameters of the predicted parameter set and the predicted 3D human key points, the predicted 2D human key points are obtained. Calculate the third loss value between the two-dimensional human key points and the predicted two-dimensional human key points; Calculate the fourth loss value between the three-dimensional human key points and the predicted three-dimensional human key points; Based on the third and fourth loss values, the critical point loss is determined.

9. The optimization method for the three-dimensional human body reconstruction model as described in claim 5, characterized in that, The step of optimizing the initial visual estimator and the neural renderer based on the original human portrait image and the reconstructed image to obtain the target visual estimator and the target neural renderer for the target 3D human body reconstruction model includes: Calculate the fifth loss value between the original portrait image and the reconstructed image; Multi-scale features are extracted from the reconstructed image and the original portrait image respectively to obtain reconstructed multi-scale features and original multi-scale features; Calculate the sixth loss value of the reconstructed multi-scale features and the original multi-scale features; The initial visual estimator and the neural renderer are optimized based on the fifth loss value and the sixth loss value, respectively, to obtain the target visual estimator and the target neural renderer. Based on the target visual estimator and target neural renderer, a target 3D human reconstruction model is obtained.

10. A method for three-dimensional human body reconstruction, characterized in that, The optimization method for the three-dimensional human body reconstruction model according to any one of claims 1-9 determines the target three-dimensional human body reconstruction model and the expressive human body model, wherein the three-dimensional human body reconstruction method includes: The target human image is used to construct a target 3D human body model to obtain a target parameter set; The target parameter set is used to obtain a three-dimensional mesh of the target human body through the expressive human body model; Based on the target human body 3D mesh and the target human image, a target 3D human body reconstruction image is generated using the target 3D human body reconstruction model.

11. An optimization device for a three-dimensional human body reconstruction model, characterized in that, The optimization device for the three-dimensional human body reconstruction model includes: The determination module is used to construct a prediction parameter set and a reference parameter set from the original human portrait image and the reference human portrait image respectively through the initial 3D human body reconstruction model; The determining module is further configured to generate a three-dimensional human body mesh based on the predicted parameter set and the reference parameter set using an expressive human body model; The determining module is further configured to determine a combined parameter set based on the predicted parameter set and the reference parameter set; Obtain the original body parameters, original head parameters, original facial expression parameters, and original camera parameters from the prediction parameter set; Obtain the reference body shape parameters and reference head shape parameters from the reference parameter set; A set of combined parameters is determined based on the original body parameters, the original head parameters, the original facial expression parameters, the original camera parameters, the reference body shape parameters, and the reference head shape parameters; The combined parameter set is used to generate a three-dimensional human body mesh through the expressive human body model, wherein the expressive human body model includes a three-dimensional face reconstruction model and a human body model; Based on the original head parameters, original facial expression parameters, and reference head shape parameters in the combined parameter set, a three-dimensional head mesh is obtained through the three-dimensional face reconstruction model. The three-dimensional head mesh includes a three-dimensional head mesh point cloud and the connection relationship of the three-dimensional head mesh point cloud. Based on the original body parameters and reference body shape parameters in the combined parameter set, a three-dimensional body mesh is obtained through the human body model. The three-dimensional body mesh includes a three-dimensional body mesh point cloud and the connection relationship of the three-dimensional body mesh point cloud. The head 3D mesh and the body 3D mesh are fused to generate a human 3D mesh; The construction module is used to construct a reconstructed image based on the human body 3D mesh and the reference human portrait image through the initial 3D human body reconstruction model; The optimization module is used to optimize the initial three-dimensional human body reconstruction model based on the original human portrait image, the prediction parameter set, the human body three-dimensional mesh, and the reconstructed image to obtain the target three-dimensional human body reconstruction model.

12. A three-dimensional human body reconstruction device, characterized in that, The optimization method for the three-dimensional human body reconstruction model according to any one of claims 1-9 determines the target three-dimensional human body reconstruction model and the expressive human body model, wherein the three-dimensional human body reconstruction device comprises: The determination module is used to obtain a target parameter set by using the target human image through the target three-dimensional human body reconstruction model; The determining module is further configured to obtain a three-dimensional mesh of the target human body by passing the target parameter set through the expressive human body model; The generation module is used to generate a target three-dimensional human reconstruction image based on the target human three-dimensional mesh and the target human image through the target three-dimensional human reconstruction model.

13. A three-dimensional human body reconstruction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the optimization method for the three-dimensional human body reconstruction model as described in any one of claims 1 to 9 or to implement the steps of the three-dimensional human body reconstruction method as described in claim 10.

14. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the optimization method of the three-dimensional human body reconstruction model as described in any one of claims 1 to 9 or the steps of the three-dimensional human body reconstruction method as described in claim 10.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the optimization method for the three-dimensional human body reconstruction model as described in any one of claims 1 to 9 or the steps of the three-dimensional human body reconstruction method as described in claim 10.

Images