Method and apparatus for neurophysiological prediction

Abstract

The Neural Human Performance Capture Framework (MVS-PERF) captures the skeleton, body shape, and clothing displacement and appearance of a person from a calibrated multi-view image set. It resolves the ambiguity of predicting absolute position in monocular human mesh reconstruction and connects volume representations from NeRF with animation-friendly performance capture. MVS-PERF comprises three modules: extracting feature maps from multi-view images and fusing them into feature volumes; regressing the feature volumes into bare human parameter vectors; generating an SMPL-X body mesh with skeletal pose, body shape, and expression; and using differentiable rendering to infer clothing displacement on the bare body using neural radiation and deformation fields. A clothed body mesh is obtained by adding interpolated displacement vectors to the vertices of the SMPL-X body mesh. The obtained radiation fields are used for free-view volume rendering of the input subject.

Description

[0001] Cross-references to related applications

[0002] Pursuant to 35 USC § 119(e), this application claims priority to U.S. Provisional Patent Application Serial No. 63 / 279,916, filed November 16, 2021, entitled “MULTIVIEW NEURAL HUMAN PREDICTION USING IMPLICIT DIFFERENTIABLE RENDERER FOR FACIAL EXPRESSION, BODY POSE SHAPE AND CLOTHES PERFORMANCE CAPTURE”, and U.S. Provisional Patent Application Serial No. 63 / 168,467, filed March 31, 2021, entitled “MULTIVIEW NEURAL HUMAN PREDICTION USING IMPLICIT DIFFERENTIABLE RENDERER FOR FACIAL EXPRESSION, BODYPOSE SHAPE AND CLOTHES DISPLACEMENT”, both of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This invention relates to three-dimensional computer vision and graphics for the entertainment industry. More specifically, this invention relates to acquiring and processing three-dimensional computer vision and graphics for the creation of film, TV, music, and game content. Background Technology

[0004] Previous systems, such as Facebook FrankMocap, predicted the shape and posture of a naked body from only a single image. Such systems cannot predict the surface of clothing. These systems are 2D image translation methods and cannot handle multi-view inputs.

[0005] Implicit Part Networks predict both the body and clothing from scanned or reconstructed point clouds, but require 3D scans and cannot handle RGB images as input, nor can they handle facial expressions and appearance. Furthermore, Implicit Part Networks only predict labels to identify voxels as either body or clothing, and then explicitly fit a prior human model, which is slow. Neural Body and Animatable NeRF use Neural Radiation Fields (NeRF) to predict clothing-clad human bodies without facial expressions. However, they require creating dense latent code volumes, which limits them to low resolution and results in coarse human geometry. They are also only able to recover volumetric human models without corresponding mesh vertices. Summary of the Invention

[0006] Multi-view neural human prediction involves predicting 3D human models, including skeleton, body shape, and clothing displacement and appearance, from a set of multi-view images calibrated by a given camera.

[0007] In one aspect, the neural network accepts an input image set (which can be a single image or multiple images) from different views and predicts a layered 3D human body model. The image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image. Camera information for the image set is known. The output model comprises three layers from the inside out: a skeleton in the predicted pose; a naked 3D body with a predicted morphology of facial expressions (e.g., an SMPL-X model parameterized by blendshape and joint rotation); and a 3D field of clothing displacement and appearance RGB color inferred from the input images. A clothed body mesh is obtained by deforming the naked 3D body mesh according to the clothing displacement field.

[0008] In another aspect, the neural network comprises three sub-networks: a multi-view stereo 3D convolutional neural network (MVS-3DCNN), which encodes the input image set into features; a human mesh recovery multilayer perceptron (HMRMLP), which regresses the features into human body parameters; and a neural radiation field multilayer perceptron (NeRF MLP), which fine-tunes the MVS-3DCNN and decodes query 3D rays (3D position and orientation) into RGB colors and clothing-to-body displacement.

[0009] On another front, in test / inference mode, within the camera's field of view in the training data, the prediction of the hierarchical 3D human model is device-independent, fully automatic, and real-time for a small input set, without any explicit numerical optimization. When predicting with the trained neural network, MVS-3DCNN accepts the multi-view image set as input, selects a frontal view as the reference view, and extracts feature volumes. HMR MLP regresses all of these feature volumes into human pose, morphology, and facial feature parameters. The SMPL-X model generates a naked human body mesh based on these parameters. The naked body mesh is then converted into an occupancy field within its bounding box. For any 3D point near the body mesh, associated with the ray direction from the center of each view, the trained NeRF MLP generates RGB colors and 3D displacement vectors pointing towards the surface of the naked body. By querying all rays emitted from all pixels of the camera view (either the same as the input view or any new view), the appearance of a clothed human body can be rendered as an RGB image. By deforming the naked body using the 3D displacement vector from the sampled points, it is possible to obtain a clothed body mesh (e.g., SMPL-X+D) corresponding to the same vertices of the SMPL-X model.

[0010] In another aspect, training the neural network involves two cases: supervised and self-supervised. In the supervised case, given a labeled dataset with known human parameters, such as the H36M dataset, the ground truth (GT) parameters and morphology are compared with the parameters and morphology regressed by the CNN. The difference is calculated as the morphological loss. Simultaneously, rays are projected from sampled pixels in the input image set, and the NeRF MLP renders the rays and regresses the parameters into color and density, which are functions of the density of the displacement of the naked body and 3D clothing. The color loss is calculated by summing the differences between the sampled pixel colors and the rendered colors. On the other hand, in most existing datasets where the GT human parameters are unknown (e.g., motion capture datasets), self-supervised / self-improving training is utilized. In each training iteration, after regressing parameters from the MVS 3DCNN, the parameters are fed to an optimization-based human prediction algorithm such as SMPLifyX and optimized using explicit numerical optimization methods. The optimized parameters are compared with the parameters regressed by the CNN and converted into the morphological loss. The remaining steps are the same as supervised training, but self-improvement training takes more epochs and longer than supervised training. The entire neural network is trained using a parallel optimization algorithm such as Adam, which minimizes both morphological and color losses and outputs optimized network weights. Attached Figure Description

[0011] Figure 1The illustration shows a flowchart of neural human prediction according to some embodiments.

[0012] Figure 2 The diagram illustrates the workflow of forward prediction represented by tensor labeling according to some embodiments, where the weights of all networks MVS 3DCNN, HMR MLP, and NeRF MLP are known.

[0013] Figure 3 The diagram illustrates the workflow of training a network using supervision according to some embodiments.

[0014] Figure 4 The diagram illustrates the workflow of training a network using a self-improvement strategy according to some embodiments.

[0015] Figure 5 The illustration shows the alignment of the MVS 3DCNN with the NeRF MLP for each view according to some embodiments. Detailed Implementation

[0016] Neural human prediction involves predicting 3D human models from image sets (single or multi-view images), including skeletal pose, body shape, and clothing displacement and appearance. Examples of neural human prediction illustrate methods using neural networks. Multi-view neural human prediction outperforms single-image-based motion capture (mocap) and human augmentation in terms of quality and robustness; simplifies the architecture of body and clothing prediction networks such as implicit partial networks, which accept sparse point clouds as input, have high storage overhead, and are slow to execute; and avoids the resolution limitations of implicit code-based networks such as the Neural Body, which encodes the entire 3D volume.

[0017] Figure 1 The diagram illustrates a flowchart of neural human prediction according to some embodiments. In step 100, an input set I (a single image or multi-view images, such as a set of pictures taken around the subject) is acquired as input. Input I is represented as a 4D tensor of size N×w×h×c, where N is the number of views, and w, h, and c are the image width, height, and channels, respectively. The camera has been calibrated, so all camera information (e.g., camera parameters) is known. Image preprocessing uses existing methods such as Detectron2 and image Grab-Cut to extract the bounding box and foreground mask of the subject. The image is cropped by the bounding box and scaled to size w×h with the same aspect ratio. The image boundaries are filled with black.

[0018] The neural network (MVS-PERF) 102 comprises three components: a multi-view stereo 3D convolutional neural network (MVS-3DCNN) 104, which encodes the input image set into features; a human mesh recovery multilayer perceptron (HMR MLP) 106, which regresses the features into human body parameters; and a neural radiation field multilayer perceptron (NeRF MLP) 108, which fine-tunes the MVS-3DCNN and decodes query 3D rays (3D position and orientation) into RGB colors and clothing-to-body displacement.

[0019] In step 104, the deep 2D CNN extracts image features from each view. Each convolutional layer, except for the last layer, is followed by a batch normalization (BN) layer and a rectified linear unit (ReLU). Two downsampling layers are also placed. The output of the 2D CNN is a feature map of size w / 4×h / 4×32.

[0020] Next, first select a view as a reference view, and set its view frustum to cover the entire workspace of the subject based on perspective projection and near and far planes. From near to far, through... The cone is sampled using a depth plane. Each depth plane is parallel to both the near and far planes. All feature maps are transformed and blended into each depth plane. For any view... , The 3×3 homography image transformation matrix for the reference view (index 1) is given by the following formula:

[0021]

[0022] in , Represents camera intrinsic and non-intrinsic parameters, It is the distance from the depth plane to the center of the camera in the reference view, and It is the normal direction of the plane at that depth.

[0023] After all images are warped to the depth plane, the coordinates The cost at a given point is determined by the variance of all features. To determine. It is the average feature value across all views. The size of the cost volume is... .

[0024] In step 106, the Human Mesh Reconstruction Multilayer Perceptron (HMR MLP) comprises three linear regression layers separated by flattening and dropout layers. It regresses the feature volumes from the MVS 3DCNN into human body parameters. 114.

[0025] Human body parameters It can manipulate parametric human body models (such as SMPL-X) into 3D naked body meshes 202. Typically, SMPL-X represents... It includes skeletal pose (3D rotation angle of each joint), body blending shape parameters that control body shape (e.g., height, weight, etc.), and facial blending shape parameters that control facial expressions. It uses blending shape parameters to create a T-shaped pose mesh and then transforms the T-shaped pose mesh into a posing mesh using the skeletal pose of a linear skinned model.

[0026] On the other hand, in step 108, the cost volume is sent to a differentiable rendering MLP, such as a neural radiation field (NeRF). The NeRF MLP is formulated as a function M, which will be determined by the 3D position. and direction The query ray represented is mapped to 4-channel RGB color. , . It is a feature mapping from the cost volume of cone-shaped MVS 3DCNN 104 to the NeRF volume, and These are the weights of the NeRF MLP network. This represents the occupancy density, indicating the probability of a 3D point being inside a mesh. This can be achieved by transforming mesh 202 within cone 104 (…). Figure 2 To directly obtain the occupancy density field of the naked body Then, the density field of a clothed body. It can be represented as a 3D displacement vector field and feature mapping Functions: The 3D displacement vector field D116 represents how points on the clothed body surface 204 relate to points on the bare body surface. The displacement vector field D is also optimized during the training of the NeRF MLP.

[0027] Figure 2 The diagram illustrates a forward prediction workflow represented by tensor notation according to some embodiments, where the weights of all networks—MVS 3DCNN, HMR MLP, and NeRF MLP—are trained and fixed. An appearance image 112 is rendered by querying all rays 200 from pixels in the perspective projection image. In some embodiments, 3D human prediction 110 is implemented. A displacement field D 116 is obtained by querying sampled points near the human body. For a human performance capture task where the output mesh of a clothed body has the same topology as the template, the naked body mesh V can be made more suitable by adding interpolated displacement vectors to each vertex. b202 transforms into a clothing-wearing body mesh V c 204.

[0028] Figure 3 The diagram illustrates a workflow for training a network using supervision, according to some embodiments. The supervised training dataset (e.g., Human3.6M) contains not only image inputs I100, but also ground truth human parameters. 300 and naked body mesh 302, typically obtained using sensors or existing methods to acquire ground-based true human body parameters. 300 and naked body mesh 302. In this case, the morphological loss is obtained directly by summing the differences between the predicted naked body and the ground truth. 304.

[0029]

[0030] in It is the joint of a naked body. This represents the perspective projection of 3D points for each camera view. To efficiently train the network, in each training step, all views are selected in turn as the reference views for the MVS 3DCNN.

[0031] Meanwhile, a non-uniform sampling strategy proportional to image saliency is typically used to sample rays 306 from the input image set 100. More rays are sampled in highly saliency regions, while fewer rays are sampled from ordinary or background regions. These rays, along with feature maps from MVS 3DCNN 104, are fed into NeRF MLP 106, which renders the sample appearance in RGB. Color 308. Color loss 310 is calculated by summing all the differences between the sampled colors in the input image and the rendered color 308.

[0032] By minimizing both morphological and color losses, parallelized stochastic optimization algorithms (such as Adam) are applied to train the weights of all networks, including MVS 3DCNN, HMR MLP, and NeRF MLP.

[0033] Figure 4 The diagram illustrates the workflow of training a network using a self-improvement strategy according to some embodiments. In this case, the training dataset only provides human images without any annotations or ground truth parameters for the human body. For each image in the input set 100, the parameters of the regression are used... 114 is used as an initial guess to apply optimization-based prediction 400, such as the SMPLifyX algorithm. The optimization-based prediction first detects 2D human keypoints on each image and applies nonlinear optimization to fit the 3D human body.

[0034] Will 404 (by) 402 (parameterized) meshes are used to represent these 2D key points.

[0035]

[0036] in This represents the 2D location of the detected keypoints, and the summation is performed for all corresponding keypoints and all views.

[0037] Although nonlinear least squares optimization is numerically slow and the fitting accuracy depends on the initial guess. But it is reliable. After sufficient iterations of fitting, This will approximate the ground truth. Therefore, the self-improving training workflow can efficiently improve towards the ground truth. As summarized in the following process.

[0038] Self-improvement training workflow:

[0039] implement

[0040] The computation is performed from MVS-3DCNN and HMR MLP based on input I.

[0041] use As an initial conjecture, I is accepted as input and computed from SMPLifyX.

[0042] Sample rays from I and calculate the sampled color c from the NeRF MLP.

[0043] Calculate shape loss and color loss

[0044] The network weights of MVS 3DCNN, HMR MLP, and NeRF MLP are updated by minimizing morphological and color losses.

[0045] Repeat this process for all training data until the weights converge.

[0046] Figure 5 The illustration shows the alignment of the MVS 3DCNN with the NeRF MLP for each view according to some embodiments.

[0047] In practice, neural human prediction can be directly applied to both commercial and / or personal label-free performance capture applications, such as human 3D surface reconstruction RGB camera setups or label-free motion capture in game studios. Other applications of multi-view neural human prediction embodiments can serve as a real-time pillar technology capable of being combined with any extension, such as combining 3D modeling, depth sensing input, or using the output to create new animations. Multi-view neural human prediction can also be applied to games, VR / AR, and any real-time human-computer interaction application. Depending on the hardware used (e.g., GPU processor speed and GPU memory size), multi-view neural human prediction is real-time when processing sparse views for prediction, while near real-time processing and prediction are possible for more views (e.g., 20).

[0048] The methods described herein can be implemented on any computing device. Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframes, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablets, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computing device.

[0049] Examples of multi-view neural human prediction using implicit differentiable renderers for facial expression, body posture, and costume performance capture.

[0050] 1. A method for programming in a non-transitory memory of a device, the method comprising:

[0051] Obtain an image set as input; and

[0052] The image set is processed using a neural network, wherein the processing includes:

[0053] Encode the image set into one or more features;

[0054] The features are then regressed into human body parameters;

[0055] Fine-tuning the neural network; and

[0056] The query 3D rays are decoded into RGB colors and clothing-to-body displacement, wherein the RGB colors are based on the image set.

[0057] 2. The method according to Clause 1, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

[0058] 3. The method according to Clause 1, wherein the neural network selects a frontal view as a reference view from the image set and extracts feature bodies.

[0059] 4. The method described in Clause 3, wherein the neural network regresses all the said features into human posture, shape, and facial expression parameters.

[0060] 5. The method according to Clause 4, wherein the neural network generates a mesh of a naked human body based on the parameters.

[0061] 6. The method described in Clause 5, wherein the naked body mesh is converted into an occupied field within a bounding box.

[0062] 7. The method according to Clause 6, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the neural network generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

[0063] 8. The method according to Clause 7, wherein the appearance of a clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of the camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

[0064] 9. The method according to Clause 1, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

[0065] 10. An apparatus, the apparatus comprising:

[0066] Non-transitory memory, configured to store an application, the application being configured to:

[0067] Obtain an image set as input; and

[0068] The image set is processed using a neural network, wherein the processing includes:

[0069] Encode the image set into one or more features;

[0070] The features are then regressed into human body parameters;

[0071] Fine-tuning the neural network; and

[0072] The query 3D ray is decoded into RGB colors and clothing-to-body displacement, wherein the RGB colors are based on the image set; and

[0073] A processor configured to process the application.

[0074] 11. The apparatus according to Clause 10, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

[0075] 12. The apparatus according to Clause 10, wherein the neural network selects a frontal view as a reference view from the image set and extracts feature volumes.

[0076] 13. The apparatus according to Clause 12, wherein the neural network regresses all said features into human posture, morphology, and facial expression parameters.

[0077] 14. The apparatus according to Clause 13, wherein the neural network generates a mesh of a naked human body based on the parameters.

[0078] 15. The apparatus according to Clause 14, wherein the naked body mesh is converted into an occupied field within a bounding frame.

[0079] 16. The apparatus according to Clause 15, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the neural network generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

[0080] 17. The apparatus according to Clause 16, wherein the appearance of a clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of a camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

[0081] 18. The apparatus according to Clause 10, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

[0082] 19. An apparatus comprising:

[0083] Non-transitory memory, configured to store applications, the applications including:

[0084] A multi-view stereo 3D convolutional neural network (MVS-3DCNN) is configured to encode an input image set into features;

[0085] Human Mesh Regression Multilayer Perceptron (HMR MLP), the human mesh regression multilayer perceptron being configured to regress the features into human body parameters; and

[0086] A Neural Radiation Field Multilayer Perceptron (NeRF MLP) is configured to fine-tune the MVS-3DCNN and decode query 3D rays (3D position and orientation) into RGB color and clothing-to-body displacement; and

[0087] A processor configured to process the application.

[0088] 20. The apparatus according to Clause 19, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

[0089] 21. The apparatus according to Clause 20, wherein the MVS-3DCNN selects a frontal view as a reference view from the image set and extracts feature volumes.

[0090] 22. The apparatus according to Clause 21, wherein the HMR MLP regresses all said features into human posture, morphology, and facial expression parameters.

[0091] 23. The apparatus according to Clause 22 further includes a model configured to generate a mesh of a naked human body based on the parameters.

[0092] 24. The apparatus according to Clause 23, wherein the naked body mesh is converted into an occupied field within a bounding frame.

[0093] 25. The apparatus according to Clause 24, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the NeRF MLP generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

[0094] 26. The apparatus according to Clause 25, wherein the appearance of a clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of a camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

[0095] The invention has been described with reference to specific embodiments, which include details to facilitate understanding of the principles of the structure and operation of the invention. Such references to specific embodiments and their details herein are not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications can be made to the embodiments chosen for the examples without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method for programming in a non-transitory memory of a device, the method comprising: Obtain an image set as input; and The image set is processed using a neural network, wherein the processing includes: Encode the image set into one or more features; The features are regressed into human body parameters using a human body mesh recovery multilayer perceptron (HMR MLP). A naked human body mesh is generated based on the human body parameters using a parametric human body model; Fine-tuning the neural network; and The query 3D ray is decoded into RGB colors and clothing-to-body displacement, and the clothed body mesh corresponding to the same vertices of the human body parameterization model is obtained based on the 3D displacement vector between the clothing and the naked body, wherein the RGB colors are based on the image set.

2. The method according to claim 1, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

3. The method according to claim 1, wherein the neural network selects a frontal view as a reference view from the image set and extracts feature bodies.

4. The method according to claim 3, wherein the neural network regresses all the feature bodies into human posture, shape, and facial expression parameters.

5. The method of claim 4, wherein the bare body mesh is converted into an occupancy field within a bounding box.

6. The method of claim 5, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the neural network generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

7. The method of claim 6, wherein the appearance of the clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of the camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

8. The method of claim 1, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

9. A device for neural human prediction, the device comprising: Non-transitory memory, configured to store an application, the application being configured to: Obtain an image set as input; and The image set is processed using a neural network, wherein the processing includes: Encode the image set into one or more features; The features are regressed into human body parameters using a human body mesh recovery multilayer perceptron (HMR MLP). A naked human body mesh is generated based on the human body parameters using a parametric human body model; Fine-tuning the neural network; and The query 3D ray is decoded into RGB colors and clothing-to-body displacement, and a clothed body mesh corresponding to the same vertices of the human body parameterization model is obtained based on the 3D displacement vector between the clothing and the naked body, wherein the RGB colors are based on the image set; and A processor configured to process the application.

10. The apparatus of claim 9, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

11. The apparatus of claim 9, wherein the neural network selects a frontal view as a reference view from the image set and extracts feature bodies.

12. The apparatus of claim 11, wherein the neural network regresses all the features into human posture, morphology, and facial expression parameters.

13. The apparatus of claim 12, wherein the bare body mesh is converted into an occupancy field within a bounding frame.

14. The apparatus of claim 13, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the neural network generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

15. The apparatus of claim 14, wherein the appearance of the clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of the camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

16. The apparatus of claim 9, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

17. A device for neural human prediction, the device comprising: Non-transitory memory, configured to store applications, the applications including: A multi-view stereo 3D convolutional neural network (MVS-3DCNN) is configured to encode an input image set into features. A human mesh recovery multilayer perceptron (HMR MLP) is configured to regress the features into human body parameters and generate a naked human body mesh based on the human body parameters using a human body parameterization model; and A Neural Radiation Field Multilayer Perceptron (NeRF MLP) is configured to fine-tune the MVS-3DCNN to decode query 3D rays into RGB colors and clothing-to-body displacements, and to obtain a clothed body mesh corresponding to the same vertices of the human body parameterization model based on the 3D displacement vector between the clothing and the naked body. A processor configured to process the application.

18. The apparatus of claim 17, wherein the image set comprises a 4D tensor of size N×w×h×c, where N is the number of views, w is the width of the image, h is the height of the image, and c is the number of channels of the image.

19. The apparatus of claim 18, wherein the MVS-3DCNN selects a frontal view as a reference view from the image set and extracts feature volumes.

20. The apparatus of claim 19, wherein the HMR MLP regresses all the features into human posture, morphology, and facial expression parameters.

21. The apparatus of claim 20, wherein the bare body mesh is converted into an occupancy field within a bounding frame.

22. The apparatus of claim 21, wherein for any 3D point near the body grid and associated with a ray direction from the center of each view, the NeRF MLP generates the RGB color and a 3D displacement vector pointing toward the surface of the naked body.

23. The apparatus of claim 22, wherein the appearance of the clothed human body is rendered as an RGB image by querying all rays emitted from all pixels of the camera view, and a clothed body mesh is obtained by deforming the naked body using the 3D displacement vector from the sampled points.

Images