Human body three-dimensional reconstruction system and method based on Mip-NeRF and skin model

Through the fusion of Mip-NeRF and SMPL models, a monocular camera or mobile phone collects image information to generate a human body three-dimensional model with texture details, solving the problems of high equipment costs, blurred textures and poor posture robustness in the existing technology, realizing the Efficient and realistic three-dimensional reconstruction and animation drive of human body.

CN119027559APending Publication Date: 2024-11-26NORTHEASTERN UNIV CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411150466.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction technology of human body has problems such as high equipment cost, blurred texture information, slow speed and poor posture robustness. In particular, the method based on parameterized models lacks clothing texture details, and the method based on implicit geometric representation is greatly affected by the environment. And the non-rigid changes have obvious effects.

Method used

Using a three-dimensional reconstruction system of human body based on Mip-NeRF and skin model, the image information around the human body is collected through a monocular camera or mobile phone, and the Mip-NeRF and SMPL model is fused to generate a three-dimensional model with texture details and 3D motion data is used to Drive animation generation to reduce the impact of non-rigid changes and improve the sense of reality.

Benefits of technology

It realizes low-cost and high-precision three-dimensional reconstruction of the human body, with texture details such as clothes and hair, and is fast, suitable for virtual fittings and special effects movies, reducing the problem of robustness reduction caused by image aliasing phenomenon and non-rigid changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027559A_ABST
    Figure CN119027559A_ABST
Patent Text Reader

Abstract

The invention discloses a human body three-dimensional reconstruction system and method based on Mip-NeRF and a skin model, and relates to the technical field of human body model reconstruction. According to the human body three-dimensional reconstruction system and method based on the Mip-NeRF and the skin model, only a single camera or a mobile phone is needed to collect image information around the human body, and the human body three-dimensional model can be reconstructed by using the collected image information around the human body and the method, so that the influence caused by non-rigid change of the human body can be weakened; the reconstructed human body three-dimensional model has texture details such as clothes and hair; the reconstructed human body three-dimensional model can be edited and operated, the reconstructed human body three-dimensional model is driven by the 3D motion data to generate a human body animation with a high sense of reality, specific actions which cannot be completed by the target human body are completed, and the method is applied to the fields of virtual fitting, special effect movies and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of human body model reconstruction, and in particular relates to a human body three-dimensional reconstruction system and method based on Mip-NeRF and a skin model. Background Art

[0002] Human body model reconstruction, also known as human body digitization, involves geometric reconstruction and texture estimation of a specific person, so that the person's true digital image can be reproduced in the computer. The technology of digitizing the human body movements of the target person in the video has great potential in various applications including human-computer interaction, social artificial intelligence and robotics. Combined with a variety of technologies, it can be widely used in remote office, online communication, virtual anchors, virtual fitting and other fields.

[0003] In recent years, researchers have been continuously innovating in methods of human body reconstruction. Reconstructing the human body by collecting multi-angle videos of people is one of the most important research directions today.

[0004] 1) Thanks to the continuous improvement of depth perception technology, the human body is reconstructed using visual devices or scanning devices, such as BodyNet, HybridFusion and other methods. Some of the above methods are based on real-time scanning fusion of high-precision equipment, some are based on free geometric expression, and some use machine learning, deep learning and other learning-based methods to estimate given images. Although high-precision real-time scanning equipment can directly collect detailed 3D human body data to build a complete model by using instruments such as depth sensors or precision scanners, such methods are expensive, which is unaffordable for ordinary consumers, and the texture information is fuzzy;

[0005] 2) Human body reconstruction technology based on parametric models, such as SMPL, SCAPE and other methods. The 3D human body model corresponding to the posture of the reconstructed human body is a gray parametric 3D mesh defined as naked, lacking clothing texture detail information. Even if clothing texture is attached to the parametric 3D mesh model through UV mapping, the reconstructed 3D model is not realistic enough due to the lack of clothing wrinkle information;

[0006] 3) Human body reconstruction based on neural implicit geometric representation, such as PIFu, PaMIR and other methods. The above methods use consumer-level passive to obtain human body image information. Although they can construct texture details, they are slow and are greatly affected by the environment. They cannot rule out non-rigid changes in the human body, and the robustness of human body posture is relatively poor. Summary of the invention

[0007] In view of the deficiencies of the above-mentioned prior art, the present invention provides a human body 3D reconstruction system and method based on Mip-NeRF and a skin model. The method only requires a single camera or mobile phone to collect image information around the human body. The collected image information around the human body and the above-mentioned method can be used to reconstruct a human body 3D model, which can not only reduce the impact of non-rigid changes in the human body, but also the reconstructed human body 3D model has texture details such as clothes and hair; the reconstructed human body 3D model can be edited, and the reconstructed human body 3D model can be driven by 3D motion data to generate a highly realistic human body animation, complete specific actions that the target human body itself cannot complete, and is used in the fields of virtual fitting, special effects movies, etc.

[0008] The technical solution of the present invention is:

[0009] On the one hand, the present invention provides a human body 3D reconstruction system based on Mip-NeRF and a skin model, comprising a human body surrounding video information collection unit, a human body image processing unit, a human body skin model generation unit, a human body reconstruction unit integrating Mip-NeRF and human body skin model, and a human body animation driving unit;

[0010] The human body surrounding video information collection unit is used to collect video information around the target person who is standing in a specified posture;

[0011] The human body image processing unit is used to extract an image from the video information around the target person according to the set interval frame number, resize the image and perform a cutout operation on the image to remove the background, and only retain the human body image of the target person in the image;

[0012] The human body skin model generating unit is used to predict the target person's skin model according to the received human body image of the target person output by the human body image processing unit and save the model;

[0013] The human body reconstruction unit integrating Mip-NeRF and human body skin model is used to reconstruct the three-dimensional model of the target person;

[0014] The human body animation driving unit is used to receive the reconstructed three-dimensional model of the target person, drive the movement of the three-dimensional model of the target person, and synthesize the specific action animation of the target person;

[0015] Furthermore, the human body surrounding video information collection unit is a monocular camera;

[0016] Further, the human image processing unit includes a video image frame extraction processing unit, an image size resetting processing unit and a human image segmentation unit;

[0017] The video image frame extraction processing unit is used to set the interval frame number and perform frame extraction operation on the video information around the target person collected by the human body surrounding video information collection unit according to the set interval frame number, and convert the video information around the target person into an image;

[0018] The image resizing processing unit is used to resize the image obtained by frame extraction, and then name the image and save it in a folder;

[0019] The human body image segmentation unit is used to segment the target person in the resized image, remove the background, and only retain the human body image of the target person in the image;

[0020] Furthermore, the human image segmentation unit adopts the RVM model;

[0021] Further, the human body skin model generation unit includes a target person skin model prediction unit and a target person skin model storage unit;

[0022] The target person skin model prediction unit adopts the VIBE model to extract the target person skin model and parameters based on the target person's human body image from the human body image processing unit; the parameters include camera parameters, posture parameters and shape parameters;

[0023] The target person skin model storage unit is used to store the target person skin model obtained by the target person skin model prediction unit;

[0024] Furthermore, the target person skin model adopts the SMPL model;

[0025] Further, the human body reconstruction unit integrating Mip-NeRF and human body skin model includes an animated posture-guided human body deformation unit, a posture optimization unit and an anti-aliasing neural radiation field model unit;

[0026] The posture optimization unit is used to adjust the posture parameters and shape parameters of the target character skin model;

[0027] The animatable pose-guided human deformation unit explicitly guides the deformation of the 3D positions of vertices in the target character skin model by inversely transforming the adjusted target character skin model, converting the 3D positions of the vertices from the observation space to the canonical space, thereby reducing pose dependency;

[0028] The 3D position is Where x, y, and z represent the three-dimensional position information of the observation space of the monocular camera used to collect the video sequence. The observation space is the space composed of the image observed at a set viewing angle or position, that is, the world coordinate system where the monocular camera is located;

[0029] The specification space is a predefined standardized space;

[0030] The anti-aliasing neural radiation field model unit introduces the adjusted target character skin model into Mip-NeRF and presents an animated neural radiation field;

[0031] Further, the anti-aliasing neural radiation field model unit includes an integrated position information encoding unit and a volume rendering unit;

[0032] The integrated position information encoding unit is used to perform a dimensionality-up operation on the 3D positions of the vertices in the target character skin model;

[0033] The volume rendering unit is used to project conical light using volume rendering technology, integrate the color and density at the 3D position of the vertex in the target person's skin model, and convert it into a visual image to obtain a reconstructed three-dimensional model of the target person;

[0034] Further, the human animation driving unit includes a human motion library and a motion fitting unit;

[0035] The human action library stores the action templates of the target person's skin model;

[0036] The action fitting unit is used to fit the action template in the action library with the three-dimensional model of the target person, so that the three-dimensional model of the target person performs the same action;

[0037] On the other hand, the present invention also provides a method for 3D reconstruction of a human body based on Mip-NeRF and a skin model, comprising the following steps:

[0038] Step 1: Make the target person stand still in a specified posture in the set scene, and set the position of the monocular camera so that the target person is completely within the shooting range of the monocular camera;

[0039] Step 2: Place the monocular camera around the target person to collect video information around the target person; the video information around the target person is represented as a video information sequence Among them I t is the image corresponding to the tth moment in the video information sequence, t is the moment, and n is the total number of moments;

[0040] Step 3: Set the interval frame number, extract the image from the video information around the target person according to the set interval frame number, then reset the size of the extracted image, and then save the image in a folder;

[0041] Step 4: Use the RVM model to perform a cutout operation on the image saved in step 3 to remove the background and retain only the human body image of the target person in the image;

[0042] Step 5: Use the VIBE model to predict the SMPL model based on the human body image of the target person, obtain the SMPL model and parameters corresponding to each human body image of the target person, and save the predicted SMPL model; the parameters include camera parameters, posture parameters and shape parameters;

[0043] Step 6: Construct a human body reconstruction model that integrates Mip-NeRF and human skin model to perform human body reconstruction;

[0044] S1: The video sequence obtained in step 3 Perform camera pose matching to obtain the 3D position of the vertices in the SMPL model in the observation space

[0045] S2: Inverse transform the SMPL model predicted in step 5 to make the 3D positions of the vertices in the SMPL model explicit and guided The deformation of the 3D position in the observation space Transform from observation space to normative space;

[0046] The 3D position in the observation space The method to transform from observation space to norm space is:

[0047]

[0048]

[0049] in, is the 3D position in the canonical space, x0, y0, z0 represent the three-dimensional position information of the canonical space respectively, To transform the 3D position in the observation space The mapping function to the canonical space, θ 0 To standardize the posture parameters of the space, The distance in the observation space The set of vertices of the SMPL model with set distance, v i for The vertex i in is the vertex number, ω i For vertex v i 3D Position The weight of the influence of ω is the influence of all vertices on the 3D position The sum of the weights of the influence of i is the posture transformation matrix;

[0050]

[0051]

[0052] in, For vertex v i The mixing weights, For vertex v i The blended skin weights, For calculation and v i The L2 distance, is a trainable parameter;

[0053] S3: Use integrated position coding to encode the 3D position converted to the canonical space, then use volume rendering technology to project cone light, integrate the color and density at each 3D position, render it into a 2D image, and then obtain the reconstructed 3D model of the target person;

[0054] The rendering formula is:

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] in, is the cumulative value of the color along the conical ray r, r is the projected conical ray, k is the number of the sampling points evenly divided by the projected conical ray, and N is the total number of evenly divided sampling points of the projected conical ray; is the 3D position in the canonical space, The projected cone of light at the 3D position The density value, δ k =||x k+1 -x k ||2 is the distance between adjacent sampling points, The projected cone of light is at the 3D position The color value of is the cumulative value of the density along the projected cone of rays r, The distance in the observation space The set of vertices of the SMPL model with set distances, is the 3D position in the observation space arrive is the weighted distance of the vertices in , δ is the distance threshold that limits the distance between the sampling points in the observation space and the surface of the SMPL model;

[0061] Step 7: training the human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a trained human body reconstruction model that integrates Mip-NeRF and the human body skin model, and reconstructing the target person using the trained human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a reconstructed 3D model of the target person;

[0062] The human body reconstruction model integrating Mip-NeRF and human body skin model is trained, specifically adjusting the posture parameter θ l and shape parameter β l To Color and density σ;

[0063]

[0064] in, is the color, r is the value of the pixel in the red channel, g is the value of the pixel in the green channel, b is the value of the pixel in the blue channel, mF is Mip-NeRF, For the general Mapping function for dimensionality increase;

[0065] The loss function L of the training process is:

[0066] L=L c +L p +λ d *L d

[0067] Among them, L c is the rendering pixel difference, L p is the attitude constraint term, L d Background constraint, λ d is a random weight parameter;

[0068] Step 8: Build an SMPL model action library, use the actions in the SMPL model action library to drive the reconstructed 3D model of the target character, and synthesize new action animations;

[0069] Step 8.1: Obtain action videos, use the VIBE model to convert the action videos into SMPL model action video sequences, save the SMPL model action video sequences as action templates, and obtain the SMPL model action library;

[0070] Step 8.2: Set the camera parameter K of the action template in the action library constructed in step 8.1 l , attitude parameter θ l and shape parameter β l The target person's 3D model is transferred to the reconstructed model to synthesize a new action animation of the target person;

[0071] Step 9: Visualize the reconstructed 3D model of the target person and the synthesized new motion animation of the target person.

[0072] Compared with the prior art, the beneficial effects of the present invention are:

[0073] (1) Compared with the traditional human body reconstruction method, the human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention does not require a multi-view camera, but only uses a single image collection device to collect images around the human body. Even common devices with video recording functions such as mobile phones can be used, which is more suitable for life and provides the possibility for the practical application of human body modeling.

[0074] (2) The human body 3D reconstruction system and method based on Mip-NeRF and skin model provided by the present invention has a simple operation process, which is convenient for users to realize the reconstruction of the target person and improves the generalizability of human body model reconstruction;

[0075] (3) The human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention can reconstruct a human body model with texture details such as hair and clothes. The reconstructed human body model is more realistic and has higher accuracy.

[0076] (4) The human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention are fast and adopt a single-stage training strategy to reduce training time and improve reconstruction speed;

[0077] (5) The human body reconstructed by the system and method for 3D reconstruction of human body based on Mip-NeRF and skin model described in the present invention can be driven by actions;

[0078] (6) The human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention integrates Mip-NeRF model and SMPL model for reconstruction, which can effectively reduce the appearance of virtual images in the image caused by image aliasing, and reduce the problem of reduced robustness caused by non-rigid changes of the human body in the human body reconstruction field;

[0079] (7) The human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention can timely supplement and update the action library according to the action that the target person wants to complete;

[0080] (8) The human body 3D reconstruction system and method based on Mip-NeRF and skin model described in the present invention utilizes existing open source models to construct an action library, so that the reconstructed static human body model can realize animation operation, providing the possibility for high-difficulty action synthesis in the film and television industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a structural diagram of a human body 3D reconstruction system based on Mip-NeRF and a skin model in an embodiment of the present invention;

[0082] Figure 2 is a flow chart of a method for 3D reconstruction of a human body based on Mip-NeRF and a skin model in an embodiment of the present invention;

[0083] Figure 3 The target person's standard posture, camera sight range and camera running trajectory diagram of the embodiment of the present invention;

[0084] Figure 4 This is a structural diagram of the RVM model in an embodiment of the present invention;

[0085] Figure 5 It is a structural diagram of a VIBE model of a human body three-dimensional reconstruction method based on anti-aliasing neural radiation field and skin model in an embodiment of the present invention;

[0086] Figure 6 This is a schematic diagram of the SMPL model in an embodiment of the present invention;

[0087] Among them, 0-pelvis (reference point); 1-left hip; 2-right hip; 3-spine 1; 4-left knee; 5-right knee; 6-spine 2; 7-left ankle; 8-right ankle; 9-spine 3; 10-left foot; 11-right foot; 12-neck; 13-left chest; 14-right chest; 15-head; 16-left shoulder; 17-right shoulder; 18-left elbow; 19-right elbow; 20-left wrist; 21-right wrist; 22-left hand; 23-right hand;

[0088] Figure 7 It is an animated Mip-NeRF flowchart of a method for 3D reconstruction of a human body based on anti-aliasing neural radiation fields and a skin model in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0090] A human 3D reconstruction system based on Mip-NeRF and skin model, such as Figure 1 As shown, it includes a human body surrounding video information collection unit, a human body image processing unit, a human body skin model generation unit, a human body reconstruction unit integrating Mip-NeRF and human body skin model, and a human body animation driving unit;

[0091] The human body surrounding video information collection unit is used to collect video information around the target person who is standing in a specified posture;

[0092] The unit for collecting video information around the human body is a monocular camera;

[0093] In this embodiment, the human body surrounding video information collection unit needs to be placed in an environment with sufficient ambient light and a relatively simple background to ensure the quality of the collected video information;

[0094] The human body image processing unit is used to extract an image from the video information around the target person according to the set interval frame number, resize the image and perform a cutout operation on the image to remove the background, and only retain the human body image of the target person in the image;

[0095] The human body image processing unit includes a video image frame extraction processing unit, an image size resetting processing unit and a human body image segmentation unit;

[0096] The video image frame extraction processing unit is used to set the interval frame number and perform frame extraction operation on the video information around the target person collected by the human body surrounding video information collection unit according to the set interval frame number, and convert the video information around the target person into an image;

[0097] The image resizing processing unit is used to resize the image obtained by frame extraction, and then name the image and save it in a folder;

[0098] The human body image segmentation unit is used to segment the target person in the resized image, remove the background, and only retain the human body image of the target person in the image;

[0099] The human image segmentation unit adopts the RVM (Robust Video Matting) model to achieve human image matting;

[0100] The human body skin model generating unit is used to predict the target person's skin model according to the received human body image of the target person output by the human body image processing unit and save the model;

[0101] The human body skin model generation unit includes a target person skin model prediction unit and a target person skin model storage unit;

[0102] The target person skin model prediction unit adopts the VIBE (Video Inference for Human Body Pose and Shape Estimation) model to extract the target person skin model and parameters based on the human body image of the target person after the human body is cut out from the human body image processing unit; the parameters include camera parameters, posture parameters and shape parameters;

[0103] The target person skin model storage unit is used to store the target person skin model obtained by the target person skin model prediction unit;

[0104] The target person skin model adopts the SMPL (A Skinned Multi-Person Linear Model) model. The human body can be understood as the sum of a basic model and deformations based on the model. PCA (Principle component analysis) is performed on the basis of the deformation to obtain the low-dimensional parameters describing the shape, namely the shape parameters (shape). At the same time, the motion tree is used to represent the posture of the human body, that is, the rotation relationship between each joint point and the parent node of the motion tree. This relationship can be expressed as a three-dimensional vector. Finally, the local rotation vector of each joint point constitutes the posture parameter (pose) of the SMPL model.

[0105] The SMPL model is a parametric model with 6890 vertices, which can be divided into posture parameters and body shape parameters. The posture parameters are composed of 24 key point vectors, including the position vectors of 23 human joints and 1 human orientation vector. The body shape parameters are composed of 10 human principal components PCA (Principle component analysis) obtained by reducing the dimension of the body shape in the data set.

[0106] The human body reconstruction unit integrating Mip-NeRF and human body skin model is used to reconstruct the three-dimensional model of the target person;

[0107] The human body reconstruction unit integrating Mip-NeRF and human body skin model includes an animated posture-guided human body deformation unit, a posture optimization unit and an anti-aliasing neural radiation field (Mip-NeRF) model unit;

[0108] The posture optimization unit is used to adjust the posture parameters and shape parameters of the target character skin model;

[0109] The animatable pose-guided human deformation unit explicitly guides the deformation of the 3D positions of vertices in the target character skin model by inversely transforming the adjusted target character skin model, converting the 3D positions of the vertices from the observation space to the canonical space, thereby reducing pose dependency;

[0110] The 3D position is Where x, y, and z represent the three-dimensional position information of the observation space of the monocular camera used to collect the video sequence. The observation space is the space composed of the image observed at a set viewing angle or position, that is, the world coordinate system where the monocular camera is located;

[0111] The specification space is a predefined standardized space;

[0112] The anti-aliasing neural radiation field (Mip-NeRF) model unit introduces the adjusted target character skin model into Mip-NeRF and presents an animated neural radiation field;

[0113] The anti-aliasing neural radiation field (Mip-NeRF) model unit includes an integrated position information encoding unit and a volume rendering unit;

[0114] The integrated position information encoding unit is used to perform a dimensionality-up operation on the 3D positions of the vertices in the target person's skin model, so as to expand the neural field perception capability;

[0115] The volume rendering unit is used to project conical light using volume rendering technology, integrate the color and density at the 3D position of the vertex in the target person's skin model, convert it into a visual image, and obtain a reconstructed three-dimensional model of the target person;

[0116] The human body animation driving unit is used to receive the reconstructed three-dimensional model of the target person, drive the movement of the three-dimensional model of the target person, and synthesize the specific action animation of the target person;

[0117] The human animation driving unit includes a human motion library and a motion fitting unit;

[0118] The human action library stores the action templates of the target person's skin model;

[0119] The action templates in this embodiment are from the open source SMPL parameter model video sequence of mixamo motion capture, and based on the video data publicly available on the Internet or the action videos of other characters, the above videos are converted into the character SMPL model action video sequence using the VIBE model;

[0120] The action fitting unit is used to fit the action template in the action library with the three-dimensional model of the target person, so that the three-dimensional model of the target person performs the same action;

[0121] like Figure 2 As shown, the human body 3D reconstruction method based on Mip-NeRF and skin model includes the following steps:

[0122] Step 1: Figure 3 As shown, in a set scene, the target person is made to stand in a specified posture and remain still, and the position of the monocular camera is set so that the target person is completely within the shooting range of the monocular camera; the monocular camera is a mobile phone;

[0123] In this embodiment, in a place with good lighting, wide field of view and relatively simple background, the target person is made to stand in a specified posture, the monocular camera is set at a distance that can completely include the target person in the field of vision, and a mobile phone is selected as the image data collection device; the specified posture in this embodiment is that the target person's legs are slightly open, with a clear gap between the legs, the target person's arms are spread out to both sides, maintaining a certain angle with the body trunk, and a clear gap is left between the arms and the body trunk;

[0124] Step 2: Place the monocular camera around the target person and collect video information around the target person, which is represented as a video information sequence. Among them I t is the image corresponding to the tth moment in the video information sequence, t is the moment, and n is the total number of moments;

[0125] In this embodiment, the target person is held in the mobile phone and kept within the viewing range of the mobile phone to collect video information around the target person;

[0126] Step 3: Set the interval frame number, extract the image from the video information around the target person according to the set interval frame number, then reset the size of the extracted image, and then save the image in a folder;

[0127] In this implementation, the video information around the target person collected by the monocular camera is extracted at intervals of 20 frames according to the requirements, and the image is reset to 800*800 according to the actual requirements to ensure that the target person is included in the image as a whole, and the image information is saved. The images are named l∈{1,2,3…m} in sequence according to the video time sequence; where l is the image file name, m is the number of images, which is related to the total number n of video timestamps and the frame extraction interval, and the image file format is png;

[0128] Step 4: Use the RVM model to perform a cutout operation on the image saved in step 3 to remove the background and retain only the human body image of the target person in the image;

[0129] In this embodiment, in order to ensure the same timing as the video, the file name of the human body image obtained by cutting out is the same as that in step 3, that is, the file name of the human body image is the same as the file name of the corresponding image before the cutting out operation;

[0130] like Figure 4 As shown in the figure, the RVM model includes an encoder for extracting features of a single frame, a recursive decoder for summarizing temporal information, and a deep guided filter module for high-resolution upsampling. The above model uses temporal memory to process videos instead of processing each frame as an independent image, and no longer uses patch-based refinement, which improves temporal coherence and portrait cutout quality.

[0131] Step 5: Use the VIBE model to predict the SMPL model based on the human body image of the target person, obtain the SMPL model and parameters corresponding to each human body image of the target person, and save the predicted SMPL model; the parameters include the camera parameter K l , attitude parameter θ l and shape parameter β l ;

[0132] In this embodiment, the file name of the SMPL model is the same as the file name of the human body image of the target person l∈{1,2,3…m};

[0133] This implementation uses the VIBE model to estimate the SMPL model, such as Figure 5 As shown in the figure, the VIBE model does not need to use 3D groundtruth data, but directly extracts the human motion timing information from the unlabeled 2D image data. It uses the adversarial learning framework to distinguish the real human motion from the motion generated by the regression network, and adds an action discriminator to judge the gap between the predicted label and the actual label. It also introduces a self-attention mechanism to output the camera parameters and the posture parameters and shape parameters of the SMPL model.

[0134] The SMPL model file format is pkl, that is, the human SMPL model of the target person in the image file p.png is p.pkl, where p∈l;

[0135] SMPL model such as Figure 6 As shown, the SMPL model defines two parameters: shape parameter (β l ) and attitude parameters (θ l ) to control the body's movement and posture changes, θ l represents the posture parameter of the target person in the l-th frame image, β l represents the shape parameter of the target person in the l-th frame image, K l is the camera parameter, including the camera direction and camera starting point. Shape parameter (β l ) is 10 dimensions, and the posture parameter (θ l ) is a 24×3 dimension, where 24 represents 24 joints of the human body, including 0-pelvis (reference point); 1-left hip; 2-right hip; 3-spine 1; 4-left knee; 5-right knee; 6-spine 2; 7-left ankle; 8-right ankle; 9-spine 3; 10-left foot; 11-right foot; 12-neck; 13-left chest; 14-right chest; 15-head; 16-left shoulder; 17-right shoulder; 18-left elbow; 19-right elbow; 20-left wrist; 21-right wrist; 22-left hand; 23-right hand.

[0136] Step 6: Figure 7 As shown, a human body reconstruction model integrating Mip-NeRF and human body skin model is constructed to perform human body reconstruction;

[0137] For traditional NeRF, blur and jagged problems will occur because a very narrow ray is sampled for each pixel. In addition, the traditional NeRF is divided into two stages: coarse sampling and fine sampling, with high sampling cost and slow training speed. Mip-NeRF is an anti-aliasing multi-scale neural radiation field, which uses a cone to replace the ray, and the sampling only goes through a single stage, with low sampling cost, fast training speed and good reconstruction effect. The process of human body reconstruction by integrating Mip-NeRF and human skin model is as follows:

[0138] S1: The video sequence obtained in step 3 Perform camera pose matching to obtain the 3D position of the vertices in the SMPL model in the observation space

[0139] S2: Inverse transform the SMPL model predicted in step 5 to make the 3D positions of the vertices in the SMPL model explicit and guided The deformation of the 3D position in the observation space Transform from observation space to normative space;

[0140] The SMPL model supports explicit transformation of spatial points (i.e., observation space to canonical space), reduces the dependence on input poses, and generalizes to invisible poses, thereby learning Mip-NeRF space from dynamic target characters and converting 3D positions in observation space to The method to transform from observation space to norm space is:

[0141]

[0142]

[0143] in, is the 3D position in the canonical space, x0, y0, z0 represent the three-dimensional position information of the canonical space respectively, To transform the 3D position in the observation space The mapping function to the canonical space, θ 0 To standardize the posture parameters of the space, The distance in the observation space The set of vertices of the SMPL model with set distance, v i for The vertex i in is the number of the vertex. In this implementation, i∈(1,2…6890), ω i For vertex v i 3D Position The weight of the influence of ω is the influence of all vertices on the 3D position The sum of the weights of the influence of i is the posture transformation matrix;

[0144]

[0145]

[0146] in, For vertex v i The mixing weights, For vertex v i The blended skin weights, For calculation and v i The L2 distance, is a trainable parameter;

[0147] S3: Use integrated position coding to encode the 3D position converted to the standard space, then use volume rendering technology to project cone light, integrate the color and density at each 3D position in a single stage with high precision, render it into a 2D image, and then obtain the reconstructed 3D model of the target person;

[0148] The rendering formula is:

[0149]

[0150]

[0151]

[0152]

[0153]

[0154] in, is the cumulative value of the color along the conical ray r, r is the projected conical ray, k is the sampling point number of the projected conical ray, N is the total number of sampling points of the projected conical ray, and the default setting is 128; is the 3D position in the canonical space, Used to set the distance x from the SMPL model surface k The density of distant points is set to 0. The projected cone of light is at the 3D position The density value, δ k =||x k+1 -x k ||2 is the distance between adjacent sampling points, The projected cone of light is at the 3D position The color value of is the cumulative value of the density along the projected cone of rays r, The distance in the observation space The set of vertices of the SMPL model with set distances, is the 3D position in the observation space arrive is the weighted distance of the vertices in , δ is the distance threshold that limits the distance between the sampling points in the observation space and the surface of the SMPL model;

[0155] Step 7: training the human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a trained human body reconstruction model that integrates Mip-NeRF and the human body skin model, and reconstructing the target person using the trained human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a reconstructed 3D model of the target person;

[0156] The human body reconstruction model integrating Mip-NeRF and human body skin model is trained, specifically adjusting the posture parameter θ l and shape parameter β l To Color and density σ;

[0157]

[0158] in, is the color, r is the value of the pixel in the red channel, g is the value of the pixel in the green channel, b is the value of the pixel in the blue channel, mF is Mip-NeRF, For the general A mapping function that performs a dimensionality-raising operation to handle human motion between frames;

[0159] The loss function L of the training process is:

[0160] L=L c +L p +λ d *L d

[0161] Among them, L c is the rendering pixel difference, L p is the attitude constraint term, L d Background constraint, λ d is a random weight parameter used to balance the importance of background constraints;

[0162] Step 8: Build an SMPL model action library, use the actions in the SMPL model action library to drive the reconstructed 3D model of the target character, and synthesize new action animations;

[0163] Step 8.1: Obtain action videos, use the VIBE model to convert the action videos into SMPL model action video sequences, save the SMPL model action video sequences as action templates, and obtain the SMPL model action library;

[0164] This embodiment uses the open source SMPL parameter model of mixamo motion capture to build an action library, which is composed of a continuous character SMPL model action video sequence; wherein mixamo is an online 3D character animation production platform based on the web version; at the same time, based on the video data publicly available on the Internet or the action videos of other characters, the above videos can be converted into character SMPL model action video sequences and added to the action library to add new actions;

[0165] Step 8.2: Since the reconstructed target character model is deformed explicitly through SMPL, it is possible to synthesize images even with non-existent poses. The structural latent code is used to process the geometry of more complex cloths, making the model more realistic. The camera parameters K of the action template in the action library constructed in step 8.1 are l , attitude parameter θ l and shape parameter β l The target person's 3D model is transferred to the reconstructed model to synthesize a new action animation of the target person;

[0166] Step 9: Visualize the reconstructed 3D model of the target person and the synthesized new motion animation of the target person.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A human body 3D reconstruction system based on Mip-NeRF and skin model, characterized in that: It includes a human body surrounding video information collection unit, a human body image processing unit, a human body skin model generation unit, a human body reconstruction unit integrating Mip-NeRF and human body skin model, and a human body animation driving unit; The human body surrounding video information collection unit is used to collect video information around the target person who is standing in a specified posture; The human body image processing unit is used to extract an image from the video information around the target person according to the set interval frame number, resize the image and perform a cutout operation on the image to remove the background, and only retain the human body image of the target person in the image; The human body skin model generating unit is used to predict the target person's skin model according to the received human body image of the target person output by the human body image processing unit and save the model; The human body reconstruction unit integrating Mip-NeRF and human body skin model is used to reconstruct the three-dimensional model of the target person; The human body animation driving unit is used to receive the reconstructed three-dimensional model of the target person, drive the movement of the three-dimensional model of the target person, and synthesize the specific action animation of the target person.

2. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 1, characterized in that: The unit for collecting video information around the human body is a monocular camera.

3. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 1, characterized in that: The human body image processing unit includes a video image frame extraction processing unit, an image size resetting processing unit and a human body image segmentation unit; The video image frame extraction processing unit is used to set the interval frame number and perform frame extraction operation on the video information around the target person collected by the human body surrounding video information collection unit according to the set interval frame number, and convert the video information around the target person into an image; The image resizing processing unit is used to resize the image obtained by frame extraction, and then name the image and save it in a folder; The human body image segmentation unit is used to segment the target person in the resized image, remove the background, and only retain the human body image of the target person in the image; The human image segmentation unit adopts the RVM model.

4. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 1, characterized in that: The human body skin model generation unit includes a target person skin model prediction unit and a target person skin model storage unit; The target person skin model prediction unit adopts the VIBE model to extract the target person skin model and parameters based on the target person's human body image from the human body image processing unit; the parameters include camera parameters, posture parameters and shape parameters; The target person skin model storage unit is used to store the target person skin model obtained by the target person skin model prediction unit; The target character skin model adopts the SMPL model.

5. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 1, characterized in that: The human body reconstruction unit integrating Mip-NeRF and human body skin model includes an animated posture-guided human body deformation unit, a posture optimization unit and an anti-aliasing neural radiation field model unit; The posture optimization unit is used to adjust the posture parameters and shape parameters of the target character skin model; The animatable pose-guided human deformation unit explicitly guides the deformation of the 3D positions of vertices in the target character skin model by inversely transforming the adjusted target character skin model, converting the 3D positions of the vertices from the observation space to the canonical space, thereby reducing pose dependency; The 3D position is Where x, y, and z represent the three-dimensional position information of the observation space of the monocular camera used to collect the video sequence. The observation space is the space composed of the image observed at a set viewing angle or position, that is, the world coordinate system where the monocular camera is located; The specification space is a predefined standardized space; The anti-aliasing neural radiation field model unit introduces the adjusted target character skin model into Mip-NeRF and presents an animated neural radiation field.

6. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 5, characterized in that: The anti-aliasing neural radiation field model unit includes an integrated position information encoding unit and a volume rendering unit; The integrated position information encoding unit is used to perform a dimensionality-up operation on the 3D positions of the vertices in the target character skin model; The volume rendering unit is used to project conical light using volume rendering technology, integrate the color and density at the 3D position of the vertices in the target person's skin model, convert it into a visual image, and obtain a reconstructed three-dimensional model of the target person.

7. The human body 3D reconstruction system based on Mip-NeRF and skin model according to claim 1, characterized in that: The human animation driving unit includes a human motion library and a motion fitting unit; The human action library stores the action templates of the target person's skin model; The action fitting unit is used to fit the action template in the action library with the three-dimensional model of the target person, so that the three-dimensional model of the target person performs the same action.

8. A method for 3D reconstruction of a human body based on Mip-NeRF and a skin model implemented by a 3D reconstruction system of a human body based on Mip-NeRF and a skin model according to claim 1, characterized in that: The steps include: Step 1: Make the target person stand still in a specified posture in the set scene, and set the position of the monocular camera so that the target person is completely within the shooting range of the monocular camera; Step 2: Place the monocular camera around the target person to collect video information around the target person; The video information around the target person is represented as a video information sequence Among them I t is the image corresponding to the tth moment in the video information sequence, t is the moment, and n is the total number of moments; Step 3: Set the interval frame number, extract the image from the video information around the target person according to the set interval frame number, then reset the size of the extracted image, and then save the image in a folder; Step 4: Use the RVM model to perform a cutout operation on the image saved in step 3 to remove the background and retain only the human body image of the target person in the image; Step 5: Use the VIBE model to predict the SMPL model based on the human body image of the target person, obtain the SMPL model and parameters corresponding to each human body image of the target person, and save the predicted SMPL model; the parameters include camera parameters, posture parameters and shape parameters; Step 6: Construct a human body reconstruction model that integrates Mip-NeRF and human skin model to perform human body reconstruction; Step 7: training the human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a trained human body reconstruction model that integrates Mip-NeRF and the human body skin model, and reconstructing the target person using the trained human body reconstruction model that integrates Mip-NeRF and the human body skin model to obtain a reconstructed 3D model of the target person; The human body reconstruction model integrating Mip-NeRF and human body skin model is trained, specifically adjusting the posture parameter θ l and shape parameter β l To Color and density σ; in, is the color, r is the value of the pixel in the red channel, g is the value of the pixel in the green channel, b is the value of the pixel in the blue channel, mF is Mip-NeRF, For the general Mapping function for dimensionality increase; The loss function L of the training process is: L=L c +L p +λ d *L d Among them, L c is the rendering pixel difference, L p is the attitude constraint term, L d Background constraint, λ d is a random weight parameter; Step 8: Build an SMPL model action library, use the actions in the SMPL model action library to drive the reconstructed 3D model of the target character, and synthesize new action animations; Step 9: Visualize the reconstructed 3D model of the target person and the synthesized new motion animation of the target person.

9. The method for 3D reconstruction of a human body based on Mip-NeRF and a skin model according to claim 8, characterized in that: The process of reconstructing the human body by fusing the human body reconstruction model of Mip-NeRF and the human body skin model in step 6 is as follows: S1: The video sequence obtained in step 3 Perform camera pose matching to obtain the 3D position of the vertices in the SMPL model in the observation space S2: Inverse transform the SMPL model predicted in step 5 to make the 3D positions of the vertices in the SMPL model explicit and guided The deformation of the 3D position in the observation space Transform from observation space to normative space; The 3D position in the observation space The method to transform from observation space to norm space is: in, is the 3D position in the canonical space, x0, y0, z0 represent the three-dimensional position information of the canonical space respectively, To transform the 3D position in the observation space The mapping function to the canonical space, θ 0 To standardize the posture parameters of the space, The distance in the observation space The set of vertices of the SMPL model with set distance, v i for The vertex i in is the vertex number, ω i For vertex v i 3D Position The weight of the influence of ω is the influence of all vertices on the 3D position The sum of the weights of the influence, M i is the posture transformation matrix; in, For vertex v i The mixing weights, For vertex v i The blended skin weights, For calculation and v i The L2 distance, is a trainable parameter; S3: Use integrated position coding to encode the 3D position converted to the canonical space, then use volume rendering technology to project cone light, integrate the color and density at each 3D position, render it into a 2D image, and then obtain the reconstructed 3D model of the target person; The rendering formula is: in, is the cumulative value of the color along the conical ray r, r is the projected conical ray, k is the number of the sampling points evenly divided by the projected conical ray, and N is the total number of evenly divided sampling points of the projected conical ray; is the 3D position in the canonical space, The projected cone of light at the 3D position The density value, δ k =||x k+1 -x k ||2 is the distance between adjacent sampling points, The projected cone of light at the 3D position The color value of is the cumulative value of the density along the projected cone of rays r, The distance in the observation space The set of vertices of the SMPL model with set distances, is the 3D position in the observation space arrive is the weighted distance of the vertices in , and δ is the distance threshold that limits the distance between the sampling points in the observation space and the SMPL model surface.

10. The method for 3D reconstruction of a human body based on Mip-NeRF and a skin model according to claim 8, characterized in that: Step 8 specifically includes: Step 8.1: Obtain action videos, use the VIBE model to convert the action videos into SMPL model action video sequences, save the SMPL model action video sequences as action templates, and obtain the SMPL model action library; Step 8.2: Set the camera parameter K of the action template in the action library constructed in step 8.1 l , attitude parameter θ l and shape parameter β l The data is passed to the reconstructed 3D model of the target person to synthesize a new action animation of the target person.