Neural radiation field three-dimensional human body reconstruction method and system considering motion capture enhancement and importance sampling

Abstract

The invention discloses a neural radiation field three-dimensional human body reconstruction method and system considering motion capture enhancement and importance sampling, and the method comprises the steps: collecting a human body RGB image and skeleton data, and carrying out the frame-to-frame alignment in time and space; human body key point space coordinates are mapped to obtain SMPL posture parameters, human body posture data are used as prior, a posture driving deformation field is based on a linear hybrid skin algorithm, a hybrid weight field is combined with a human body skeleton, accurate mapping from an observation posture to a standard posture is achieved, and a standard model of a human body is obtained; based on the RGB data and the attitude data, importance sampling of volume rendering is realized by estimating camera ray transmissivity distribution in a volume rendering process; and finally, predicting the volume density and color value of any point and any direction in the space based on a neural radiation field method, and realizing human body rendering by using a volume rendering method. According to the method, the reconstruction quality of the dynamic human body is improved, a virtual human in a new posture can be generated, and action editing of a reconstruction object is realized.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional human body reconstruction and rendering, and in particular to a method and system for three-dimensional human body reconstruction using a neural radiation field taking motion capture enhancement and importance sampling into consideration. Background Art

[0002] 3D human reconstruction aims to recover the 3D geometry and appearance of the human body from images, videos and depth data. As an important research direction in the fields of human-computer interaction, virtual reality and film and television animation, 3D human reconstruction methods have been widely studied for decades. Traditional 3D human reconstruction methods are based on complex multi-camera optical systems. After data acquisition, view estimation, mesh generation and texture mapping, a high-precision digital human model is obtained. The high cost, limited acquisition environment and complex and time-consuming processing procedures make it difficult for traditional 3D human reconstruction methods to be applied in daily life scenes. With the popularization of applications such as remote office, online teaching, virtual fitness and virtual try-on, how to effectively use computing resources to achieve high-quality 3D human reconstruction on consumer devices such as personal computers, smartphones and VR / AR glasses has become a problem that needs to be solved urgently in academia and industry. In recent years, with the rapid development of deep learning, 3D human reconstruction technology has made great progress. Neural Radiance Fields (NeRF) is an emerging 3D reconstruction technology. It uses deep neural networks to fit the color and volume density information in the scene, uses a differentiable volume rendering process to generate images, and optimizes the rendering error between the synthetic image and the real image to achieve high-precision reconstruction. It also realizes the modeling and rendering of dynamic human bodies from static high-fidelity rendering. Although the reconstruction method of dynamic human bodies based on neural radiation fields can achieve good rendering quality under existing postures, it is still expected to achieve better rendering quality. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for three-dimensional human body reconstruction of a neural radiation field taking into account motion capture enhancement and importance sampling, which adds skeletal information collected by an optical motion capture device to a data acquisition module, thereby achieving more accurate learning of human body posture; an importance sampling method is used to select sampling points in a valid area, and the volume density and color values ​​of the sampling points in any direction in space are predicted; a volume rendering method is used to achieve human body rendering, thereby improving the reconstruction quality of a dynamic human body.

[0004] The technical solution to achieve the purpose of the present invention is as follows: In the first aspect, the present invention provides a method for reconstructing a three-dimensional human body using a neural radiation field taking into account motion capture enhancement and importance sampling, comprising the steps of:

[0005] Collect human RGB images and skeleton data;

[0006] Image preprocessing: synchronize multimodal data and achieve time alignment for each frame.

[0007] Map the skeleton data to the SMPL human body model, calculate the SMPL posture parameters, use the human body posture parameters and RGB images as the input of the human body deformation network, divide the human body movement into rigid deformation and non-rigid deformation, and realize the modeling of the standard human body model;

[0008] In the sampling process, stratified importance sampling is used, and uniform sampling is combined with importance sampling to select more effective sampling points;

[0009] Given a standard human body model, the three-dimensional coordinates of the position-encoded sampling points are input into the body density prediction network, and the body density and feature vector are output;

[0010] The feature vector and the position-encoded viewing angle information are input into the color prediction network, and the color value of the corresponding sampling point is output;

[0011] Using volume rendering methods, the pixel values ​​of images of the human body at any viewing angle are predicted to achieve photorealistic rendering.

[0012] Furthermore, the multi-view RGB images are evenly distributed and collected by multiple Azure Kinect cameras.

[0013] Furthermore, the human body posture information is collected in spatial coordinates by an OptiTrack optical motion capture camera through reflection of infrared light.

[0014] Furthermore, the RGB image has a height of 1080 pixels and a width of 1920 pixels.

[0015] Furthermore, the rigid deformation is implemented based on a linear hybrid skinning algorithm, which maps points in the observed posture into a canonical space:

[0016]

[0017] Among them, x represents the three-dimensional coordinates of the query point before transformation, ω o (x) is the mixing weight function defined in the observation space, G k is the transformation matrix generated by the kth part of the human skeleton definition, K = 24; after the transformation, the query point x' in the standard space is obtained;

[0018] Furthermore, the non-rigid deformation network learns personalized human details by introducing a displacement field related to the pose, which is expressed as:

[0019] F Δw :(x,S)→Δw

[0020] Among them, x is the point coordinate, S is the human posture, and the input displacement field F Δw , the output Δw is the residual vector of the mixing weights.

[0021] Furthermore, the importance sampling simulates the light emitted by the camera, uniformly samples each light ray, and the number of sampling points is 64; the sampling point coordinates are position-encoded, and the position coding is transmitted through 8 fully connected ReLU layers, each layer has 256 channels, and the volume density value of the corresponding sampling point is output; the corresponding weight value is calculated from the volume density value, and the probability density function is solved; the second layer sampling is performed by the probability density function, and the number of sampling points is 64, so that the sampling points are concentrated in the area where sampling is required, avoiding blank sampling, and better predicting the pixel value of the rendered image;

[0022] Furthermore, the position encoding of the coordinates of the points mapped to the standard space is passed through 8 fully connected ReLU layers, each with 256 channels. The volume density and 256-dimensional feature vector are output; the feature vector is connected to the position encoding of the input viewing direction and processed by an additional fully connected ReLU layer with 128 channels; the RGB value of the current position of the current view is output.

[0023] Furthermore, given the RGB and volume density values, the color of any light that passes through the scene is rendered using a volume rendering method. The color value rendered by the camera ray r(t)=o+td can be expressed as:

[0024]

[0025] where t n and t f represents the near and far points on the camera ray, and the sampling points are selected between these two points; T(t) represents the distance from the ray to t n The cumulative transmittance along the ray from t to t, that is, the ray from t n The probability of not encountering any other particles at t; σ(r(t)) represents the volume density value of a query point on the ray, and c(r(t), d) represents the color of the point at a certain viewing angle.

[0026] In a second aspect, the present invention provides a three-dimensional human body reconstruction system using the neural radiation field considering motion capture enhancement and importance sampling, comprising:

[0027] The data acquisition module is used to collect RGB images of the human body using the Azure Kinect camera and to collect the three-dimensional coordinates of the human skeleton points using the OptiTrack camera;

[0028] Data preprocessing module, used to time-align skeleton data with RGB data and calculate SMPL model parameters;

[0029] The human body deformation module uses a posture-driven deformation field based on a linear hybrid skinning algorithm to achieve accurate mapping from observed posture to standard posture and obtain a standard model of the human body;

[0030] The human body rendering module realizes the importance sampling of volume rendering by estimating the distribution of camera ray transmittance during the volume rendering process. It takes the three-dimensional spatial coordinate parameters and viewing angle parameters of the sampling points as input, performs effective training through a multi-layer perceptron, predicts the volume density and color values ​​of the corresponding sampling points, and uses the volume rendering method to realize the rendering of the human body.

[0031] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0033] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention uses two different modalities of data, RGB images and optical motion capture posture data, to more accurately describe and learn the posture characteristics of a dynamic human body, and performs hierarchical importance sampling to concentrate the sampling points in the area where sampling is required, thereby achieving photo-level high-fidelity rendering of a dynamic three-dimensional human body under a sparse perspective; compared with human body posture prediction based on 2D images, the key point data in three-dimensional space directly collected by the optical motion capture device based on infrared reflected light is more accurate, thereby improving the accuracy of human body posture mapping and the fidelity of rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of the three-dimensional human body reconstruction and rendering algorithm of the present invention.

[0036] Figure 2 Schematic diagram of stratified importance sampling of the present invention. DETAILED DESCRIPTION

[0037] Combination Figure 1 , a neural radiation field 3D human body reconstruction method considering motion capture enhancement and importance sampling, comprising the following steps:

[0038] Azure Kinect cameras are evenly arranged in the space to collect RGB images of the human body, and OptiTrack cameras are evenly arranged in the space to collect the three-dimensional coordinates of the human skeleton points;

[0039] Record the timestamp of each frame of the collected skeleton data and RGB data, perform time alignment, and calculate the SMPL model parameters;

[0040] Taking the relevant parameters of the SMPL model as input, a posture-driven deformation field based on the linear mixed skinning algorithm is proposed to achieve accurate mapping from the observed posture to the standard posture and obtain a standard model of the human body;

[0041] By estimating the distribution of camera ray transmittance during volume rendering, importance sampling of volume rendering is achieved, reducing invalid sampling points. Based on RGB image information and camera parameters, the three-dimensional space coordinate parameters and viewing angle parameters of the sampling points are used as input, and the multi-layer perceptron is effectively trained to predict the volume density and color values ​​of the corresponding sampling points. The volume rendering method is used to achieve high-fidelity rendering of the human body;

[0042] Furthermore, RGB images and skeleton data were collected by Azure Kinect camera and OptiTrack optical motion capture camera respectively. The height of RGB images was 1080 and the width was 1920 pixels. The original key point data was output through Motive software as human skeleton information of each frame in the form of bvh format, and the SMPL posture vector was calculated based on the skeleton information with a dimension of 72 (24 joints, each with three degrees of freedom). The two types of devices were connected to the same local computer with an acquisition frequency of 30fps / s, and the start acquisition time and the timestamp of each frame were recorded in a distributed manner. Based on the time information of the RGB frame, the timestamp information of the skeleton data was used to align with the RGB frame to achieve time synchronization;

[0043] Furthermore, the human body deformation module is a proposed posture-driven deformation field. Taking human body posture information as a priori, the deformation field is defined as a combination of rigid deformation and non-rigid deformation. Rigid deformation is implemented based on a linear blended skinning algorithm, which associates the vertices of the 3D model with the bone structure and realizes the deformation of the model by weighting the position of the vertex to one or more bones. Based on this method, the human body posture in the observation space is mapped to the standard space (the posture is T-shaped):

[0044]

[0045] Among them, ω o (x) is the mixing weight function defined in the observation space, G k It is the K transformation matrices generated by the K parts defined by the human skeleton. Each bone has a transformation matrix that represents its position, rotation, and scale in world space, which is composed of the local transformation of the bone and the transformation of the parent bone.

[0046] Non-rigid deformation learns personalized human details by introducing a displacement field related to the pose, making up for the problem that the linear mixed skin transformation cannot learn details such as clothing wrinkles. The displacement field is specifically expressed as:

[0047] F Δw :(x,S)→Δw

[0048] Among them, x is the input point coordinate, S is the observed posture of the human body in the current frame, and the input displacement field F Δw In the above equation, the output Δw is the residual vector of the mixing weights.

[0049] Combining rigid deformation with non-rigid deformation, we get x'+Δw, which is the mapping of the sampling points of the person in the observed posture to the standard space, to achieve accurate deformation of the human body in motion;

[0050] Furthermore, a hierarchical importance sampling strategy is used in the human body rendering module. The light emitted by the camera is simulated, and each ray is sampled uniformly, with a number of 64 sampling points. The sampling point coordinates are position-encoded, and the position coding is passed through 8 fully connected ReLU layers, each with 256 channels, and the volume density value of the corresponding sampling point is output. The corresponding weight value is calculated by the volume density value, and the probability density function is solved. The second layer of sampling is performed by the probability density function, with a number of 64 sampling points, so that the sampling points are concentrated in the area where sampling is required, avoiding blank sampling, and better predicting the pixel value of the rendered image. In the volume density prediction network, in addition to outputting the volume density value corresponding to the sampling point, a 256-dimensional feature vector is also output. The feature vector is connected to the position coding of the input viewing direction and processed by an additional fully connected ReLU layer with 128 channels. The RGB value of the current position of the current view is output. Given the RGB and volume density values, the color of any light passing through the scene is rendered using the volume rendering method. The color value rendered by the camera ray r(t)=o+td can be expressed as:

[0051]

[0052] The function T(t) represents the distance along the ray from t. n The cumulative transmittance from t to t, that is, the ray from t n The probability of hitting any other particle by t.

[0053] Based on the same inventive concept, the present invention also provides a neural radiation field 3D human body reconstruction system considering motion capture enhancement and importance sampling, which is used to implement the above method. The system includes:

[0054] The data acquisition module is used to collect RGB images of the human body using the Azure Kinect camera and to collect the three-dimensional coordinates of the human skeleton points using the OptiTrack camera;

[0055] Data preprocessing module, used to time-align skeleton data with RGB data and calculate SMPL model parameters;

[0056] The human body deformation module uses a posture-driven deformation field based on a linear hybrid skinning algorithm to achieve accurate mapping from observed posture to standard posture and obtain a standard model of the human body;

[0057] The human body rendering module realizes the importance sampling of volume rendering by estimating the distribution of camera ray transmittance during the volume rendering process. It takes the three-dimensional spatial coordinate parameters and viewing angle parameters of the sampling points as input, performs effective training through a multi-layer perceptron, predicts the volume density and color values ​​of the corresponding sampling points, and uses the volume rendering method to realize the rendering of the human body.

[0058] The specific implementation method of the above modules is the same as the aforementioned method for reconstructing a three-dimensional human body using a neural radiation field, and will not be described in detail in the present invention.

[0059] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0060] Example

[0061] In this embodiment, the neural radiation field three-dimensional human body reconstruction method considering motion capture enhancement and importance sampling, the system is divided into a data acquisition and processing module, a human body deformation module and a human body rendering module. The data acquisition and processing module uses 3 Azure Kinect cameras to obtain multi-view RGB images and 6 OptiTrack cameras to collect the three-dimensional coordinates of key points of the human body, and calculates the SMPL model parameters. The human body reconstruction module uses a posture-driven deformation field to realize the mapping from the observation space to the standard space, and obtains a standard human body model; based on RGB data and posture data, by estimating the camera ray transmittance distribution in the volume rendering process, the importance sampling of volume rendering is realized, which reduces invalid sampling points and improves the quality of human body reconstruction; finally, based on the neural radiation field method, the volume density and color value of any point in any direction in the space are predicted, and the human body rendering is realized using the volume rendering method. The flow chart of the neural radiation field three-dimensional human body reconstruction method considering motion capture enhancement and importance sampling is as described Figure 1 As shown, the method comprises the following steps:

[0062] Step 1: Multiple computers are connected to Azure Kinect cameras and OptiTrack optical motion capture cameras to obtain RGB images and human skeleton motion information;

[0063] Step 2: The main control computer performs time alignment between frames of the RGB image and the human skeleton motion information;

[0064] Step 3: Fit the human skeleton motion information to the SMPL model and calculate the SMPL parameters;

[0065] Step 4: Mapping the observed posture to the standard posture based on SMPL parameters;

[0066] Step 5: Based on the neural radiance field method, hierarchical importance sampling is used to achieve more realistic human body rendering;

[0067] In one of the embodiments, in the third step, the human skeleton motion information is fitted into the SMPL model, and the posture information recognized by the Mediapipe method is combined with the motion capture information to calculate the SMPL parameters.

[0068] In one embodiment, the human body deformation and mapping module is specifically as follows: the posture-driven deformation field is divided into rigid deformation and non-rigid deformation. The rigid deformation is implemented based on the linear hybrid skinning algorithm, and the points under the observed posture are mapped to the standard space:

[0069]

[0070] Among them, ω o (x) is the mixing weight function defined in the observation space, G k are the K transformation matrices generated by the K parts defined by the human skeleton.

[0071] Non-rigid learning learns personalized human details by introducing a displacement field related to the pose, which is expressed as:

[0072] F Δw :(x,S)→Δw

[0073] Among them, x is the point coordinate, S is the human posture, input F Δw , the output Δw is the residual vector of the mixing weights.

[0074] In one embodiment, the high-fidelity human body rendering method uses a neural radiation field method, which is specifically implemented as follows:

[0075] Step 1: Pass the position encoding of the coordinates of the point mapped to the standard space through 8 fully connected ReLU layers, each with 256 channels. Output volume density and 256-dimensional feature vector. This feature vector is concatenated with the position encoding of the input viewing direction and processed by an additional fully connected ReLU layer with 128 channels. Output the RGB value of the current position of the current view.

[0076] Step 2: Given the RGB and volume density values, use volume rendering to render the color of any light that passes through the scene. The color value rendered by the camera ray r(t)=o+td can be expressed as:

[0077]

[0078] The function T(t) represents the distance along the ray from t. n The cumulative transmittance from t to t, that is, the ray from t n The probability of hitting any other particle by t.

[0079] Step 3: Implement stratified sampling strategy. Figure 2 As shown. First, a set of sampling points are uniformly sampled to obtain the output of the neural radiation field network. The weight of this sampling is obtained based on this set of outputs, and the probability density function (PDF) is solved. Importance sampling is implemented based on PDF, that is, the sampling points are concentrated in the area where sampling is required.

[0080] The above embodiments are descriptions of specific implementation methods of the present invention rather than limitations of the present invention. Technical personnel in the relevant technical field can make various changes and modifications to obtain corresponding equivalent technical solutions without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions should be included in the patent protection scope of the present invention.

Claims

1. A method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling, characterized in that: The following steps are involved: Collect human RGB images and skeleton data; Image preprocessing: synchronize multimodal data and achieve time alignment for each frame. Map the skeleton data to the SMPL human body model, calculate the SMPL posture parameters, use the human body posture parameters and RGB images as the input of the human body deformation network, divide the human body movement into rigid deformation and non-rigid deformation, and realize the modeling of the standard human body model; In the sampling process, stratified importance sampling is used, and uniform sampling is combined with importance sampling to select more effective sampling points; Given a standard human body model, the three-dimensional coordinates of the position-encoded sampling points are input into the body density prediction network, and the body density and feature vector are output; The feature vector and the position-encoded viewing angle information are input into the color prediction network, and the color value of the corresponding sampling point is output; Using volume rendering methods, the pixel values ​​of images of the human body at any viewing angle are predicted to achieve photorealistic rendering.

2. The method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling according to claim 1, characterized in that: The multi-view RGB images are evenly collected by multiple Azure Kinect cameras.

3. The method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling according to claim 1, characterized in that: The human body posture information is collected through the OptiTrack optical motion capture camera through the reflection of infrared light to collect spatial coordinates.

4. The method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling according to claim 1, characterized in that: The RGB image has a height of 1080 pixels and a width of 1920 pixels.

5. The method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling according to claim 1, characterized in that: The rigid deformation is implemented based on the linear hybrid skinning algorithm, which maps the points in the observed posture into the canonical space: Among them, x represents the three-dimensional coordinates of the query point before transformation, ω o (x) is the mixing weight function defined in the observation space, G k is the transformation matrix generated by the kth part of the human skeleton definition, K = 24; after the transformation, the query point x' in the standard space is obtained; The non-rigid deformation network learns personalized human details by introducing a displacement field related to the pose, which is expressed as: F Δw :(x,S)→Δw Among them, x is the point coordinate, S is the human posture, and the input displacement field F Δw , the output Δw is the residual vector of the mixing weight; By combining the rigid deformation with the non-rigid deformation, we obtain x'+Δw, which is the mapping of the sampling points of the person in the observed posture to the standard space, and realize the precise deformation of the human body in motion.

6. The method for 3D human body reconstruction based on neural radiation field considering motion capture enhancement and importance sampling according to claim 1, characterized in that: Importance sampling simulates the light emitted by the camera, uniformly samples each ray, and the number of sampling points is 64; the sampling point coordinates are position-encoded, and the position coding is passed through 8 fully connected ReLU layers, each layer has 256 channels, and the volume density value of the corresponding sampling point is output; the corresponding weight value is calculated from the volume density value, and the probability density function is solved; the second layer sampling is performed based on the probability density function, and the number of sampling points is 64; The position encoding of the coordinates of the point mapped to the standard space is passed through 8 fully connected ReLU layers, each with 256 channels; output volume density and a 256-dimensional feature vector; this feature vector is concatenated with the position encoding of the input viewing direction and processed by an additional fully connected ReLU layer with 128 channels; output the RGB value of the current position of the current view; Given the RGB value and volume density value, use the volume rendering method to render the color of any light passing through the scene; define the starting point o of the camera ray, the direction in which the ray is emitted as the emission vector d of the ray, and the parameter t controls the position of each point on the ray. The color value rendered by the camera ray r(t) = o + td is expressed as: where t n and t f represents the near and far points on the camera ray, and the sampling points are selected between these two points; T(t) represents the distance from the ray to t n The cumulative transmittance along the ray from t to t, that is, the ray from t n The probability of not encountering any other particles at t; σ(r(t)) represents the volume density value of a query point on the ray, and c(r(t), d) represents the color of the point at a certain viewing angle.

7. A neural radiation field 3D human body reconstruction system considering motion capture enhancement and importance sampling, characterized in that: For implementing the method described in any one of claims 1 to 6, the system comprises: The data acquisition module is used to collect RGB images of the human body using the Azure Kinect camera and to collect the three-dimensional coordinates of the human skeleton points using the OptiTrack camera; Data preprocessing module, used to time-align skeleton data with RGB data and calculate SMPL model parameters; The human body deformation module uses a posture-driven deformation field based on a linear hybrid skinning algorithm to achieve accurate mapping from observed posture to standard posture and obtain a standard model of the human body; The human body rendering module realizes the importance sampling of volume rendering by estimating the distribution of camera ray transmittance during the volume rendering process. It takes the three-dimensional spatial coordinate parameters and viewing angle parameters of the sampling points as input, performs effective training through a multi-layer perceptron, predicts the volume density and color values ​​of the corresponding sampling points, and uses the volume rendering method to realize the rendering of the human body.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images