A digital human body reconstruction method and device based on neural radiation fields
By employing the neural radiation field method, utilizing inverse pose transformation and decoupling of human representation, and multi-layer neural networks, combined with non-local attention and channel spatial attention modules, the accuracy and rendering complexity issues of digital human reconstruction in existing technologies are solved, achieving efficient digital human reconstruction and rendering from any angle.
Patent Information
- Application Number
- CN202511008352.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-22
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-07-22
AI Technical Summary
Existing technologies rely on manual operation and complex algorithms when reconstructing digital human bodies, which limits their widespread application, especially the accuracy of digital human body reconstruction under monocular video and the effect of rendering reconstructed images from free point perspectives in arbitrary frames.
The neural radiation field method is adopted, which decouples the inverse pose transformation from human representation. It utilizes skeleton-driven rigid deformation and multi-layer neural network to learn non-rigid deformation detail offset. Combined with non-local attention module, residual learning module, channel and spatial attention module, and trained with multiple loss functions, it realizes the mapping and rendering of target object video.
It improves the accuracy and reliability of digital human body reconstruction, and can generate reconstruction effect drawings from any angle for each frame, thus enhancing the accuracy and reliability of rendering.
Smart Images

Figure CN120782970B_ABST
Abstract
Description
Technical Field
[0001] This article belongs to the technical fields of computer graphics and deep learning, and in particular relates to a method and device for digital human body reconstruction based on neural radiation fields. Background Technology
[0002] With the rapid development of computer vision and graphics, 3D reconstruction of the digital human body has become possible, and this emerging technology has enormous application prospects in our lives. Digital humans have demonstrated rich development potential in fields such as medicine, education, film and entertainment, and business broadcasting. In the medical field, digital humans assist doctors in medical diagnosis, improving the accuracy of diagnoses; in education, digital human teachers, with their flexible and realistic appearance, enhance students' interest and motivation in online learning; in film and entertainment, digital humans digitize real people, showcasing more possibilities for artistic creation; and in business broadcasting, live streaming platforms use digital humans for broadcasting, significantly reducing broadcasting costs.
[0003] In recent years, many solutions for reconstructing digital human bodies have been developed, from structured mesh-based methods to subsequent geometric representation methods based on point clouds, voxels, cubes, etc. Although significant progress has been made, these methods generally rely on manual operation, complex algorithms, and specific equipment to generate relevant data, which limits their widespread application. Summary of the Invention
[0004] To address the aforementioned problems in existing technologies, this paper aims to provide a method and apparatus for digital human body reconstruction based on neural radiation fields, which can improve the accuracy of digital human body reconstruction under monocular video and render reconstructed images from free point perspectives in any frame.
[0005] To solve the above-mentioned technical problems, the specific technical solution presented in this paper is as follows: On the one hand, this paper provides a digital human body reconstruction method based on neural radiation fields, the method comprising: Based on the decoupling of inverse pose transformation and human representation, coarse deformation is obtained through rigid deformation driven by the skeleton, and non-rigid deformation detail offset is learned by using a multi-layer neural network to map the target object video from the observation space to the canonical space. The object parameters of the target object video are extracted using a nonlocal attention module in the neural radiation field, and the multi-scale features are fused using a residual learning module to obtain a fused feature vector, wherein the object parameters are human pose parameters. Accurate perception modeling of non-rigid deformation is achieved by utilizing the channel attention module and spatial attention module in a neural network. The neural radiation field is trained by using joint training with multiple loss functions to obtain the trained neural radiation field. The coordinate transformation matrix between the observation space and the normal space, along with the target object video, are input into the trained neural radiation field to render the reconstructed target object video.
[0006] Furthermore, based on the decoupling of inverse pose transformation and human representation, detail offsets in the target object video are determined using non-rigid deformation and skeleton-driven rigid deformation to map the target object video from the observation space to the canonical space, including: Define the motion field of the target object and map the target object video from the observation space to the normal space; Based on the inverse pose transformation and human characterization, the motion field is decoupled to determine the rigid deformation driven by the skeleton, as follows: Where x is the spatial position of the joint of the target object, and d is the skeletal pose input of the target object; By using a multi-layer neural network to perform offset processing on rigid deformations driven by bones, we obtain the detail offset of non-rigid deformations, represented as: .
[0007] Furthermore, the rigid deformation based on the skeleton is transformed into a coarse deformation driven by the standard skin.
[0008] Furthermore, a nonlocal attention module in the neural radiation field is used to extract multi-scale features from the object parameters of the target video, and a residual learning module is used to fuse the multi-scale features to obtain a fused feature vector, including: The object parameters are subjected to feature projection processing using the pose encoding feature matrix to obtain the projected embedding layer vector, which is represented as follows: Where x is the object parameter, This is the pose encoding feature matrix corresponding to the key value. This is the pose encoding feature matrix corresponding to the query value. This is the pose encoding feature matrix corresponding to the true value; The projected embedding layer vector is flattened to obtain the flattened vector, denoted as: ; ; ; The projected embedding layer vectors are concatenated, activated, and normalized to obtain the attention weights, which are expressed as follows: Where A is the attention weight, N is the normalization factor, N is the total number of pixels in the virtual space, and Conv is a 1×1 convolution operation; Based on the attention weights and the flattened vector, feature fusion processing is performed using the convolution kernel weight matrix to obtain the fused feature vector, expressed as: ,in This is a 1×1 convolution kernel weight matrix used to map the aggregated features back to the original number of channels.
[0009] Furthermore, after obtaining the fused feature vector, the process also includes: The fused feature vector is input into a multilayer perceptron neural network to obtain the predicted pose of the target object.
[0010] Furthermore, the channel attention module and spatial attention module in the neural radiation field are used to optimize the detail offset in non-rigid deformation, resulting in optimized modeled non-rigid deformation, including: The channel attention weights are obtained by shifting the channel dimension based on the fused feature vectors, thus generating a channel attention map with values between [0,1], represented as follows: ,in , It is a tensor with the channel dimension shifted backward. It is a dimension-reducing fully connected layer. It is a fully connected layer of higher dimension. Channel attention weights; The fused feature vector is enhanced based on the channel attention weights to obtain the enhanced features, as follows: , For enhanced features; The enhanced features are subjected to multi-layer convolution processing to obtain spatial attention weights, which are expressed as follows: ,in, These are sigmoid-normalized spatial attention weights. It's a 7x7 convolution dimensionality reduction. It is the number of channels recovered by 7x7 convolution dimensionality reduction; Based on the spatial attention weights, the enhanced features are jointly enhanced to obtain spatial channel jointly enhanced features, which are represented as follows: ; The joint enhancement features of the spatial channels are subjected to dimensional flattening and mapping to obtain the non-rigid deformation detail offset, which is expressed as: ,in, It is the last layer of weight matrix. It is the bias vector of the last layer.
[0011] Furthermore, the multiple loss function is expressed as: , ; in, Let the mean squared error loss function be . For the perceptual loss function, This is the weighting balance coefficient. Representing camera rays, it is a vector used to describe information such as the direction and position of the rays. It is the set of all camera rays, and represents an element in that set. This represents the rendered color of the camera's light rays, specifically the color value corresponding to that light ray obtained through network rendering. It represents the true color of the light from the camera.
[0012] Furthermore, the coordinate transformation matrix from the observation space to the normal space and the target object video are input into the trained neural radiation field to render the reconstructed target object video, including: A continuous motion field in a normal space is constructed as follows: ,in γ(d) represents the position code of the rotation angle, and d represents the five-dimensional data input of the function's 3D coordinates and 2D coordinates; For each ray of light in the target object video, the rendered color is represented as follows: ,in , yes , It is the first and The spacing between sampling points. It was before The cumulative probability that a sample is occluded is the transmittance. . yes Transmittance weight, yes The color of the area.
[0013] Furthermore, when the probability of the i-th sample belonging to the foreground is low, Approaching zero.
[0014] On the other hand, this paper also provides a digital human body reconstruction device for neural radiation fields, the device comprising: The decoupling module is used to decouple the inverse pose transformation from the human representation. It obtains a coarse deformation through rigid deformation driven by the skeleton and uses a multi-layer neural network to learn the non-rigid deformation detail offset in order to map the target object video from the observation space to the normal space. The fusion feature vector acquisition module is used to extract multi-scale features of the object parameters of the target object video using the non-local attention module in the neural radiation field, and to fuse the multi-scale features using the residual learning module to obtain a fusion feature vector, wherein the object parameters are human pose parameters. The channel-space joint attention optimization module is used to accurately perceive and model non-rigid deformations by utilizing the channel attention module and the spatial attention module in the neural network. The training module is used to train the neural radiation field using joint training with multiple loss functions to obtain the trained neural radiation field. The rendering module is used to input the coordinate transformation matrix between the observation space and the normal space, as well as the target object video, into the trained neural radiation field to render the reconstructed target object video.
[0015] Using the above technical solution, the digital human body reconstruction method and device based on neural radiation field described in this paper decouples inverse pose transformation from human body representation. It obtains coarse deformation through rigid deformation driven by the skeleton, and utilizes a multi-layer neural network to learn non-rigid deformation detail offsets to map the target object video from the observation space to the normal space. A non-local attention module in the neural radiation field is used to extract multi-scale features from the object parameters of the target object video, and a residual learning module is used to fuse these multi-scale features to obtain a fused feature vector, where the object parameters are human pose parameters. Channel attention and spatial attention modules in the neural network are used to optimize the detail offsets in the non-rigid deformation, resulting in optimized modeled non-rigid deformation. Multiple loss functions are used for joint training to train the neural radiation field, resulting in a trained neural radiation field. The coordinate transformation matrix between the observation space and the normal space, along with the target object video, are input into the trained neural radiation field to render the reconstructed target object video. This allows for the generation of reconstruction effect images of the human body at any angle for each frame, improving the accuracy and reliability of the rendering.
[0016] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 The illustration shows a schematic diagram of the steps of a digital human body reconstruction method based on a neural radiation field provided in the embodiments of this article.
[0019] Figure 2 A schematic diagram of the structure of a digital human body reconstruction device for neural radiation fields provided in the embodiments of this article is shown.
[0020] Figure 3 A schematic diagram of the structure of the computer device provided in the embodiments of this article is shown. Detailed Implementation
[0021] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.
[0022] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0023] Regarding solutions for reconstructing digital human bodies, significant progress has been made from methods based on structured meshes to subsequent methods based on geometric representations such as point clouds, voxels, and cubes. However, these methods generally rely on manual operation, complex algorithms, and specific equipment to generate relevant data, which limits their widespread application.
[0024] To address the aforementioned issues, this paper provides a digital human body reconstruction method based on neural radiation fields, which can improve the accuracy and reliability of digital human body reconstruction. Figure 1 This is a schematic diagram illustrating the steps of a digital human body reconstruction method for neural radiation fields provided in the embodiments of this document. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel. Specifically, as shown in the embodiments or accompanying drawings... Figure 1 As shown, the method may include: S101: Based on the decoupling of inverse pose transformation and human representation, coarse deformation is obtained through rigid deformation driven by the skeleton, and non-rigid deformation detail offset is learned by using a multi-layer neural network to map the target object video from the observation space to the canonical space. S102: The nonlocal attention module in the neural radiation field is used to extract multi-scale features of the object parameters of the target object video, and the residual learning module is used to fuse the multi-scale features to obtain a fused feature vector, wherein the object parameters are human pose parameters. S103: Accurate perception modeling of non-rigid deformation using the channel attention module and spatial attention module in a neural network; S104: The neural radiation field is trained using joint training with multiple loss functions to obtain the trained neural radiation field. S105: Input the coordinate transformation matrix between the observation space and the normal space and the target object video into the trained neural radiation field to render the reconstructed target object video.
[0025] This paper can be understood as follows: It decouples inverse pose transformation from human representation, uses a multi-layer neural network to learn the detailed offsets after non-rigid deformation skeletal motion, then optimizes multi-scale feature extraction through pose optimization based on non-local attention, fuses shallow local details with deep global features through residual connections, optimizes non-rigid deformation modeling using a channel-joint spatial dual-dimensional attention module, employs joint training with multiple loss functions, and balances pixel-level accuracy and semantic-level consistency through a multi-scale supervision mechanism. The loss function selection also uses the mean squared error loss function to match pixel-level appearance. Free-point rendering of the digitally reconstructed human body is achieved through neural radiation fields, generating reconstructed images of the human body at any angle for each frame. Balancing accuracy and aesthetics, it has significant application value.
[0026] In the embodiments of this specification, based on the decoupling of inverse pose transformation and human representation, the detail offset in the target object video is determined using non-rigid deformation and skeleton-driven rigid deformation, so as to map the target object video from the observation space to the canonical space, including: Define the motion field of the target object and map the target object video from the observation space to the normal space; Based on the inverse pose transformation and human characterization, the motion field is decoupled to determine the rigid deformation driven by the skeleton, as follows: Where x is the spatial position of the joint of the target object, and d is the skeletal pose of the target object; By using a multi-layer neural network to perform offset processing on rigid deformations driven by bones, we obtain the detail offset of non-rigid deformations, represented as: .
[0027] In this context, the human body representation is torn and deformed to the observed pose through the volumetric appearance Gc, which is used to generate the output appearance volume Go that can represent a person in a state of movement. That is, the representation of the position in the observation space mapped to the normal space is as follows: , In Gc, x(a,σ) maps the position x in the normal space to the color a and density σ. T(x,d) defines a motion field that maps points in the space from the observation space back to the normal space, guided by the observation posture d. d can be represented as the posture of the human body at each joint in space, such as using axis-angle vectors to represent the rotation of local joints. Through such representation, the final drawing of the rendering step in the normal space is completed.
[0028] To handle complex human motion, the motion field represented by T is decoupled into a skeleton-driven deformation T. skel and the offset T of the position NR , represented as: T skel This represents bone-driven deformation, which is essentially an inverse volumetric blend of linear skinning, T NR It starts with a bone-driven deformation and generates a corresponding offset. T skel What can be provided is coarse deformation driven by standard skin, while T NR It can provide more non-rigid effects, which can be processed through multi-layer neural networks, such as deformations in clothing and deformations caused by muscle changes.
[0029] In the embodiments of this specification, a nonlocal attention module in a neural radiation field is used to extract multi-scale features of object parameters from a target video, and a residual learning module is used to fuse the multi-scale features to obtain a fused feature vector, including: The object parameters are subjected to feature projection processing using the pose encoding feature matrix to obtain the projected embedding layer vector, which is represented as follows: Where x is the object parameter, This is the pose encoding feature matrix corresponding to the key value. This is the pose encoding feature matrix corresponding to the query value. This is the pose encoding feature matrix corresponding to the true value; The projected embedding layer vector is flattened to obtain the flattened vector, denoted as: ; ; ; The projected embedding layer vectors are concatenated, activated, and normalized to obtain the attention weights, which are expressed as follows: Where A is the attention weight, N is the normalization factor, N is the total number of pixels in the virtual space, and Conv is a 1×1 convolution operation; Based on the attention weights and the flattened vector, feature fusion processing is performed using the convolution kernel weight matrix to obtain the fused feature vector, expressed as: ,in This is a 1×1 convolution kernel weight matrix used to map the aggregated features back to the original number of channels.
[0030] It should be noted that the core idea of non-local attention is that the response at a certain pixel is the sum of the feature weights at all other pixels, with the general formula as follows: , Where x is the input signal, which in this invention corresponds to the joint rotation angle as a human posture parameter; i represents the output position, which is the index in the human space; and the response of i is obtained by enumerating j and then calculating it. This represents the response at the i-th pixel. The function represents the correlation between two pixels. Indicates a point linear mapping of embedding, This represents a normalization operation, where the final y is obtained through the response factor. This is obtained after standardization.
[0031] By summing the feature weights of each pixel in each target object video as described above, the human posture in the target object video can be reshaped, thereby updating the human posture parameters, i.e., the joint rotation angle.
[0032] Furthermore, the input is first a batch of pose encoding feature matrices, with dimensions where the feature dimension of each sample is 69. The corresponding human pose parameters are the joint rotation angles, i.e., the updated joint rotation angles mentioned above. The embedding layer generates Query, Key, and Value for feature projection, represented as follows: To increase the flexibility of extending to real-world spatial dimensions, feature dimensions are adjusted: , , , in Flatten the feature map into (H) A vector of length W) Flattened Used for subsequent weighted summation. Flatten And replicate W times in the last dimension, Flatten Furthermore, it replicates H times in the third dimension, encoding spatial locations as vectors, and expands the dimensions to align interactions between all locations. Unlike traditional nonlocal attention, which uses dot product or cosine similarity to calculate feature correlations and explicitly models global dependencies through high-dimensional matrix multiplication, this approach generates attention weights through concatenation and convolution. The steps are feature concatenation, convolution, activation, and normalization. , in, Input features, the shape of which is a pose encoding generated by 1×1 convolution after 2D expansion of the key projection features. Query projection features are generated through 1×1 convolution. , The [] symbol represents the channel-level concatenation operation, merging query and key features. `Conv` is a 1×1 convolution operation, and `ReLU` is a non-linear activation function to enhance the expressive power of the attention weights. `N` is a normalization factor, here representing the total number of pixels in the virtual space. A more efficient method is achieved through implicit convolutional interactions, `ReLU` activation and release, and a lightweight computational architecture. Feature aggregation is achieved through matrix multiplication, followed by explicit residual linking to realize feature fusion of the non-local attention module. , in The weight matrix is a 1×1 convolutional kernel used to map the aggregated features back to the original number of channels. The enhanced features are added to the original input to retain basic features and avoid information loss. In the embodiments of this specification, the four-dimensional pose vector after feature fusion is restored to the original two-dimensional state and then fed into the MLP for pose refinement and prediction. ,in It is the Rodrigues transformation, which converts a 3D rotation vector into an orthogonal rotation matrix, and its expansion formula, given a rotation vector... in When the rotation matrix is: , It is an antisymmetric matrix. An antisymmetric matrix is the transpose of a matrix, which is itself. The specific operation to convert a vector into an antisymmetric matrix is to normalize the rotation vector of the skeleton, and then convert the normalized vector into an antisymmetric matrix. It is the matrix obtained by multiplying an antisymmetric matrix by itself.
[0033] In the embodiments of this specification, the channel attention module and spatial attention module in the neural radiation field are further used to optimize the detail offset in non-rigid deformation, resulting in optimized modeled non-rigid deformation, including: The channel attention weights are obtained by shifting the channel dimension based on the fused feature vectors, thus generating a channel attention map with values between [0,1], represented as follows: ,in , It is a tensor with the channel dimension shifted backward. It is a dimension-reducing fully connected layer. It is a fully connected layer of higher dimension. Channel attention weights; The fused feature vector is enhanced based on the channel attention weights to obtain the enhanced features, as follows: , For enhanced features; The enhanced features are subjected to multi-layer convolution processing to obtain spatial attention weights, which are expressed as follows: ,in, These are sigmoid-normalized spatial attention weights. It's a 7x7 convolution dimensionality reduction. It is the number of channels recovered by 7x7 convolution dimensionality reduction; Based on the spatial attention weights, the enhanced features are jointly enhanced to obtain spatial channel jointly enhanced features, which are represented as follows: ; The joint enhancement features of the spatial channels are subjected to dimensional flattening and mapping to obtain the non-rigid deformation detail offset, which is expressed as: ,in, It is the last layer of weight matrix. It is the bias vector of the last layer.
[0034] This can be understood as a channel space combined with an attention module, which is an attention mechanism that can amplify global-dimensional interactive features while reducing information diffusion. Given the input feature map, intermediate states, and output states, it can be represented as: ,in and These are channel attention maps and spatial attention maps, respectively. This indicates element-wise multiplication. The channel attention submodule uses a three-dimensional arrangement to preserve information in three dimensions. It then amplifies cross-dimensional channel spatial dependencies using a two-layer multilayer perceptron. In the spatial attention submodule, two convolutional layers are used to fuse spatial information to focus on spatial information. The same reduction ratio r as BAM is also used from the channel attention submodule. Meanwhile, max pooling negatively impacts information utilization. Pooling operations are removed to further preserve feature mappings. In channel-based spatial attention methods, channel attention computation learns feature channels for non-rigid deformation prediction. More importantly, dimensional permutation is performed first, moving the permuted channel dimensions to the end so that the fully connected layers process according to the channel dimensions. Specifically, , ,in, It is a tensor with the channel dimension shifted backward. L1 is a dimension-reducing fully connected layer, and L2 is a dimension-increasing fully connected layer. Channel attention weights. The number of channels is compressed from d to d / 4 using a fully connected L1 layer, reducing computational cost. The sigmoid function is applied to generate a channel attention map with values between [0,1]. The channel attention map is then multiplied element-wise with the original input to obtain the channel-enhanced features. : ,in This is element-wise multiplication, an operation that multiplies corresponding elements in two matrices, restoring the dimension of the channel attention weights. Spatial attention calculation uses convolutional kernels to simulate spatial context; the second convolution increases the number of channels to 'd', generating the spatial attention weights. ,in It's a 7x7 convolution dimensionality reduction. It is also a 7x7 convolution dimensionality reduction to restore the number of channels. The 7x7 convolution preserves compatibility with spatial context. These are sigmoid-normalized spatial attention weights. (The last sentence appears to be incomplete and possibly refers to a different topic.) Features enhanced by channel Element-wise multiplication yields the final joint enhancement feature of the spatial channels. Global scaling is applied to each channel based on channel weighting: , Combining channel space with attention output 4D features Flattened into a 1D vector, the displacement is predicted after restoring the MLP-processable format, and the displacement value of non-rigid deformation is output to adjust the position of the input point: ,in, It is the last layer of weight matrix. This is the final layer bias vector, which, after calculation, maps the channel space combined with attention-enhanced features into a 3D displacement vector. This displacement is used to adjust the position of the input point, modeling non-rigid deformation. In the process of digital human body reconstruction, by modeling both rigid and non-rigid deformations, human body deformation can be accurately represented. This paper proposes a solution to the problem of modeling non-rigid deformations such as wrinkles in human body reconstruction.
[0035] In the embodiments described in this specification, the multiple loss functions are expressed as follows: , ; in, Let the mean squared error loss function be . For the perceptual loss function, is the weight balancing coefficient, used to adjust the importance of the perceptual loss function in the final loss function; r represents the camera ray, a vector used to describe the direction, position, and other information of the ray; R is the set of all camera rays, and represents an element in the set. This represents the rendered color of the camera's light rays, specifically the color value corresponding to that light ray obtained through network rendering. It represents the true color of the light from the camera.
[0036] In the embodiments of this specification, the coordinate transformation matrix between the observation space and the canonical space, along with the target object video, are input into the trained neural radiation field to render the reconstructed target object video, including: A continuous motion field in a normal space is constructed as follows: ,in γ(d) represents the position code of the rotation angle; For each ray of light in the target object video, the rendered color is represented as follows: ,in , yes , It is the first and The spacing between sampling points. It was before The cumulative probability that a sample is occluded is the transmittance. . yes Transmittance weight, yes The color of the area.
[0037] Furthermore, when the approximate foreground probability f(x) is low, the definition is further expanded as follows: , in yes The foreground probability, predicted by a neural network, ranges from 0 to 1, representing the probability that the position belongs to the foreground. At lower levels, Approaching zero, the background's contribution to the color integral can be reduced, enabling the separation of foreground and background in dynamic human rendering and improving rendering quality.
[0038] For example, in the experiment, data was selected from the ZJU-MoCap dataset. Five human figures with different movements (377, 386, 387, 392, and 393) were ultimately chosen as experimental data for model training. During model training, because the training involved multiple movements, data from camera 1 was used as input experimental data, while the remaining camera data was used as evaluation data. The experiment applied the camera matrix parameters, body poses, and segmentation data from the dataset.
[0039] This dataset features humans performing complex movements, including spinning, Tai Chi, arm swings, warm-ups, boxing, and kicking. The ZJU-MoCap dataset has been widely used in research fields such as pose estimation, action recognition, motion retargeting, and virtual human animation synthesis, demonstrating significant value, especially in handling challenging scenarios involving occlusion, rapid movement, and non-rigid deformation. In addition to videos of each performer, the ZJU-MoCap dataset provides masks for different performers to achieve foreground and background segmentation, helping researchers using the dataset to more quickly obtain the complete image of the person.
[0040] The specific parameter settings for the experimental training of this method are as follows: using the Adam optimizer, with hyperparameters β1=0.9 and β2=0.99, for optimization. Specifically, the learning rate parameter for the standard MLP is 5×10⁻⁶. -4 All other MLP parameters, including the learning rate, are set to a uniform learning rate of 5 × 10⁻⁶. -5 The batch size was 8, with 128 samples per ray. During the experiment, it was observed that no significant improvement in reconstruction performance occurred after more than 400k iterations; therefore, the overall number of iterations was set to 400k. The entire experiment required 400k iterations. For the ZJU-MoCap dataset, a delay optimization strategy was adopted: the non-rigid deformation sensing module was activated only after 50k iterations. To ensure the scientific rigor of the comparative experiments, all experiments used the same number of iterations and the same hardware and software equipment to maintain a consistent experimental environment.
[0041] Compared to the classic neural rendering method for reconstructing the digital human body, HumanNeRF, this invention demonstrates significant improvements in capturing multi-scale features and accurately perceiving non-rigid deformations, from a subjective perspective. This invention can reconstruct multi-detail features and achieve more accurate detail capture and extraction, capturing both large-scale features and small-scale features related to subtle movements of local joints.
[0042] Table 1 shows the experimental data results. From a quantitative analysis perspective, the experimental data evaluation shows that the present invention outperforms the HumanNeRF and Neural Body methods in three evaluation metrics: PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and LPIPS (Perceptual Image Loss). This demonstrates that the present invention is superior to the HumanNeRF and Neural Body methods in terms of visual quality presentation. A high PSNR may indicate that the model successfully reconstructs the basic geometry, a high SSIM indicates that details such as clothing folds are more natural, and a low LPIPS score further verifies the continuity of movement and visual realism.
[0043] Table 1 shows the quantitative results of the experimental data, comparing the PSNR, SSIM, and LPIPS evaluation indicators of different methods for reconstructing digital human bodies.
[0044] Table 1 lpips = lpips ×
[0045]
[0046] This invention verifies the effectiveness of nonlocal attention detail capture pose optimization and channel space combination optimization nonrigid deformation modeling by conducting ablation studies.
[0047] To evaluate the proposed dynamic skinning weights and adaptive vertex offset adjustment, we compared the following methods: removing the non-local attention detail capture pose optimization module and the channel space combined optimization non-rigid deformation modeling module (without both); removing the non-local attention detail capture pose optimization module (without NLB); removing the channel space combined optimization non-rigid deformation modeling module (without CSC); and our own method. It can be seen that without the non-local attention module (without NLB), some features are lost in detail extraction. For example, in the hand detail extraction and representation stage, adding the non-local attention module allows for the acquisition of more hand detail features during reconstruction. Also, in leg simulation with trousers, adding the non-local attention module results in clearer trouser edges and legs that better match realistic images. Without the channel space combined attention module (without CSC), there are deficiencies in the non-rigid deformation perception stage of digital human body reconstruction. After adding the channel space combined attention module, the belt simulation more closely resembles a realistic natural curve, rather than a vertical downward curve. It can also be seen that when both nonlocal attention modules and channel-space combined attention modules are added simultaneously, the optimization for capturing multi-scale features and accurately perceiving non-rigid deformations is additive. Ablation experiments show that adding both nonlocal attention modules and channel-space combined attention modules is necessary and reasonable; their simultaneous addition does not cancel each other out but rather complements each other, resulting in a more realistic and natural reconstruction of the digital human body.
[0048] Table 2 shows the ablation experiment data. Quantitatively, it demonstrates that when both channel-space combined attention modules and non-local attention modules are added, the complete neural network model outperforms the single-module model in all three evaluation metrics: PSNR, SSIM, and LPIPS. Furthermore, even with only a single module, the performance surpasses HumanNeRF. The table provides data-driven confirmation of the effectiveness of the channel-space combined attention module and the non-local attention module, as well as the true performance of the complete model.
[0049] Table 2 presents the quantitative results, comparing the results of the non-local attention detail capture pose optimization module (w / o NLB) and the non-channel space combination optimization non-rigid deformation modeling module (w / o CSC).
[0050] Table 2 PSNR↑ SSIM↑ LPIPS*↓ Ours (with / no both) 30.30 0.9681 34.81 Ours (with / o NLB) 30.46 0.9701 34.42 Ours (without CSC) 30.39 0.9699 34.37 Ours (full) 30.62 0.9737 34.21 Based on the methods provided above, this paper also provides a digital human body reconstruction device for neural radiation fields, such as... Figure 2 As shown, the device includes: The decoupling module 210 is used to decouple the inverse pose transformation from the human body representation, obtain a coarse deformation through rigid deformation driven by the skeleton, and learn the non-rigid deformation detail offset using a multi-layer neural network to map the target object video from the observation space to the normal space. The fusion feature vector acquisition module 220 is used to extract multi-scale features of the object parameters of the target object video using the non-local attention module in the neural radiation field, and to fuse the multi-scale features using the residual learning module to obtain a fusion feature vector, wherein the object parameters are human pose parameters. The channel-space joint attention optimization module 230 is used to accurately perceive and model non-rigid deformations by utilizing the channel attention module and the spatial attention module in the neural network. Training module 240 is used to train the neural radiation field using joint training with multiple loss functions to obtain the trained neural radiation field. The rendering module 250 is used to input the coordinate transformation matrix between the observation space and the normal space and the target object video into the trained neural radiation field to render the reconstructed target object video.
[0051] The beneficial effects obtained by the above-described device are the same as those obtained by the above-described method, and will not be repeated here.
[0052] This embodiment provides a computer device, the internal structure of which can be shown in the following diagram. Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection.
[0053] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0054] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0056] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0057] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0058] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.
[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.
[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0061] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.
[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.
[0063] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.
Claims
1. A method for digital human body reconstruction using neural radiation fields, characterized in that, The method includes: Based on the decoupling of inverse pose transformation and human representation, coarse deformation is obtained through rigid deformation driven by the skeleton, and non-rigid deformation detail offset is learned by using a multi-layer neural network to map the target object video from the observation space to the canonical space. The object parameters of the target object video are extracted using a nonlocal attention module in the neural radiation field, and the multi-scale features are fused using a residual learning module to obtain a fused feature vector, wherein the object parameters are human pose parameters. Accurate perception and modeling of non-rigid deformations are achieved by utilizing the channel attention module and spatial attention module in a neural network. The neural radiation field is trained by using joint training with multiple loss functions to obtain the trained neural radiation field. The coordinate transformation matrix from the observation space to the normal space and the target object video are input into the trained neural radiation field, and the reconstructed target object video is obtained by rendering. Specifically, based on the decoupling of inverse pose transformation and human representation, non-rigid deformation and skeleton-driven rigid deformation are used to determine the detail offset in the target object video, so as to map the target object video from the observation space to the normal space, including: Define the motion field of the target object and map the target object video from the observation space to the normal space; Based on the inverse pose transformation and human characterization, the motion field is decoupled to determine the rigid deformation driven by the skeleton, as follows: ,in, For rigid deformation representation, x is the spatial position of the joint of the target object, and d is the skeleton pose input of the target object; By using a multi-layer neural network to perform offset processing on rigid deformations driven by bones, we obtain the detail offset of non-rigid deformations, represented as: , d represents the detail offset, and d is the skeleton pose input of the target object.
2. The digital human body reconstruction method for neural radiation fields according to claim 1, characterized in that, Rigid deformation driven by the skeleton is a coarse deformation driven by standard skinning.
3. A digital human body reconstruction device based on neural radiation fields, characterized in that, The device includes: The decoupling module is used to decouple the inverse pose transformation from the human representation. It obtains a coarse deformation through rigid deformation driven by the skeleton and uses a multi-layer neural network to learn the non-rigid deformation detail offset in order to map the target object video from the observation space to the normal space. The fusion feature vector acquisition module is used to extract multi-scale features of the object parameters of the target object video using the non-local attention module in the neural radiation field, and to fuse the multi-scale features using the residual learning module to obtain a fusion feature vector, wherein the object parameters are human pose parameters. The channel-space joint attention optimization module is used to accurately perceive and model non-rigid deformations by utilizing the channel attention module and the spatial attention module in the neural network. The training module is used to train the neural radiation field using joint training with multiple loss functions to obtain the trained neural radiation field. The rendering module is used to input the coordinate transformation matrix between the observation space and the normal space and the target object video into the trained neural radiation field to render the reconstructed target object video. Specifically, based on the decoupling of inverse pose transformation and human representation, non-rigid deformation and skeleton-driven rigid deformation are used to determine the detail offset in the target object video, so as to map the target object video from the observation space to the normal space, including: Define the motion field of the target object and map the target object video from the observation space to the normal space; Based on the inverse pose transformation and human characterization, the motion field is decoupled to determine the rigid deformation driven by the skeleton, as follows: ,in, For rigid deformation representation, x is the spatial position of the joint of the target object, and d is the skeleton pose input of the target object; By using a multi-layer neural network to perform offset processing on rigid deformations driven by bones, we obtain the detail offset of non-rigid deformations, represented as: , d represents the detail offset, and d is the skeleton pose input of the target object.
Citation Information
Patent Citations
Three-dimensional human body reconstruction method and system capable of shaping nerve radiation representation
CN116805345A
Human body nonlinear deformation method based on dense graph attention network
CN117392291A
