Multi-view neural human body rendering

Abstract

An end-to-end neural human rendering tool (NHR) is provided for dynamic human body photography in multi-view scenarios. The Feature Extraction (FE) module employs PointNet++ to analyze and extract features based on structure and semantics. The Projection and Rasterization (PR) module maps 3D features onto the target camera to form a 2D feature map. The Rendering (RE) module renders the final image derived from the feature map at the new viewpoint, employing an anti-aliased CNN to handle holes and noise. A foreground human mask generated by the NHR at each newly synthesized viewpoint is used to construct the visible shell to handle dark areas such as textureless areas and black clothing.

Description

Technical Field

[0001] This invention relates generally to the field of image processing, and more particularly to multi-view neural human body rendering. Background Technology

[0002] There is a significant demand for generating high-quality 3D models of the human body in motion. Applications range from creating hyper-realistic representations in virtual and augmented reality to enabling holographic immersive telecommunications supported by the latest data transmission networks. Currently, most existing methods rely on the following modeling and rendering workflow: first, the 3D geometry of the performer is captured using an active (e.g., depth camera like Microsoft Kinect) or passive (e.g., multi-camera hemispherical cameras) system and stored as a 3D point cloud; then, this point cloud is triangulated, texture-mapped, compressed, streamed, and rendered on the viewing device. To achieve high-fidelity reconstruction, hemispherical camera-based systems require numerous densely sampled cameras to handle occlusion, textureless areas, and detailed geometry such as hands. Depth camera-based solutions like Holoportation remain limited by resolution and often require significant manual labor to achieve commercial-quality results.

[0003] Image-based modeling and rendering (IBMR) aims to interpolate new perspectives or rays from sampled viewpoints or rays, under the guidance of low-quality reconstruction. Early techniques such as Lumigraph used coarse geometric substitutes, such as planes or visible shells, to select sampled images or rays and then blend them. The quality of such techniques largely depends on the accuracy of the substitutes. Image-based visible shells utilize image-space ray ordering to avoid generating 3D substitutes. In practice, existing IBMR methods are susceptible to occlusion and fail to preserve fine details. Furthermore, the geometry of the substitute can be improved by fitting an adjustable 3D model. Skinned Multi-Person Linear (SMPL) models are a pioneering technique that pre-scans 1786 human body shapes and then learns the human model based on data including static pose templates, blending weights, pose-dependent blending shapes, identity-dependent blending shapes, and vertex-to-joint regressions. Subsequently, it estimates the shape parameters of the restored point cloud. However, SMPL uses a "bare-bones" model and cannot directly handle clothing or strong shape changes in complex poses. While the problem can be partially mitigated by shape deformation, shape deformation is quite sensitive to reconstruction noise and holes.

[0004] It should be noted that the information disclosed in this background section is intended only to facilitate understanding of the background technology of this invention, and therefore may contain information already known to those skilled in the art. Summary of the Invention

[0005] In view of the above-mentioned limitations of the prior art, the present invention provides a multi-view neural human body rendering method that overcomes these limitations. Other features and advantages of the inventive concept will become apparent from the following detailed description, or may be partially learned by practicing the inventive concept.

[0006] One aspect of the present invention relates to an image-based method for modeling and rendering a three-dimensional model of an object. The method may include: obtaining a three-dimensional point cloud at each frame of a synchronized multi-view video of the object; extracting feature descriptors for each point in the point cloud of the multiple frames, without storing the feature descriptors for each frame; generating a two-dimensional feature map of the target camera; and decoding the feature map into an image and a foreground mask using an anti-aliased convolutional neural network. The video may include multiple frames.

[0007] In some implementations, the point cloud may not be formed into a mesh through triangulation.

[0008] In some embodiments, the method may further include: using a 3D scanning sensor including a depth camera to obtain a 3D point cloud at each frame of a synchronized multi-view video of the object.

[0009] In some implementations, the method may further include: capturing synchronized multi-view video of the object using multiple cameras in multiple viewing directions; and reconstructing the three-dimensional point cloud at each frame of the video.

[0010] In some implementations, the method may further include taking the viewpoint direction into account when extracting the feature descriptor.

[0011] In some implementations, each point in the point cloud may include a color. The method may further include applying the recovered color of each point to the feature descriptor.

[0012] In some implementations, the method may further include: extracting the feature descriptors using PointNet++.

[0013] In some implementations, the method may further include removing the classification branch of PointNet++.

[0014] In some implementations, the feature description may include a feature vector with at least 20 dimensions.

[0015] In some implementations, the feature vector may contain 24 dimensions.

[0016] In some embodiments, the method may further include: generating a two-dimensional feature map of the target camera by mapping the feature descriptors to a corresponding target viewpoint, wherein gradient backpropagation of the two-dimensional feature map can be performed directly on the three-dimensional point cloud at the target viewpoint; and calculating a depth map of the target camera.

[0017] In some implementations, the method may further include: using U-Net to decode the feature map into an image and a foreground mask.

[0018] In some implementations, the method may further include: inputting the feature map and the depth map into a U-Net to generate the image and foreground mask.

[0019] In some implementations, the method may further include replacing the downsampling operation of U-Net with MaxBlurPool and ConvBlurPool.

[0020] In some implementations, the method may further include replacing the convolutional layers of the U-Net with gated convolutional layers.

[0021] In some implementations, the method may further include maintaining the translation invariance of the feature map.

[0022] In some implementations, the method may further include: training the convolutional neural network using training data captured by multiple sampling cameras, without labeling the training data.

[0023] In some implementations, the method may further include: training the convolutional neural network using a foreground mask of a true background image.

[0024] In some implementations, the method may further include performing a two-dimensional image transformation on the training data.

[0025] In some implementations, the method may further include: generating multiple masks for multiple viewpoints; and using the masks as self-shadows to reconstruct the visible shell of the mesh.

[0026] In some implementations, the method may further include: dividing the space of interest into a plurality of discrete voxels; classifying the voxels into two classes; and calculating a signed distance field to recover the mesh.

[0027] In some implementations, the object may be a performer.

[0028] In some embodiments, the method may further include: training the convolutional neural network using data from multiple performers; and training the convolutional neural network using data from the performers.

[0029] It should be understood that the above overview and the following details are for illustrative purposes only and are not intended to limit the invention claimed. Attached Figure Description

[0030] The accompanying figures are incorporated in and form a part of this document, serving to illustrate embodiments of the invention and, together with this document, to elucidate the disclosed principles. It is apparent that these figures only show some embodiments of the invention, and those skilled in the art can obtain figures of other embodiments without inventive effort based on these figures.

[0031] Figure 1 This is an example diagram of a neural human rendering tool (NHR) that generates photorealistic free-view video (FVV) from multi-view dynamic human body shooting content according to one embodiment of the present invention.

[0032] Figure 2 This is an example of an NHR processing flow that enables multi-view rendering by performing neural rendering on low-quality 3D spatiotemporal point clouds according to an embodiment of the present invention, and how the rendering results can be used to further improve multi-view reconstruction by repairing holes and textures.

[0033] Figure 3 An example diagram of an NHR consisting of the following three modules according to an embodiment of the present invention is shown: a feature extraction (FE) module for processing spatiotemporal point cloud sequences based on PointNet++; a projection and rasterization (PR) module for feature projection; and a rendering module for feature decoding based on U-Net.

[0034] Figure 4 This is an example diagram illustrating the geometric refinement of NHR according to one embodiment of the present invention.

[0035] Figure 5 This diagram illustrates a statistical hole repair technique that utilizes visible shell results to repair holes in terms of hole distance distribution, according to one embodiment of the present invention.

[0036] Figure 6 This is an example diagram comparing the rendering quality of different methods. In this example, NHR according to one embodiment of the present invention successfully handles complex situations such as hair, hands, feet, and basketballs, while 3D reconstruction methods fail to handle these.

[0037] Figure 7This is an example of a free-view video result processed using NHR for a high-difficulty dance scene according to an embodiment of the present invention.

[0038] Figure 8 This is an example diagram illustrating the FE features or color-coded features of different frames in a point cloud sequence after visualizing the point cloud sequence by color according to an embodiment of the present invention. Detailed Implementation

[0039] Various exemplary embodiments are described more fully below with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as being limited to the forms given herein. Rather, these embodiments are intended to enable a person to fully and thoroughly understand the invention and to fully convey the concepts of these exemplary embodiments to others skilled in the art.

[0040] Furthermore, the features, structures, and characteristics described herein can be combined in any suitable manner in one or more embodiments. To enable a thorough understanding of the invention, numerous specific details are set forth below; however, those skilled in the art will recognize that various embodiments can also be practiced with one or more of the specific details omitted, or with the addition of other methods, components, materials, etc. In various embodiments, to avoid interfering with the description of various aspects of the invention, well-known structures, materials, or operations are sometimes not shown or described in detail.

[0041] 1. Introduction

[0042] This invention improves IBMR through Neural Rendering (NR). Existing NR "fixes" visual artifacts using deep networks. For example, one method improves rendering by utilizing semantic information embedded in the captured image data. However, existing methods require large amounts of training data, i.e., densely sampled input images. Furthermore, NR can be applied to the geometry stage of traditional graphics rendering pipelines, for example, by directly refining the input 3D and texture data. Another approach proposes a neural texture technique for handling noisy 3D geometry. However, it cannot handle severe defects such as holes caused by occlusion. Moreover, almost all existing NR techniques are designed for static models, not dynamic ones. Brute-force training at different time points is neither efficient nor practical.

[0043] This invention provides an end-to-end neural human rendering tool (NHR) that achieves high-quality rendering using low-fidelity 3D point clouds of dynamic human models.

[0044] Figure 1This is an example diagram of a neural human rendering tool (NHR) that generates photorealistic free-view video (FVV) from multi-view dynamic human body shooting content according to one embodiment of the present invention.

[0045] Figure 2 This invention presents an NHR (Neural Reconstruction and Rendering) process for multi-view rendering by neural rendering of low-quality 3D spatiotemporal point clouds, and an example diagram illustrating how the rendering results can be used to further improve multi-view reconstruction through hole and texture patching. NHR is trained on multi-view videos and consists of three modules: Feature Extraction (FE); Projection and Rasterization (PR); and Rendering (RE). The FE module uses PointNet++ to analyze and extract features based on structure and semantics. The extracted features are used to establish correspondences between reconstructed models over time, even in the presence of strong topological structures or reconstruction inconsistencies. More importantly, it utilizes temporal coherence, eliminating the need for dense view sampling. The PR module forms a 2D feature map by mapping 3D features onto a target camera, where gradient backpropagation for this 2D map can be performed directly on the 3D point cloud. Finally, the RE module renders the final image using the feature map from a new viewpoint. Specifically, the RE module aims to improve translation invariance by using an anti-aliased CNN with gated convolutional layers to process incomplete and noisy geometries.

[0046] like Figure 2 As shown, the newly synthesized perspectives from NHR further improve 3D reconstruction. Specifically, the corresponding processing flow is improved to include an additional foreground human mask. High-fidelity visible shell reconstruction is achieved by rendering a dense set of new perspectives. Specifically, the visible shell reconstructed by NHR supplements the MVS point cloud and effectively handles dark areas such as textureless areas or black clothing. Comprehensive experiments show that NHR significantly outperforms the current state-of-the-art IBR technology and can reliably handle areas that are difficult to reconstruct using traditional methods, such as hands, hair, nose, and feet, even in dense shooting conditions.

[0047] 2. Related work

[0048] The rapid development of 3D scanning and reconstruction technology over the past decade has laid the foundation for 3D modeling of the human body and, more recently, rendering.

[0049] Reconstruction: Passive human body reconstruction schemes follow the traditional reconstruction process of shooting a person with numerous cameras. First, the intrinsic and extrinsic parameters of the cameras can be estimated using the structure-of-motion (SOMO) method. Sometimes, to further improve robustness and accuracy, the intrinsic parameters can be pre-calculated. Then, the point cloud of the human object can be extracted using the multi-view stereo (MVS) method. It is worth noting that the density and quality of the point cloud largely depend on the availability of texture: rich textures often result in dense reconstructions, while textureless or dark areas can lead to sparse and unreliable reconstructions. Recently, several methods have employed hemispherical cameras composed of pairs of stereo cameras, where each pair can obtain more reliable estimation results through stereo matching, partially addressing the textureless problem. Subsequently, efficient texture mapping is achieved by triangulating the point cloud into a mesh (e.g., using Poisson surface completion). In practical applications, in addition to the textureless problem, human body reconstruction faces additional challenges: the human body has a complex topological structure, which can cause occlusion, resulting in holes in the reconstructed image. In particular, when using brute force for surface completion, adhesion artifacts will be produced, such as an arm sticking to the torso, or fingers sticking together. To this day, even commercial solutions (such as 8i and DGene), and even when using hundreds of cameras, cannot achieve high-quality 3D reconstructions.

[0050] Parametric modeling: Other modeling approaches attempt to fit parametric models to the acquired image or point cloud. Several such models estimate the optimal human geometry using prior knowledge of shape, pose, and appearance. A strong assumption of this type of model is the "bare-skin" model: because clothing varies considerably and cannot be easily simplified into simple parametric models, all such models require the subject to wear tight clothing. For example, the well-known SMPL technique has this requirement. Even for videos or single images, the results from such models are reasonable. However, due to clothing limitations, such parametric models often require the subject to wear elaborate clothing, thus greatly limiting their applicability.

[0051] Rendering: If the goal is to render objects as realistically as possible from a new viewpoint (e.g., through Image-Based Modeling and Rendering (IBMR)), it may be possible to bypass the 3D reconstruction process. Such methods utilize a coarse geometry obtained through simpler methods such as multi-view stereo modeling or even self-shading shape reconstruction to interpolate the sampled viewpoint into a new perspective. The geometric substitute can be as simple as a plane or as complex as a parametric human figure, and the perspective interpolation can be efficiently achieved through viewpoint-dependent texture mapping or unstructured lumen map shaders. In the past, when display resolutions were relatively low, rendering artifacts could be "hidden" by adding blur or ghosting. Recent optical flow-based rendering techniques can partially improve rendering quality, but perceptible visual artifacts still occur at occlusion edges.

[0052] The method of this invention employs neural rendering, which has shown promising results in image synthesis. Unlike IBMR, NR aims to mitigate visual artifacts by learning from sampled images. For several image generation tasks, such as annotation, deblurring, and super-resolution, GAN-based methods have produced impressive results after learning image distributions. NR is used to bridge the gap between low-quality 3D reconstruction and high-quality image synthesis in dynamic 3D human subjects. For static scenes, NR can also be combined with traditional IBR methods to achieve viewpoint-dependent rendering, image-based relighting, mesh denoising, and correspondence matching simultaneously at both the voxel and point levels.

[0053] Recent generative CNNs aim to synthesize human appearance and / or joints. Such techniques can repair artifacts in filmed 3D performances and improve low-quality 3D face reconstruction, but they cannot handle dynamic models. On the other hand, since the rich variations in body shape and appearance over time are sufficient to compensate for the lack of sufficient viewpoints, the techniques of this invention utilize changes in shape over time to compensate for the sparsity of viewpoint sampling, thereby enabling the handling of dynamic models. Furthermore, the rendering results can be used to further improve the reconstruction results.

[0054] 3. Method Overview

[0055] The following section explains the annotation symbols. While active 3D sensing can be naturally integrated into this workflow by bypassing the reconstruction process, multi-view stereo (MVS) input is assumed here. The input to the NHR workflow consists of a synchronized multi-view video sequence of the performer. Composition, where c is the camera number, n c n is the total number of cameras, t is the frame number, and n is the number of frames. t Let c be the total number of frames. The intrinsic and extrinsic parameters of each camera c are assumed to be... and To facilitate training, the human foreground mask is extracted for all frames. In the MVS scenario, each frame Point clouds are constructed for each point. Each point in the point cloud is assumed to have a color, which is calculated by reprojecting it onto the image from the input viewpoint.

[0056] The first task of NHR is to synthesize high-quality new perspectives through the rendering process (Section 4). In addition to RGB color rendering, model refinement is facilitated by generating foreground masks (Section 5). Specifically, the initial point cloud sequence... Not only is it noisy, but it also contains many holes caused by occlusion. The model refinement process can effectively fill in the holes in the synthesized new perspective, thus making it usable for further improving the rendering effect. Figure 2 The diagram shows the iterative rendering model process.

[0057] 4. NHR rendering

[0058] 4.1 Rendering Process

[0059] The NHR rendering process consists of three modules: Feature Extraction (FE); Projection and Rasterization (PR); and Rendering (RE).

[0060] Figure 3 An example diagram of an NHR consisting of the following three modules according to an embodiment of the present invention is shown: a feature extraction (FE) module for processing spatiotemporal point cloud sequences based on PointNet++; a projection and rasterization (PR) module for feature projection; and a rendering module for feature decoding based on U-Net.

[0061] Feature Extraction: Existing point cloud neural rendering methods require learning feature descriptors beyond the original RGB colors for each 3D point. Unlike static 3D models, it has been observed that in dynamic human body photography, the reconstruction depends on the MVS technique, resulting in differences in the number and density of points in the recovered point cloud at each time point. This inconsistency leads to the following additional challenges: learning the feature descriptors for each point at each time point is computationally expensive and requires a large amount of storage space. Furthermore, the number of viewpoint cameras is relatively small, thus limiting the number of samples available for descriptor learning. This invention addresses this by utilizing all images at all time points. Specifically, it explores the semantic features of human body shape and their temporal coherence. These features are obtained through end-to-end supervised learning.

[0062] Specifically, Pointnet++ can be effectively used as a feature extraction tool. In the MVS scenario, the appearance at different viewpoints can vary due to factors such as lighting direction, fabric material, and skin reflection. Therefore, in the FE process, the viewpoint-dependent effect is mitigated by considering the viewpoint direction. Simultaneously, the recovered 3D point colors are applied as prior information. Equation 1 illustrates this FE process:

[0063]

[0064] Where, ψ fe This corresponds to Pointnet++.

[0065] In this embodiment of the invention, the classification branch in the original network is removed, and only the segmentation branch is retained as the FE branch. The point cloud and its features at each time step are used as input to obtain the feature descriptor D. t V = {v} i} indicates the direction point The (normalized) view direction, where o is the projection center (CoP) of the target view camera, and {·} represents the combination of the point's color and the normalized view direction as the initial attribute (or feature) of the point provided to ψ. fe The joint operation. The coordinates of the point are obtained through... Normalization.

[0066] Projection and Rasterization: After obtaining the feature descriptor D of the point cloud, a new perspective is synthesized. For intrinsic and extrinsic parameters... and Given a target camera, the point cloud is projected onto the camera, and the scattered points are linearly projected onto the pixel coordinates on the image plane. This step rasterizes each point into pixel blocks. A Z-buffer is used to maintain the correct depth order, thus preserving the correctness of occlusion. This yields the projected 2D feature map S: in, For P t The i-th point in The feature descriptor after Z-buffer depth sorting to pixel (x,y). It's important to note that the projected points only cover a small number of pixels in the target image, while other pixels are assigned a learnable default feature vector θ. d The complete PR process for generating a two-dimensional feature map S. pr It can be represented as:

[0067]

[0068] Where E is the depth map of the current viewpoint.

[0069] Rendering: The feature map S generated above encodes the new perspective of the target. In the final rendering stage (RE), S is decoded into the corresponding RGB image and foreground mask using a convolutional neural network (CNN).

[0070] Recently, the U-Net architecture, consisting of a multi-layer encoder / decoder structure employing skip-layer connections, has achieved great success in image denoising, deblurring, and style transfer applications. In this embodiment of the invention, U-Net is used to decode S. It should be noted that the point cloud obtained by MVS is relatively sparse, and the projected feature map contains holes, and even perspective artifacts exist in areas where foreground points are missing. Such artifacts are treated as semantic noise. When using NR, the goal is to remove such incorrect pixels, and gated convolutional layers are used instead of the convolutional layers in U-Net. Specifically, the goal is to train the network to identify the location of such semantic noise, and then use an attention masking mechanism in the convolutional layers to correct the feature map.

[0071] The depth map E generated by PR contains rich information about scene geometry. Specifically, abrupt changes in depth values, especially low depth values, are a significant indicator of semantic noise. Therefore, to reduce semantic noise, S and standard normalized depth maps are used. All of these serve as inputs to the RE network:

[0072]

[0073] Where, ψ render This refers to the modified U-Net.

[0074] ψ render The final layer outputs an image through four channels. The first three channels generate an RGB image I*, and the last channel generates a foreground human mask M* using an Sigmoid curve function.

[0075] It is important to note that NHR is designed to render human objects along any viewpoint direction, meaning that human objects can appear anywhere within the image. This implies that neural rendering should maintain the translation invariance of the feature map S. In this embodiment, the inconsistencies caused by target camera translation are mitigated by replacing the downsampling operations (including pooling layers and convolutional layers with strides) of the original U-Net with MaxBlurPool and ConvBlurPool. To briefly reiterate, existing techniques employ Gaussian blur during downsampling to enhance the anti-aliasing capabilities of the feature map. In this embodiment, translation invariance is significantly improved.

[0076] 4.2 Network Training

[0077] To acquire training data, a multi-camera hemispherical system consisting of up to 80 synchronized, industrial-grade high-resolution cameras was employed. Consistent with traditional image-based rendering, these cameras are referred to as sampling cameras. A green screen was used for the hemispherical cameras to facilitate foreground segmentation. All cameras were pointed inwards at the performer, but as described in Section 6, most cameras could only capture a portion of the performer, not their entirety. A ground truth foreground mask was first obtained through chroma keying-based segmentation, followed by manual restoration. All cameras were pre-calibrated intrinsically using a checkerboard pattern via structure-of-motion reconstructive testing, and extrinsically calibrated using a patterned dummy model.

[0078] For training purposes, one of the sampling cameras is set as the target camera to utilize the background image. and Supervised training is performed. As described in Section 4, the end-to-end network updates parameters through backpropagation of the loss function gradient from the 2D image to the 3D point cloud. Since the goal is to render the target viewpoint in a way that is as close to photorealistic as possible, perceptual loss and L1 loss are used as the loss functions:

[0079]

[0080] Where, n b For batch size, and Let ψ be the output image after rendering the i-th image and the mask for the minimum batch. vgg (·) is used to extract feature maps from layers 2 and 4 of the VGG-19 network, which is pre-trained on the ImageNet dataset.

[0081] Since the number of sampling cameras constituting a hemispherical camera system is quite limited (up to 80 cameras in one implementation), the training data is augmented through two-dimensional image transformations to train the network to be more adaptable to any viewpoint. Specifically, three types of transformations are employed: random translation, random scaling, and random rotation. All of these transformations can be easily implemented by modifying the intrinsic / extrinsic parameters of the cameras and re-rendering the 3D point cloud.

[0082] Conceptually, depending on the type of input data, two training methods can be employed: individual training and shared training. The former trains each performer individually. This method is suitable for situations where only a small number of performers are filmed, or where network fine-tuning is needed for specific performers. The latter trains a large number of performers, and the training process shares the same ψ... renderThe network weights are distributed, but different FE weights are generated. This allows the FE module to learn the performer's unique geometry and appearance features separately, while maintaining a unified feature embedding space. The shared rendering module is further used to decode the feature descriptors into the target image.

[0083] In this embodiment of the invention, a shared training method is first used for network self-training, followed by individual training for fine-tuning the network for each performer. For a new performer, ψ is first determined. render Then, using a shared training method, the FE module is trained from scratch. After five rounds of shared training, individual training begins. This strategy significantly accelerates the training process.

[0084] 5. Geometric Refinement

[0085] In traditional MVS methods, especially those employing a sparse set of sampling perspectives, occlusion is the most detrimental factor in reconstruction artifacts. Clearly, if no camera can see a certain area, even in dense MVS scenarios, large areas of holes will still appear. The human body can exhibit a variety of occlusion scenarios (e.g., an arm obscuring the torso, the inner thigh being obscured by the outer thigh), so the recovered geometry is highly likely to contain holes. NHR can patch these holes at almost any point in time and produce satisfactory results. However, when rendering video sequences, even if each individual frame produces a reasonable result, the synthesized area initially corresponding to the holes will exhibit flickering artifacts in the overall result. Existing NR techniques also suffer from similar artifacts. This problem can be mitigated by reducing holes through geometric refinement.

[0086] One possible solution is to directly fit the restored 3D point cloud using a parametric model. The fitted model can then be used to patch the holes, while the rest retains the original restored form. A significant limitation of this approach is that most existing parametric models, such as skinned multi-person linear models, employ a "naked skin" model—a model of a person covered in tight clothing. For models wearing real clothing, the fitting process described above can lead to significant errors. Furthermore, using a "naked skin" model to patch holes corresponding to clothing can result in significant visual artifacts, such as… Figure 4 The discontinuity shown.

[0087] Since NHR also generates an auxiliary human mask for each new viewpoint, a visible shell method is employed. Compared to detailed and noisy RGB images, the masks generated by the NHR network have a much higher degree of cleanliness. During geometry refinement, a dense set of new viewpoints is first rendered, and then a visible shell reconstruction based on spatial sculpting or self-shadowing shape reconstruction (SfS) is performed using the resulting mask as a self-shadow. The approximate visible shell can then be used to patch holes.

[0088] Figure 4 The following is an example of geometric refinement using NHR according to an embodiment of the present invention, wherein (a) shows the visible shell result obtained by NHR from a dense rendering perspective (using self-shadowing to reconstruct the shape); (b) shows the original three-dimensional reconstruction effect achieved by SfM; (c) shows the high coherence of the visible shell result and the SfM geometry; and (d) shows the repair effect after repairing the hole in (b) using (a). Figure 4 The bottom row is a close-up view of the NHR results with and without geometry refinement.

[0089] Mask and Shape Generation: To briefly reiterate, the MVS point cloud is used to train the rendering module to output a mask similar to an RGB image. First, the mask is rendered for a set of uniformly sampled new viewpoints pointing towards the performer, where each new viewpoint has known camera intrinsic and extrinsic parameters. Specifically, the mask is rendered at a resolution of 800. Subsequently, the human body mesh is reconstructed using voxel-based SfS, where the space of interest is first divided into discrete voxels, and then each voxel is classified into "performer interior" and "performer exterior" categories. Finally, SfS calculates the signed distance field, where the final mesh is recovered using a marching cube algorithm. In one embodiment of the invention, 80 new viewpoints are generated, resulting in approximately 160 viewpoints for constructing the visible shell. The mask generated by NHR is actually a mask, not a binary mask, and the corresponding binary mask is generated by setting a uniform threshold. In one embodiment, the threshold is set to 0.4.

[0090] Point Sampling and Coloring: SfS Results It contains only geometric shapes, no colors. For Each point in Its color can be utilized in the MVS point cloud P t The calculation is performed using the nearest point within the range. Therefore, we can obtain the result... corresponding

[0091] Hole Repair: While SfS results can be directly used for refining geometry, it is well known that volume reconstruction is limited by its resolution and the number of input viewpoints. Furthermore, shapes recovered from self-shading generally appear as polygons. Therefore, only for... P in t The hole U inside t (Right now Repair it.

[0092] Figure 5 This diagram illustrates a statistical hole repair technique that utilizes visible shell results to repair holes in terms of hole distance distribution, according to one embodiment of the present invention.

[0093] Specifically, assuming This represents the Euclidean distance between two three-dimensional points. The hole repair scheme is based on the following observation: for U... t Every point in Its relationship with P t The point closest to it Between Usually greater than the distance The distances between points in the middle. Therefore, statistical methods are used to find U. t And P t It should be farther than most points, where U t Point cloud China and P t The subset corresponding to the holes in. Therefore, it can be determined by... The point in P and its position t Statistical analysis was performed on the distances between the nearest points to obtain the U-shaped distance used for repairing the hole. t approximation.

[0094] set up And the threshold τ1 is set as:

[0095]

[0096] Where, λ t It is a weighting factor, and in one implementation it is set to 0.2.

[0097] Subsequently, Zhongyu Count the number of points whose distance is less than τ1, and record the result as τ1. in, Conceptually speaking, This point belongs to U t The probability is inversely proportional to the probability.

[0098] Finally, regarding sets Calculate all points A histogram with 15 bars, evenly separated by their maximum distance values, was observed. It was found that in all cases, the first bar contained significantly more points than the second bar, thus directly aiding in the identification of nearest points. Accordingly, the maximum distance in the first bar was used as the basis for identifying U. t The threshold τ2 for selection:

[0099]

[0100] like Figure 4 As shown, the SfS-based geometry refinement using hole repair technology significantly reduces flickering during viewpoint changes. It is worth noting that since the quality of the final geometry depends on the reliability of τ1 and τ2, artifacts may still exist.

[0101] 6. Experimental Results

[0102] All experiments were conducted using 3D dynamic human body data collected by a multi-camera hemispherical system containing up to 80 cameras. All cameras were synchronized and filmed at a resolution of 2048×1536 and a shooting speed of 25 frames per second. In this embodiment, five datasets were used, showing performers wearing different clothing and performing different actions. All video sequences were 8–24 seconds long. Specifically, sport1, sport2, and sport3 correspond to dumbbell lifting while wearing relatively tight clothing, dancing while wearing complex and easily deformable clothing, and basketball actions involving interaction between a player and a basketball, respectively. As described above, green screen keying was performed first, followed by manual restoration to extract the underlying true image mask for all viewpoints. For 3D reconstruction, MetaShape, one of the best commercial SfM software programs, was used to calculate the initial 3D point cloud for all frames.

[0103] In the network training and forward prediction steps, the target rendering resolution was set to 800×600. The FE module extracted 24-dimensional feature vectors. By default, four rounds of shared training were performed first, followed by another four rounds of personalized training for each performer to achieve fine-tuning. The Adam algorithm was used for optimization training with a learning rate of 7e-5, and a warm-up period was also set. All training was performed on a single 2080Ti, with a batch size of 2 due to GPU memory limitations. Shared training took approximately 24 hours, and personalized training took approximately 5 hours. After training, the network's execution speed reached 300ms / frame.

[0104] Comparison: NHR is compared with several traditional and NR methods.

[0105] True Background (GT): The size of the captured image data is adjusted to 800×600 resolution, consistent with the network's output resolution.

[0106] Point cloud rendering (PCR): The recovered color point cloud is directly projected onto the target camera. The PR module is used to render the projected pixels to form the final RGB image sequence.

[0107] Texture Mesh (TM): The point cloud is triangulated using MetaShape, and a texture map is constructed for all frames. The result is rendered onto the image using standard rasterization.

[0108] PCR+U-net (PCR-U): Projects the RGB point cloud onto the target viewpoint and feeds it directly into U-Net to refine the rendering results.

[0109] NHR with geometry refinement (NHR with GR): First, refine the geometry as described in Section 5, and then retrain the network for 3 rounds using the refined point cloud.

[0110] To verify the effectiveness of this method, rendering results with various algorithm outcomes were compared with ground truth (GT). For fairness, only foreground rendering was compared. Since NHR has predicted the mask for each target viewpoint, its results can be used to segment the foreground after NHR rendering. For other techniques such as PCR and IBR, the foreground can be directly separated from the background using a mesh.

[0111] Figure 6 This is an example comparing the rendering quality of different methods. In PCR-U, the use of U-Net partially reduces noise and repairs holes in the final rendered result, but the result suffers from excessive blurring in such areas. In contrast, NHR exhibits significantly less blurring. This is because the NHR network is specifically designed to extract spatial features that remain consistent across different time series, while most existing U-Net solutions are designed for static meshes. In other words, NHR technology can infer the missing geometric information in a particular frame based on other frames in a dynamic sequence. This method is suitable for common TM methods such as... Figure 6 The NHR network exhibits a particularly significant advantage in handling severely damaged 3D geometries, such as missing noses, missing fingers, and holes in the body. In fact, even when the GT mask contains errors such as jagged edges or broken edges, the NHR network can still manage to repair the mask and generate a higher-quality mask. For example... Figure 6 The quantitative comparisons show that NHR outperforms other methods in both PSNR and SSIM. For further results and comparisons, please refer to the supplementary materials.

[0112] Figure 7This image illustrates a free-view video result processed using NHR (Non-High-Resolution Image) for a challenging dance scene according to an embodiment of the present invention. The red top in the image presents a significant challenge to 3D reconstruction. Figure 7 As shown, NHR can achieve high-quality FVV rendering results despite poor reconstruction results.

[0113] like Figure 4 As shown, the geometry refinement (GR) process is performed through, as... Figure 4 The filling of holes caused by occlusion further improves the quality of the model and rendering. This is particularly important for avoiding flickering in NR-based rendering: after hole filling, the rendering tool can correctly handle depth ordering, thus preventing unwanted perspective artifacts. Existing NR techniques take measures to prevent perspective in image space, which causes the "patched" part to change sharply during viewpoint transitions, resulting in flickering. By filling holes, this flickering can be significantly reduced. For the sport1 dataset, rendering quality is improved by GR: PSNR increases from 29.2 to 31.7, and SSIM increases from 0.964 to 0.974.

[0114] Other applications: The compositing of dynamic human perspectives can give rise to a range of new rendering techniques. For example, the bullet-time effect, first seen in the feature film *The Matrix*, has been widely used in creating the amazing illusion of time stopping in feature films and television commercials. However, currently available dome camera systems cannot meet the needs of film quality production, and the 3D reconstruction shown in this embodiment of the invention cannot achieve high-quality rendering. Since NHR can reduce or even eliminate strong visual artifacts, it can be used to create bullet-time effects. It is particularly noteworthy that existing NR technology suffers from strong flicker artifacts, which are detrimental to creating bullet-time effects. The geometry refinement process guided by NHR can significantly reduce flicker. In addition to the ordinary bullet-time effect (i.e., changing the viewpoint while maintaining a fixed time), NHR can be used to change both time and viewpoint simultaneously, meaning that the performer continues their actions while the viewer changes their perspective.

[0115] Because the features extracted through the FE module maintain spatiotemporal coherence throughout the sequence, they have the potential to be used to generate animation meshes (AMs) from point cloud sequences. AMs, in principle, have consistent vertices and connectivity. The mesh generated from the first frame can be used as a template, while the extracted features can be used to establish the temporal correspondences of the vertices to guide the generation of the AM.

[0116] Figure 8This is an example diagram illustrating the FE features or color-coded features of different frames in a point cloud sequence visualized by color according to an embodiment of the present invention. Even though the point cloud of each frame is constructed in a non-coherent manner, the illustrated features still exhibit strong semantic coherence, thus demonstrating the effectiveness of the FE module and its application potential in correspondence matching.

[0117] 7. Conclusion

[0118] According to embodiments of the present invention, a novel neural human rendering tool (NHR) is provided for high-quality rendering of dynamic 3D human models captured by a multi-view hemispherical camera system. Compared to most existing neural rendering (NR) techniques that focus on static scenes or objects, NHR explicitly utilizes temporal correspondences to compensate for the sparsity of spatial / angular sampling. By using PointNet++ for feature extraction, semantic tagging for feature matching, and anti-aliased CNN for rendering, the method of the present invention establishes a spatiotemporal 3D correspondence, and subsequently utilizes this correspondence to significantly improve rendering quality even in cases of poor 3D reconstruction. Specifically, NHR exhibits superior performance for objects such as hair, hands, nose, and feet, which are difficult to reconstruct correctly even when using densely sampled cameras or active 3D sensors. Furthermore, by using correspondences, the flicker artifact problem present in existing NR methods based on per-frame processing is also mitigated.

[0119] In addition, it has been demonstrated that the new perspective of NHR compositing can be used to further improve the 3D reconstruction effect by reconstructing shapes through self-shading.

[0120] 8. Quantitative comparison

[0121] Table 1 shows the scene parameter details in the dataset of this invention. This section provides a quantitative comparison with other methods, including PCR, TM, PCR-U, image-based rendering (IBR), and NHR with and without GR. PSNR, SSIM, and MSE are used to measure the gap difference between GT images. Tables 2, 3, and 4 show the average PSNR, SSIM, and MSE results for each scene using different methods.

[0122] Scene Number of cameras Frames Number of images Average points sport1 56 200 11200 17.8w sport2 56 200 11200 30.6w sport3 56 200 11200 29.4w basketball 72 195 14040 16.5w dance 80 600 48000 25.6w

[0123] Table 1

[0124] sport1 sport2 sport3 dance basketball PCR 17.458 20.002 19.760 20.664 19.463 PCR-U 25.104 25.654 26.006 22.796 24.130 TM 22.419 22.103 22.318 20.749 21.947 IBR 22.632 22.369 22.644 20.269 22.120 NHR without GR 26.951 26.713 27.218 22.602 24.660 NHR with GR 27.385 26.925 26.889 23.367 25.138

[0125] Table 2

[0126] sport1 sport2 sport3 dance basketball PCR 0.827 0.855 0.849 0.890 0.839 PCR-U 0.963 0.968 0.969 0.967 0.964 TM 0.957 0.954 0.956 0.957 0.955 IBR 0.960 0.958 0.961 0.961 0.961 NHR without GR 0.975 0.975 0.976 0.969 0.969 NHR with GR 0.977 0.979 0.975 0.973 0.971

[0127] Table 3

[0128] sport1 sport2 sport3 dance basketball PCR 0.0250 0.0133 0.0147 0.0139 0.0150 PCR-U 0.0034 0.0032 0.0029 0.0086 0.0040 TM 0.0067 0.0067 0.0066 0.0113 0.0067 IBR 0.0064 0.0063 0.0062 0.0144 0.0065 NHR without GR 0.0020 0.0023 0.0021 0.0087 0.0038 NHR with GR 0.0021 0.0024 0.0023 0.0074 0.0035

[0129] Table 4

Claims

1. An image-based method of modeling and rendering a three-dimensional model of an object, characterized in that, The method includes: Obtain a 3D point cloud at each frame of a synchronized multi-view video of an object, wherein the video comprises multiple frames; The feature descriptor of each point in the point cloud of the multiple frames is extracted, but the feature descriptor of each frame is not saved. After obtaining the feature descriptor of the point cloud, a new perspective is synthesized. Projecting the point cloud onto the target camera to generate a two-dimensional feature map of the target camera after projection wherein: The projected points only cover a small number of pixels in the target image, while other pixels are assigned with a default feature vector learned The complete PR process for generating a two-dimensional feature map is represented as: is represented as: In the formula, is the MVS point cloud, is the feature descriptor, and is the intrinsic and extrinsic parameters of the target camera; And using an anti-aliased convolutional neural network to decode the feature map into an image and a foreground mask; For a set of uniformly sampled new viewpoints pointing at the performer, render the foreground mask, where each new viewpoint has known camera intrinsic and extrinsic parameters; By performing voxel-based SfS to reconstruct the human body mesh, the signed distance field is calculated using SfS. For the output of SfS Each point in Color is used in MVS point clouds Calculate using the nearest point within; MVS point cloud in is patched. The holes in are patched.​ 2. The method of claim 1, wherein, The point cloud is not formed into a mesh through triangulation.

3. The method of claim 1, wherein, Further includes: Using a 3D scanning sensor including a depth camera, a 3D point cloud is obtained at each frame of a synchronized multi-view video of the object.

4. The method of claim 1, wherein, Further includes: Synchronized multi-view video of the object is captured using multiple cameras from multiple perspectives. as well as Reconstruct the 3D point cloud at each frame of the video.

5. The method of claim 4, wherein, Further includes: When extracting the feature descriptor, the viewpoint direction is taken into account.

6. The method of claim 1, wherein, Each point in the point cloud includes a color, and the method further includes: The restored colors of each point are applied to the feature descriptor.

7. The method of claim 1, wherein, Further includes: The feature descriptors are extracted using PointNet++.

8. The method of claim 7, wherein, Further includes: Remove the classification branch from PointNet++.

9. The method of claim 1, wherein, The feature description includes a feature vector with at least 20 dimensions.

10. The method of claim 9, wherein, The feature vector has 24 dimensions.

11. The method of claim 1, wherein, Further includes: By mapping the feature descriptors to the target viewpoint to generate a two-dimensional feature map of the target camera, gradient backpropagation of the two-dimensional feature map can be performed directly on the three-dimensional point cloud at the target viewpoint. And calculate the depth map of the target camera.

12. The method of claim 1, wherein, Further includes: Using U-Net, the feature map is decoded into the image and the foreground mask.

13. The method of claim 12, wherein, Further includes: The feature map and depth map are input into U-Net to generate the image and the foreground mask.

14. The method of claim 13, wherein, Further includes: Replace the downsampling operations of U-Net with MaxBlurPool and ConvBlurPool.

15. The method of claim 13, wherein, Further includes: Replace the convolutional layers of U-Net with gated convolutional layers.

16. The method as described in claim 13, characterized in that, Further includes: The feature map is kept translation invariant.

17. The method of claim 1, wherein, Further includes: The convolutional neural network is trained using training data captured by multiple sampling cameras, without labeling the training data.

18. The method of claim 17, wherein, Further includes: The convolutional neural network is trained using a foreground mask based on the background image.

19. The method of claim 18, wherein, Further includes: Perform two-dimensional image transformation on the training data.

20. The method of claim 1, wherein, Further includes: Generate multiple masks from multiple viewpoints; as well as The mask is used as a self-shadow for reconstructing the visible shell of the mesh.

21. The method of claim 20, wherein, Further includes: Divide the space of interest into multiple discrete voxels; The voxels are divided into two categories; as well as Calculate the signed distance field to reconstruct the mesh.

22. The method of claim 1, wherein, The subject is the performer.

23. The method of claim 22, wherein, Further includes: The convolutional neural network is trained using data from multiple performers; and training the convolutional neural network using data of the performer.

Images