Non-rigid object three-dimensional reconstruction method and system based on single-view neural radiation field

Abstract

The invention discloses a non-rigid object three-dimensional reconstruction method based on a single-view neural radiation field, and the method comprises the steps: firstly inputting a single RGB image, extracting global features through a pre-trained deep learning backbone network, and generating a mask image of an object in combination with a semantic segmentation network; secondly, predicting volume density and color distribution by adopting a neural radiation field variant pixelNeRF, embedding a dynamic deformation field and a parameterized template to constrain shape rationality, and solving the problem of shape ambiguity in single-view reconstruction; thirdly, extracting a grid model through a Marking Cubes algorithm, complementing shielding and details by using a reconstruction algorithm, and enhancing the sense of reality of the surface by using a texture super-resolution technology; and finally, providing a user interaction interface for local optimization, and correcting a motion posture through a differentiable physical engine to generate a drivable three-dimensional animation model. By combining the single-view neural radiation field and the dynamic deformation modeling technology, the method has the advantages of high precision, high efficiency, support of interactive editing, capability of generating driving animations and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method for non-rigid objects based on a single-view embedded dynamic deformation field and a parameterized template neural radiation field variant, specifically a three-dimensional reconstruction method for non-rigid objects based on a single-view neural radiation field. Background Art

[0002] Currently, the main methods for 3D reconstruction of non-rigid objects include 3D scanning, multi-view reconstruction, and single-view reconstruction. 3D scanning requires a 3D scanner, which, while highly accurate and effective, is expensive and complex, making it difficult to popularize.

[0003] Multi-view 3D reconstruction methods have emerged with the development of deep learning technology and are currently the mainstream approach for reconstructing non-rigid objects. Multi-view reconstruction has strong geometric constraints, high accuracy, and can be used to reconstruct complex deformations. Key approaches include the neural radiance field method, which uses multi-scale, multi-view image feature priors combined with an attention mechanism, and 3D reconstruction methods based on structure from motion-multiview stereo (SfM-MVS). However, these methods typically require setting up a camera matrix and synchronizing multiple camera systems in the environment, resulting in significant financial and time costs. Furthermore, in many special cases, it is impossible to obtain multi-view images of an object. Consequently, single-view reconstruction techniques have gradually evolved.

[0004] Single-view reconstruction requires simple input, often requiring only a single RGB image. It excels in 3D reconstruction of rigid objects, but 3D reconstructions of non-rigid objects often suffer from blur and occlusion. This lack of appropriate physical and geometric constraints and insufficient model generalization results in poor reconstruction. Currently, leading single-view 3D reconstruction methods for non-rigid objects include semantic relationship-based neural implicit field 3D reconstruction, fused convolutional Catmull-Rom spline free-form deformation (CR-FFD), and biased Transformer capsule networks. However, occlusion can lead to feature matching confusion, making it difficult to separate geometry from appearance, resulting in undesirable artifacts in the rendered output. Summary of the Invention

[0005] In order to solve the problem of single vision in the existing technology Figure 3 In order to solve the problems of high shape ambiguity, difficulty in modeling dynamic objects, lack of texture details and insufficient interactivity in the three-dimensional reconstruction method, the present invention provides a three-dimensional reconstruction method for non-rigid objects based on single-view neural radiation field. By combining the single-view neural radiation field with dynamic deformation modeling technology, it achieves high precision, high efficiency, supports interactive editing, and can generate driven animation. Technical effects.

[0006] According to one aspect of the present invention, a method for 3D reconstruction of a non-rigid object based on a single-view neural radiation field is provided, comprising: Step 1: Input a single RGB image and use a pre-trained deep learning backbone network to extract features, including the object's texture, local geometry, posture, and global structure. Combined with a semantic segmentation network, it segments the object's contours and separates background interference. The discriminative features learned by the classification network are semantically projected into pixel space to generate a binary mask image dataset. Step 2: Using pixelNeRF, a variant of Neural Radiance Field (NeRF), we predict 3D volume density and color distribution from a single-view image to preliminarily construct a primitive 3D model, embedding prior knowledge of non-rigid objects, including dynamic deformation fields and parameterized templates. Step 3: Extract the mesh model from the volume density output by pixelNeRF using the Marching Cubes algorithm. Use a reconstruction algorithm to complete the missing geometric structure or hair details due to occlusion. Combined with a global optimization strategy to improve the reconstruction completeness, the texture super-resolution technology based on the generative adversarial network is used to enhance the surface details of the model. Step 4: Annotate key points and / or edit local areas based on the user interaction interface, locally optimize the user-annotated area through the diffusion model, and provide real-time rendering feedback.

[0007] As a further technical solution, the deep learning backbone network in step 1 adopts a hybrid architecture of ResNet-50 and Vision Transformer, wherein ResNet-50 extracts local features of a single image through its residual connection structure, and uses a pre-trained convolutional neural network layer to capture the texture details and local geometric features of the object surface; Vision Transformer divides the input image into blocks and establishes long-distance dependencies through a self-attention mechanism to capture the global structure and posture features of the object; the output features of the two networks are spliced ​​and reduced in dimension through a feature fusion module to form a unified feature representation containing multi-scale information.

[0008] As a further technical solution, the classification network outputs the category probability of the object through a fully connected layer and a Softmax function based on the features extracted by the deep learning backbone network, and embeds the category probability as prior knowledge into the semantic segmentation network; the semantic segmentation network uses the fused features output by the deep learning backbone network as input, adopts the cascade structure of Mask R-CNN and U-Net, and outputs a binary mask image dataset with pixel-level accuracy.

[0009] As a further technical solution, the initial construction of the original three-dimensional model in step 2 further includes: The binary mask image dataset of the single-view image is used for feature encoding and projection to extract the volume features of the image pixels in the plane coordinates; Neural radiance field prediction using learnable positional encodings and extracted volumetric features; Based on the prediction results, preliminary three-dimensional reconstruction is performed in combination with the volume rendering principle of the neural radiation field.

[0010] As a further technical solution, the method further includes: We establish a spatial deformation representation of a Gaussian sphere based on a dynamic deformation field, parameterize the local deformation through learnable position encoding, and use a physics-inspired regularization term to constrain the continuity and smoothness of the deformation. A hybrid multi-model architecture combining SMPL-X, SMAL, and AniCap is used to generate parameterized templates to support 3D modeling of non-rigid objects, providing kinematic constraints and biomechanical priors. The rendered volume model is jointly optimized based on a dynamic deformation field and a parameterized template.

[0011] As a further technical solution, the reconstruction algorithm in step 3 includes Poisson global optimization reconstruction and multi-layer Shell-Based hair modeling. The Poisson global optimization reconstruction is used to complete the occluded part of the object, and the multi-layer Shell-Based hair modeling is used to complete the occluded hair and maintain a natural transition of the hair flow.

[0012] As a further technical solution, step 4 further includes: The motion posture is physically corrected through a differentiable physics engine, supporting real-time skeleton-driven animation generation, obtaining FBX animation files that can drive virtual images, and completing sparse perspective 3D reconstruction of non-rigid objects.

[0013] According to one aspect of the present invention, a non-rigid object 3D reconstruction system based on a single-view neural radiation field is provided, comprising: The feature extraction and segmentation module takes a single RGB image as input and uses a pre-trained deep learning backbone network to extract features, including the object's texture, local geometry, posture, and global structure. It then uses a semantic segmentation network to segment object contours and remove background interference. It then semantically projects the discriminative features learned by the classification network into pixel space to generate a binary mask image dataset. Neural Radiance Field Reconstruction Module, which uses pixelNeRF, a variant of Neural Radiance Field (NeRF), to predict 3D volume density and color distribution from a single-view image, initially construct the original 3D model, and embed prior knowledge of non-rigid objects, including dynamic deformation fields and parameterized templates; The geometry optimization and texture enhancement module uses the Marching Cubes algorithm to extract a mesh model from the volume density output by pixelNeRF. It then uses a reconstruction algorithm to complete some geometric structures or hair details lost due to occlusion. It also combines a global optimization strategy to improve reconstruction integrity and enhances surface details using texture super-resolution technology based on a generative adversarial network. The user interaction and output module is used to mark key points and / or edit local areas based on the user interaction interface, locally optimize the user-marked area through the diffusion model, and provide real-time rendering feedback.

[0014] According to one aspect of the present invention, a device for 3D reconstruction of non-rigid objects based on a single-view neural radiation field is provided, comprising: one or more graphics processors; a storage device for storing one or more program modules; The storage device stores program instructions executed by the graphics processor, and the graphics processor calls the program instructions to execute the non-rigid object three-dimensional reconstruction method based on the single-view neural radiation field.

[0015] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method for three-dimensional reconstruction of non-rigid objects based on a single-view neural radiation field.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a technology that combines single-view neural radiation field variants with dynamic deformation modeling to achieve three-dimensional reconstruction of non-rigid objects. Based on the pixelNeRF architecture and physical constraint optimization algorithm, it can quickly reconstruct a three-dimensional model with reasonable deformation directly from a single RGB image. It has the advantages of high precision, high efficiency, support for interactive editing, and the ability to generate driven animations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic flow chart of a method for 3D reconstruction of non-rigid objects based on a single-view neural radiation field provided in an embodiment of the present invention.

[0019] Figure 2A schematic diagram of a non-rigid object 3D reconstruction system based on a single-view neural radiation field provided by an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of optimization comparison provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0023] 1 , the present invention provides a method for 3D reconstruction of non-rigid objects based on a single-view neural radiation field, comprising the following steps: Step S1: Take a single RGB image of a non-rigid object as input.

[0024] In one embodiment, the input of the present invention is a single RGB image of a non-rigid object, such as a human body, an animal, or other deformable object. To ensure the quality of reconstruction, it is recommended that the resolution of the input image be no less than 1920×1080 pixels, and that it be taken in an environment with uniform lighting and a simple background as much as possible to reduce interference with subsequent segmentation and feature extraction. For dynamic objects, such as animals or humans in motion, it is recommended to select frames with clear postures and less occlusion as input. In practical applications, images can be collected through common devices such as smartphones, digital cameras, or surveillance cameras, without relying on expensive professional camera equipment.

[0025] Step S2: Feature extraction using a pre-trained deep learning backbone network, combined with a semantic segmentation network to project discriminative features into the pixel space and generate a mask image.

[0026] In one embodiment, the input image is first subjected to feature extraction using a pre-trained deep learning backbone network. The deep learning backbone network adopts a hybrid architecture of ResNet-50 and Vision Transformer. ResNet-50 effectively extracts local features of a single image through its residual connection structure, and captures texture details and local geometric features of the object surface using pre-trained convolutional neural network layers. Vision Transformer processes the input image in blocks and establishes long-range dependencies through a self-attention mechanism, which is used to capture global structure and pose features of the object. The output features of the two networks are spliced and dimensionally reduced through a feature fusion module to form a unified feature representation containing multi-scale information. Among them, the features extracted by ResNet-50 include texture details (such as color, material) and local geometric features (such as edge, curvature) of the object surface, and the features extracted by Vision Transformer include global structure (such as overall shape) and pose features (such as joint angle, symmetry) of the object. The fused features retain texture, local geometry, global structure, and pose information, which are used for subsequent semantic segmentation and three-dimensional reconstruction.

[0027] The classification network outputs the class probability of the object based on the features extracted by the deep learning backbone network through a fully connected layer and a Softmax function, and embeds the class information as prior knowledge into the semantic segmentation network.

[0028] The semantic segmentation network performs accurate segmentation on the input image to generate a binary mask of the object. Specifically, the semantic segmentation network takes the fused features output by the deep learning backbone network as input and adopts a cascaded structure of Mask R-CNN and U-Net. Mask R-CNN first generates object candidate boxes through its region proposal network, extracts accurate local features using the RoI Align layer, and outputs preliminary object masks and bounding boxes. U-Net receives the intermediate feature maps of Mask R-CNN, gradually refines the segmentation edges through the encoder-decoder structure with skip connections while maintaining the spatial resolution, and finally outputs a pixel-level accurate binary mask image dataset. The mask and bounding box are used to constrain the geometric range of subsequent three-dimensional reconstruction and define the boundary conditions of the deformable region in physical simulation. This step effectively separates the target object from the background interference, ensuring that the subsequent reconstruction process focuses on the geometric and appearance modeling of the object itself.

[0029] Step S3: Using the neural radiance field variant pixelNeRF, the volume density and color distribution are predicted through the mask image, and the dynamic deformation field and parameterized template are embedded to constrain the rationality and dynamic changes of the object shape.

[0030] The proposed neural radiance field variant pixelNeRF maps 2D features to 3D ray space via learnable position encoding, adopts a layered sampling strategy, and jointly optimizes geometry and appearance using a differentiable rendering loss.

[0031] In one embodiment, a single-image mask image dataset is first used for feature encoding and projection, and a feature volume W is generated by a fully convolutional encoder E: .

[0032] in, is the binary mask image preprocessed from the single-view input in step 1, . H and W represent the input binary mask image ( I ), 、 Represents the spatial dimensions (height and width) of the feature volume W after processing by the fully convolutional encoder E. C represents the feature volume W The number of channels, that is, the dimension of the feature vector at each spatial position.

[0033] For 3D query points For each pixel point of the input image, the initial ray direction is obtained according to the back projection of the camera intrinsic parameter matrix, and layered sampling is performed along the ray direction within the preset depth range to obtain a sampling point set, which is then projected to the image plane coordinates , and extract the corresponding features from the feature volume W by bilinear interpolation .

[0034] Then the neural radiation field prediction is performed through the learnable position encoding and the corresponding features extracted:

[0035] in, is the position code, is the viewing direction, is the bulk density, It is an RGB color.

[0036] Preliminary 3D reconstruction is performed based on the volume rendering principle of Neural Radiation Field (NeRF), which is achieved through layered sampling and numerical integration:

[0037] in, is the camera ray; 、 are the near and far depth ranges integrated along the ray, respectively, which are usually set manually to cover the scene content; is the transmittance, The cumulative probability of not being absorbed when it spreads to t; is the volume density of the ray at depth, predicted by the neural radiation field; is the RGB color value of the ray at depth t, predicted by the neural radiance field; For rays The final rendered pixel color is the color of all sampling points The weighted sum of In the integral formula of volume rendering, σ(s) represents the depth of the ray. s The volume density at a point in space is the probability density of light being blocked or scattered, which determines the attenuation of light during propagation.

[0038] Prior knowledge of non-rigid objects includes a Gaussian dynamic deformation field (3DGS) and a parameterized template, which is used to constrain shape plausibility and resolve ambiguity in single-view reconstruction. A spatial deformation representation of the Gaussian sphere is established based on the 3DGS. Local deformations are parameterized through learnable position encodings, and a physics-inspired regularization term is used to constrain the continuity and smoothness of the deformation. The parameterized template utilizes a hybrid multi-model architecture combining SMPL-X, SMAL, and AniCap. It supports 3D modeling of common non-rigid objects such as humans and animals, providing kinematic constraints and biomechanical priors.

[0039] Finally, the rendering target is jointly optimized:

[0040] in, is the target total loss; is the reconstruction loss, which is used to supervise the consistency between the rendered image and the real image; Vertices or implicit surfaces reconstructed for NeRF; Vertices generated for the parameterized template; is the template matching loss, which is used to constrain the geometric consistency between NeRF output and parameterized template; It is a physical constraint loss to ensure that the reconstruction results conform to physical laws; is the weight coefficient, is the template weight, The role of joint optimization is to perform geometric correction on the rendered volume model to ensure the rationality of the motion of non-rigid objects and improve the training stability of the model.

[0041] Step S4: Extract the mesh model from the output volume density using the Marching Cubes algorithm, use the reconstruction algorithm to complete the occlusion and details, improve the global optimization integrity, and use the texture super-resolution technology ESRGAN to enhance the surface details and improve the realism of the model.

[0042] In one implementation, a mesh model is extracted from the volume density output by pixelNeRF using the Marching Cubes algorithm. For areas with occlusion or missing details, a Poisson global optimization algorithm is used to complete the geometry, and a multi-layered Shell-Based modeling strategy is combined to reconstruct fine features such as hair to ensure geometric integrity.

[0043] To further enhance realism, ESRGAN, a generative adversarial network (GAN), was used to super-resolution enhance the mesh surface texture. ESRGAN generates high-frequency details through adversarial training, enabling the reconstructed model to support 4K resolution rendering and significantly improving the visual quality of materials such as skin and hair.

[0044] Step S5: Provide a user interaction interface and perform local optimization on it, follow up the model rendering in real time, use the differentiable physics engine to make reasonable corrections, and generate a drivable three-dimensional virtual animation model.

[0045] In one embodiment, to meet practical application needs, the present invention provides a user interface that supports refinement of the reconstructed model through key point annotation or local deformation editing. User-annotated areas are optimized using a diffusion model, and real-time rendering feedback ensures efficient interaction. Furthermore, a differentiable physics engine, DiffPD, is introduced to perform physical plausibility correction on motion poses, eliminating unnatural deformations or collisions.

[0046] Finally, skeletal drivers generate animation files in FBX format, supporting applications such as avatar drivers and film and television special effects production. The entire process operates in real-time or near-real-time on NVIDIA RTX 30 series or higher-performance graphics cards, achieving both high precision and high efficiency.

[0047] The present invention adopts the technology of combining single-view neural radiation field variant with dynamic deformation modeling to realize three-dimensional reconstruction of non-rigid objects, which has the advantages of high precision, high efficiency, support for interactive editing, and ability to generate driven animation.

[0048] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a non-rigid object three-dimensional reconstruction system based on a single-view neural radiation field, which is used to execute the non-rigid object three-dimensional reconstruction method based on a single-view neural radiation field in the above method embodiment.

[0049] See also Figure 2 The system includes: a feature extraction and segmentation module, which inputs a single RGB image and uses a pre-trained deep learning backbone network to extract features, including the object's texture, local geometry, posture, and global structure; a semantic segmentation network is combined to segment the object contour, separate background interference, and semantically project the discriminative features learned by the classification network into the pixel space to generate a binary mask image dataset; a neural radiance field reconstruction module, which uses a neural radiance field (NeRF) variant (pixelNeRF) to predict 3D volume density and color distribution from a single view image, preliminarily construct the original 3D model, and embed prior knowledge of non-rigid objects, including dynamic deformation fields and parameterized templates; a geometric optimization and texture enhancement module, which uses the Marching Cubes algorithm to extract a mesh model from the volume density output by pixelNeRF, uses a reconstruction algorithm to complete some geometric structures or hair details lost due to occlusion, and combines a global optimization strategy to improve the reconstruction completeness, and enhances the model surface details based on the texture super-resolution technology of the generative adversarial network; a user interaction and output module, which uses a user interaction interface to annotate key points and / or edit local areas, locally optimizes the user-annotated areas through a diffusion model, and provides real-time rendering feedback.

[0050] The embodiment of the present invention provides a non-rigid object 3D reconstruction system based on single view neural radiation field, which is aimed at the single view neural radiation field in the prior art. Figure 3 The 3D reconstruction method has the problems of high shape ambiguity, difficulty in modeling dynamic objects, lack of texture details and lack of interactivity. Figure 2 Several modules in it achieve technical effects of high precision, high efficiency, support for interactive editing, and ability to generate driven animation by combining single-view neural radiation field and dynamic deformation modeling technology.

[0051] like Figure 2As shown, the system described in the embodiment of the present invention first extracts multi-scale features of the input RGB image through a pre-trained ResNet-50 and VisionTransformer hybrid backbone network, and generates an accurate object mask by combining the cascaded Mask R-CNN and U-Net semantic segmentation network; then the extracted 2D features are projected into the 3D space through a fully convolutional encoder to form a feature volume, and the volume density and color value of each 3D query point are predicted using the improved pixelNeRF architecture, in which the Gaussian dynamic deformation field (3DGS) is specially introduced to perform parameterized modeling of local deformation, and the biomechanical prior knowledge provided by parameterized templates such as SMPL-X / SMAL is integrated to constrain the shape rationality; then, the Marching The Cubes algorithm extracts an initial mesh model from the predicted volume density, uses Poisson reconstruction and Shell-Based hierarchical modeling to complete the missing geometric structure due to occlusion, and utilizes ESRGAN for texture super-resolution enhancement. Finally, it provides a user interactive interface to support key point annotation and local deformation editing, and uses the differentiable physics engine DiffPD to perform physical rationality correction on motion postures. Ultimately, it outputs a drivable 3D animation model with high-fidelity geometric details and realistic textures. The entire process achieves efficient interactive editing capabilities while ensuring reconstruction accuracy.

[0052] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0053] like Figure 3As shown in the figure, the optimization method based on dynamic deformation field and parameterized template proposed in the present invention significantly improves the performance of single-view neural radiation field in non-rigid object reconstruction. Due to the lack of shape constraints and dynamic modeling capabilities, the reconstruction mesh model of traditional single-view NeRF has obvious holes and distortions (such as broken human arms or missing animal hair), and the rendered image is blurred and has artifacts; the optimization effect of the present invention is shown on the right. The local deformation is parameterized by Gaussian dynamic deformation field (3DGS), so that the Gaussian sphere can deform reasonably according to physical laws. At the same time, combined with the biomechanical priors provided by parameterized templates such as SMPL-X / SMAL, the rationality of the reconstructed shape is effectively constrained. Finally, the surface texture details are enhanced by ESRGAN super-resolution technology, realizing a complete reconstruction process from the input single-view RGB image to a high-fidelity, drivable three-dimensional model, solving key problems such as high shape ambiguity and difficulty in dynamic modeling in traditional methods.

[0054] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention also provides a non-rigid object three-dimensional reconstruction device based on a single-view neural radiation field, comprising: one or more graphics processors; a storage device for storing one or more program modules; the storage device stores program instructions executed by the graphics processor, and the graphics processor calls the program instructions to execute the non-rigid object three-dimensional reconstruction method based on a single-view neural radiation field.

[0055] These program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0057] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the method for three-dimensional reconstruction of non-rigid objects based on a single-view neural radiation field.

[0058] In summary, the application provides a single-view neural radiance field-based non-rigid object three-dimensional reconstruction method, which belongs to the field of computer vision and three-dimensional reconstruction technology. First, input a single RGB image, extract global features through a pre-trained deep learning backbone network, and generate a mask image of the object in combination with a semantic segmentation network. Second, use the neural radiance field variant pixelNeRF to predict volume density and color distribution, embed a dynamic deformation field and a parameterized template to constrain shape rationality, and solve the shape ambiguity problem in single-view reconstruction. Next, extract a grid model through the Marching Cubes algorithm, use a reconstruction algorithm to complete occlusion and details, and use texture super-resolution technology to enhance surface realism. Finally, provide a user interaction interface for local optimization, and correct the motion posture through a differentiable physics engine to generate a drivable three-dimensional animation model. The application combines single-view neural radiance field and dynamic deformation modeling technology, and has the advantages of high precision, high efficiency, support for interactive editing, and generation of drivable animation.

[0059] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the application.

Claims

1. A method for 3D reconstruction of non-rigid objects based on a single-view neural radiation field, characterized in that: include: Step 1: Input a single RGB image and use a pre-trained deep learning backbone network to extract features, including the object's texture, local geometry, posture, and global structure. Combined with a semantic segmentation network, it segments the object's contours and separates background interference. The discriminative features learned by the classification network are semantically projected into pixel space to generate a binary mask image dataset. Step 2: Using pixelNeRF, a variant of Neural Radiance Field (NeRF), we predict 3D volume density and color distribution from a single-view image to preliminarily construct a primitive 3D model, embedding prior knowledge of non-rigid objects, including dynamic deformation fields and parameterized templates. Step 3: Extract the mesh model from the volume density output by pixelNeRF using the Marching Cubes algorithm. Use a reconstruction algorithm to complete the missing geometric structure or hair details due to occlusion. Combined with a global optimization strategy to improve the reconstruction completeness, the texture super-resolution technology based on the generative adversarial network is used to enhance the surface details of the model. Step 4: Annotate key points and / or edit local areas based on the user interaction interface, locally optimize the user-annotated area through the diffusion model, and provide real-time rendering feedback.

2. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 1, characterized in that: The deep learning backbone network in step 1 adopts a hybrid architecture of ResNet-50 and Vision Transformer, wherein ResNet-50 extracts local features of a single image through its residual connection structure and uses a pre-trained convolutional neural network layer to capture texture details and local geometric features of the object surface; Vision Transformer processes the input image in blocks and establishes long-distance dependencies through a self-attention mechanism to capture the global structure and posture features of the object; the output features of the two networks are spliced ​​and reduced in dimension through a feature fusion module to form a unified feature representation containing multi-scale information.

3. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 2, characterized in that: The classification network outputs the category probability of the object through a fully connected layer and a Softmax function based on the features extracted by the deep learning backbone network, and embeds the category probability as prior knowledge into the semantic segmentation network; the semantic segmentation network takes the fused features output by the deep learning backbone network as input, adopts the cascade structure of Mask R-CNN and U-Net, and outputs a binary mask image dataset with pixel-level accuracy.

4. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 1, characterized in that: The initial construction of the original three-dimensional model in step 2 also includes: The binary mask image dataset of the single-view image is used for feature encoding and projection to extract the volume features of the image pixels in the plane coordinates; Neural radiance field prediction using learnable positional encodings and extracted volumetric features; Based on the prediction results, preliminary three-dimensional reconstruction is performed in combination with the volume rendering principle of the neural radiation field.

5. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 4, characterized in that: The method further comprises: We establish a spatial deformation representation of a Gaussian sphere based on a dynamic deformation field, parameterize the local deformation through learnable position encoding, and use a physics-inspired regularization term to constrain the continuity and smoothness of the deformation. A hybrid multi-model architecture combining SMPL-X, SMAL, and AniCap is used to generate parameterized templates to support 3D modeling of non-rigid objects, providing kinematic constraints and biomechanical priors. The rendered volume model is jointly optimized based on a dynamic deformation field and a parameterized template.

6. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 1, characterized in that: The reconstruction algorithm in step 3 includes Poisson global optimization reconstruction and multi-layer Shell-Based hair modeling. The Poisson global optimization reconstruction is used to complete the occluded part of the object, and the multi-layer Shell-Based hair modeling is used to complete the occluded hair and maintain a natural transition of the hair flow.

7. The method for 3D reconstruction of non-rigid objects based on single-view neural radiation field according to claim 1, characterized in that: The step 4 further comprises: The motion posture is physically corrected through a differentiable physics engine, supporting real-time skeleton-driven animation generation, obtaining FBX animation files that can drive virtual images, and completing sparse perspective 3D reconstruction of non-rigid objects.

8. A 3D reconstruction system for non-rigid objects based on a single-view neural radiation field, characterized in that: include: The feature extraction and segmentation module takes a single RGB image as input and uses a pre-trained deep learning backbone network to extract features, including the object's texture, local geometry, posture, and global structure. It then uses a semantic segmentation network to segment object contours and remove background interference. It then semantically projects the discriminative features learned by the classification network into pixel space to generate a binary mask image dataset. Neural Radiance Field Reconstruction Module, which uses pixelNeRF, a variant of Neural Radiance Field (NeRF), to predict 3D volume density and color distribution from a single-view image, initially construct the original 3D model, and embed prior knowledge of non-rigid objects, including dynamic deformation fields and parameterized templates; The geometry optimization and texture enhancement module uses the Marching Cubes algorithm to extract a mesh model from the volume density output by pixelNeRF. It then uses a reconstruction algorithm to complete some geometric structures or hair details lost due to occlusion. It also combines a global optimization strategy to improve reconstruction integrity and enhances surface details using texture super-resolution technology based on a generative adversarial network. The user interaction and output module is used to mark key points and / or edit local areas based on the user interaction interface, locally optimize the user-marked area through the diffusion model, and provide real-time rendering feedback.

9. A device for 3D reconstruction of non-rigid objects based on a single-view neural radiation field, characterized in that: include: one or more graphics processors; a storage device for storing one or more program modules; The storage device stores program instructions executed by the graphics processor, and the graphics processor calls the program instructions to execute the method for three-dimensional reconstruction of non-rigid objects based on a single-view neural radiation field according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the method for three-dimensional reconstruction of non-rigid objects based on single-view neural radiation fields according to any one of claims 1 to 7.

Images