A NeRF-based hierarchical object modeling and fast reconstruction method

Abstract

This invention discloses a NeRF-based hierarchical object modeling and fast reconstruction method, comprising the following steps: image preprocessing, initially acquiring image and video frames through a camera, and processing the input images including semantic segmentation, depth estimation, normal estimation, and intrinsic / extrinsic parameter calibration to generate corresponding semantic maps, depth maps, normal maps, and camera parameters; spatial partitioning, dividing the scene based on semantic map information and defining bounding boxes; adaptive sampling, performing adaptive sampling by combining the depth map, normal map, and camera parameters to determine the set of sampling points; hierarchical vectorized SDF reconstruction, using these sampling points to perform hierarchical vectorized SDF reconstruction to obtain a hierarchical SDF representation describing the object's shape, converting the hierarchical SDF into a triangular mesh model to complete 3D object modeling and fast reconstruction. This invention provides a NeRF-based hierarchical object modeling and fast reconstruction method that rapidly reconstructs objects in a scene, supporting downstream editing tasks, interactive applications, and more.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a hierarchical object modeling and rapid reconstruction method based on NeRF. Background Technology

[0002] NeRF (Neural Radiation Fields) is a neural network-based 3D scene representation method that renders high-quality images from any viewpoint by learning a continuous 3D radiation field. The core idea of ​​NeRF is to use an MLP to encode the radiation field in 3D space. The input is the 3D coordinate position (x, y, z) and the viewing direction (θ, φ), and the output is the color c and volume density σ at that position. Integration is performed along each ray to calculate the color value of each pixel. NeuS and Vol SDF are two NeRF-based 3D surface reconstruction techniques that introduce SDF (Signed Dimension Function) as an implicit representation of the 3D surface and propose SDF-based volume rendering methods, but each employs different strategies to optimize reconstruction quality and efficiency.

[0003] SDF provides a technical solution for constructing a composite neural implicit representation of objects for high-fidelity 3D reconstruction and object representation. This framework imposes explicit surface constraints by simulating a scene by combining the SDF of individual objects. Semantic information is transformed into a function of the object's SDF, and a unified and compact representation is developed for both the scene and the objects.

[0004] NeRF, as a novel paradigm for synthesis with a fresh perspective, has shown great potential in generating high-quality 3D scenes and objects. However, due to the computational requirements of MLP and large-scale sampling, there are limitations to rapid object-level reconstruction, especially in complex scenes, which restricts its application in many downstream tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a NeRF-based hierarchical object modeling and rapid reconstruction method that can quickly reconstruct each object in a scene, while supporting downstream editing tasks, interactive applications, and other functions.

[0006] The technical solution adopted in the NeRF-based hierarchical object modeling and fast reconstruction method disclosed in this invention is as follows:

[0007] A NeRF-based hierarchical object modeling and fast reconstruction method includes the following steps:

[0008] Image preprocessing initially involves obtaining image and video frames from the camera, and the input images undergo relevant processing including semantic segmentation, depth estimation, normal estimation, and intrinsic and extrinsic parameter calibration, thereby generating corresponding semantic maps, depth maps, normal maps, and camera parameters.

[0009] Spatial partitioning involves dividing the scene and defining bounding boxes based on semantic graph information;

[0010] Adaptive sampling determines the key set of sampling points by combining depth maps, normal maps, and camera parameters.

[0011] Hierarchical vectorized SDF reconstruction utilizes these sampling points to perform hierarchical vectorized SDF reconstruction, obtaining a hierarchical SDF representation describing the object's shape. The MarchingCubes algorithm is then used to convert the hierarchical SDF into a triangular mesh model, thereby completing the final 3D object modeling and rapid reconstruction.

[0012] As a preferred embodiment, the image preprocessing specifically includes the following:

[0013] Obtain the corresponding instance semantic segmentation mask S = {S1, S2, ..., S...} through semantic segmentation. N};

[0014] The corresponding depth map D = {D1, D2, ..., D} is obtained through depth estimation. N};

[0015] The corresponding normal graph N = {N1, N2, ..., N} is obtained through normal estimation. N};

[0016] Colmap calculates the pose P = {P1, P2, ..., P} using the camera's extrinsic parameters. N};

[0017] The intrinsic parameter of the calibrated camera is K;

[0018] Where: N is the number of images.

[0019] As a preferred approach, in the spatial partitioning stage, the entire space is divided into multiple independent object regions using the semantic segmentation map S and the depth map D, as follows:

[0020] Extract the pixel set of each object in the semantic segmentation image Where: i represents a certain pixel, and n represents the number of pixels;

[0021] Extract the depth value d at the corresponding pixel position. i ;

[0022] Construct and align the 3D point cloud of the object;

[0023] Using the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|T], the pixel coordinates are transformed to the world coordinate system.

[0024] P world =R·K -1 ·(u i ,v i ,1)·d i +T

[0025] Where: K is the camera's intrinsic parameter matrix, consisting of the focal length and principal point coordinates; [R|T] is the camera's extrinsic parameter matrix, where R represents the rotation matrix and T represents the translation vector; (u i ,v i ,1) represents the homogeneous coordinates of the pixel, d i This is the depth value;

[0026] Obtaining 3D bounding boxes: Calculate the 3D bounding box B for each class of point cloud using library functions such as Open3D. j =

[0027] [C,W,H,D], where: C represents the center of the 3D bounding box; W,H,D represent the width, height, and depth of the bounding box, respectively.

[0028] As a preferred approach, the adaptive sampling strategy based on curvature and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows:

[0029] Curvature estimation is performed by calculating the first derivatives of the normal vector in the u and v directions using the finite difference method. and Construct the Weingarten map and calculate the principal curvatures k1 and k2 using the eigenvalues ​​of the Weingarten map;

[0030] Calculate the curvature: Gaussian curvature K is the product of the two principal curvatures, and mean curvature H is the algorithmic average of the two principal curvatures.

[0031] Define a sampling density function ρ(x) based on the curvature K(x) of the object's surface and the maximum depth value D. max The decision is that sampling points are generated at higher densities in areas with high curvature or far from the viewpoint;

[0032] Sampling point generation: For each object, initial uniform sampling is first performed to generate uniformly distributed sampling points, resulting in a sampling point set S. initial Based on the initial sampling, the sampling points are adjusted in conjunction with the sampling density function ρ(x).

[0033] As a preferred embodiment, the specific steps of the hierarchical vectorized SDF reconstruction are as follows:

[0034] Semantic class sampling points: Obtain the set S of sampling points corresponding to each semantic class in the scene. c and the number of sampling points N c ,

[0035] S c ={x i ∈R 3 |i=1,2,...,N c}

[0036] Where: x i Represents the three-dimensional coordinates of the sampling point;

[0037] The NeRF hierarchical processing divides NeRF into n levels, each corresponding to a different resolution. The NeRF hierarchical formula is as follows:

[0038]

[0039] Among them: NeRF l Represents the NeRF of level l;

[0040] Vectorized training is performed using an MLP implicit function f for each level of NeRF. θ Calculate the SDF and color values ​​of the sampling points, and perform vector training using vmap, as shown in the following formula:

[0041]

[0042] Y = vmap(f θ (X)

[0043] Where: y i For sampling point x i The SDF values ​​and color values, X = [x1, x2, ..., x m [y1, y2, ..., y] represents the sample point matrix of the batch input, where Y = [y1, y2, ..., y]. c ] = [SDF1,SDF2,...,SDF c C1, C2, ..., C c [This refers to the batch output of SDF and color values;]

[0044] SDF to Mesh: Use the Marching Cubes algorithm to analyze the zero-crossing points of the SDF to generate triangular meshes of isosurfaces.

[0045] The beneficial effects of the NeRF-based hierarchical object modeling and fast reconstruction method disclosed in this invention are as follows: By improving reconstruction accuracy, accelerating training speed, optimizing sampling efficiency, and enhancing the ability to handle complex scenes, it significantly improves the practical performance and applicability of 3D scene reconstruction. Employing an object-combined implicit 3D representation method, it can not only capture the 3D shape and appearance of the entire scene but also independently represent individual objects within the scene. This combined representation method enhances the understanding and processing capabilities of complex scenes. Based on an adaptive sampling strategy of curvature and depth estimation, it dynamically adjusts the sampling density to ensure sufficient sampling of key feature regions. This not only reduces invalid sampling but also improves reconstruction efficiency. By using a small MLP (Multilayer Perceptron) to represent each object separately, it can more accurately capture the geometric details and texture information of individual objects. Attached Figure Description

[0046] Figure 1 This is a framework diagram of a NeRF-based hierarchical object modeling and rapid reconstruction method of the present invention. Detailed Implementation

[0047] The present invention will be further described and illustrated below with reference to specific embodiments and the accompanying drawings:

[0048] Please refer to Figure 1 A NeRF-based hierarchical object modeling and fast reconstruction method includes the following steps:

[0049] Image preprocessing

[0050] Initially, image and video frames are acquired through cameras such as monocular, binocular, and RGBD. The input images undergo relevant processing, including semantic segmentation, depth estimation, normal estimation, and intrinsic and extrinsic parameter calibration, thereby generating corresponding semantic maps, depth maps, normal maps, and camera parameters.

[0051] Image preprocessing specifically includes the following:

[0052] Obtain the corresponding instance semantic segmentation mask S = {S1, S2, ..., S...} through semantic segmentation. N};

[0053] The corresponding depth map D = {D1, D2, ..., D} is obtained through depth estimation. N};

[0054] The corresponding normal graph N = {N1, N2, ..., N} is obtained through normal estimation. N};

[0055] Colmap calculates the pose P = {P1, P2, ..., P} using the camera's extrinsic parameters. N};

[0056] The intrinsic parameter of the calibrated camera is K;

[0057] Where: N is the number of images.

[0058] Spatial partitioning

[0059] The scene is divided and bounding boxes are defined based on semantic graph information.

[0060] In the spatial partitioning stage, the entire space is divided into multiple independent object regions using the semantic segmentation map S and the depth map D. The steps are as follows:

[0061] Extract the pixel set of each object in the semantic segmentation image Where: i represents a certain pixel, and n represents the number of pixels;

[0062] Extract the depth value d at the corresponding pixel position. i ;

[0063] Construct and align the 3D point cloud of the object;

[0064] Using the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|T], the pixel coordinates are transformed to the world coordinate system.

[0065] P world =R·K -1 ·(u i ,v i ,1)·d i +T

[0066] Where: K is the camera's intrinsic parameter matrix, consisting of the focal length and principal point coordinates; [R|T] is the camera's extrinsic parameter matrix, where R represents the rotation matrix and T represents the translation vector; (u i ,v i ,1) represents the homogeneous coordinates of the pixel, d i This is the depth value;

[0067] Obtaining 3D bounding boxes: Calculate the 3D bounding box B for each class of point cloud using library functions such as Open3D. j =

[0068] [C,W,H,D], where: C represents the center of the 3D bounding box; W,H,D represent the width, height, and depth of the bounding box, respectively.

[0069] Adaptive sampling

[0070] By combining depth maps, normal maps, and camera parameters to perform adaptive sampling, the key set of sampling points is determined.

[0071] The adaptive sampling strategy based on curvature and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows:

[0072] Curvature estimation is performed by calculating the first derivatives of the normal vector in the u and v directions using the finite difference method. and

[0073]

[0074] Where Δu and Δv are small step sizes.

[0075] Construct the Wei ngarten map,

[0076]

[0077] Where: r u and r v It is the partial derivative of the surface in the u and v directions.

[0078] The principal curvatures k1 and k2 are calculated using the eigenvalues ​​of the Weingarten map, and det(W-kI) = 0.

[0079] Where: det is the determinant, and I is the identity matrix;

[0080] The curvature is calculated as follows: the Gaussian curvature K is the product of the two principal curvatures, K = K1·K2.

[0081] The mean curvature H is the algorithmic average of the two principal curvatures.

[0082] Define a sampling density function ρ(x) based on the curvature K(x) of the object's surface and the maximum depth value D. max The decision is that sampling points are generated at higher densities in areas with high curvature or far from the viewpoint.

[0083] The formula is defined as follows:

[0084]

[0085] Where: α is the curvature weighting coefficient, β is the curvature depth weighting coefficient, K(x) is the curvature weighting coefficient, and D(x) is the curvature weighting coefficient. max The maximum depth value of the category;

[0086] Sampling point generation: For each object, initial uniform sampling is first performed to generate uniformly distributed sampling points, resulting in a sampling point set S. initial ,

[0087] S initial ={x1,x2,...,x N}

[0088] Where: x i S represents sampling points uniformly distributed on the surface of the object.initial This is the original set of sampling points;

[0089] Based on the initial sampling, the sampling points are adjusted using the sampling density function ρ(x), at point x i

[0090] Wherein: S refined Let θ be the adjusted set of sampling points, and let θ be the threshold value controlled by the density function.

[0091] Hierarchical Vectorized SDF Reconstruction

[0092] These sampling points are used to perform hierarchical vectorized SDF reconstruction, resulting in a hierarchical SDF representation describing the shape of the object. The MarchingCubes algorithm is then used to convert the hierarchical SDF into a triangular mesh model, thereby completing the final 3D object modeling and rapid reconstruction.

[0093] The specific steps for hierarchical vectorized SDF reconstruction are as follows:

[0094] Semantic class sampling points: Obtain the set S of sampling points corresponding to each semantic class in the scene. c and the number of sampling points N c ,

[0095] S c ={x i ∈R 3 |i=1,2,...,N c}

[0096] Where: x i Represents the three-dimensional coordinates of the sampling point;

[0097] The NeRF hierarchical processing divides NeRF into n levels, each corresponding to a different resolution. The NeRF hierarchical formula is as follows:

[0098] Among them: NeRF l Represents the NeRF of level l;

[0099] Vectorized training is performed using an MLP implicit function f for each level of NeRF. θ Calculate the SDF and color values ​​of the sampling points, and perform vector training using vmap, as shown in the following formula:

[0100]

[0101] Y = vmap(f θ (X)

[0102] Where: y i For sampling point x iThe SDF values ​​and color values, X = [x1, x2, ..., x m [y1, y2, ..., y] represents the sample point matrix of the batch input, where Y = [y1, y2, ..., y]. c ] = [SDF1,SDF2,...,SDF c C1, C2, ..., C c [This refers to the batch output of SDF and color values;]

[0103] SDF to Mesh: Use the Marching Cubes algorithm to analyze the zero-crossing points of the SDF to generate triangular meshes of isosurfaces.

[0104] This invention provides a NeRF-based hierarchical object modeling and rapid reconstruction method, which significantly improves the practical performance and applicability of 3D scene reconstruction by increasing reconstruction accuracy, accelerating training speed, optimizing sampling efficiency, and enhancing the ability to handle complex scenes.

[0105] An implicit 3D representation method using object composition can not only capture the 3D shape and appearance of the entire scene, but also represent individual objects within the scene independently. This compositional representation method enhances the ability to understand and process complex scenes.

[0106] An adaptive sampling strategy based on curvature and depth estimation dynamically adjusts the sampling density to ensure sufficient sampling of key feature regions. This not only reduces invalid sampling but also improves reconstruction efficiency.

[0107] By using a small MLP (Multilayer Perceptron) to represent each object separately, it is possible to capture the geometric details and texture information of individual objects more accurately.

[0108] These improvements have enabled the technology to demonstrate outstanding performance in a variety of practical applications, including virtual reality, augmented reality, architectural design, cultural heritage preservation, and autonomous driving. For example, in the AR&VR field, by providing higher-quality and smoother virtual environments, it attracts more users, increasing platform subscription fees and advertising revenue. While the direct economic benefits of cultural heritage preservation are limited, digital displays can attract more tourists, indirectly increasing tourism revenue.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A NeRF-based hierarchical object modeling and fast reconstruction method, characterized in that, Includes the following steps: Image preprocessing initially involves obtaining image and video frames from the camera, and the input images undergo relevant processing including semantic segmentation, depth estimation, normal estimation, and intrinsic and extrinsic parameter calibration, thereby generating corresponding semantic maps, depth maps, normal maps, and camera parameters. Spatial partitioning involves dividing the scene and defining bounding boxes based on semantic graph information; Adaptive sampling determines the key set of sampling points by combining depth maps, normal maps, and camera parameters. Hierarchical vectorized SDF reconstruction: These sampling points are used to perform hierarchical vectorized SDF reconstruction to obtain a hierarchical SDF representation describing the shape of the object. The MarchingCubes algorithm is used to convert the hierarchical SDF into a triangular mesh model, thereby completing the final 3D object modeling and fast reconstruction. The adaptive sampling strategy based on curvature and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows: Curvature estimation is performed by calculating the first derivatives of the normal vector in the u and v directions using the finite difference method. Construct the Weingarten map and calculate the principal curvature using the eigenvalues ​​of the Weingarten map. and ; Calculate the curvature: Gaussian curvature K is the product of the two principal curvatures, and mean curvature H is the algorithmic average of the two principal curvatures. Define the sampling density function. The curvature of the object's surface and maximum depth value The decision is that sampling points are generated at higher densities in areas with high curvature or far from the viewpoint; Sampling point generation: For each object, initial uniform sampling is first performed to generate uniformly distributed sampling points, thus obtaining a set of sampling points. Based on the initial sampling, combined with the sampling density function Adjust the sampling points.

2. The NeRF-based hierarchical object modeling and fast reconstruction method as described in claim 1, characterized in that, The image preprocessing specifically includes the following: Obtain the corresponding instance semantic segmentation mask through semantic segmentation. ; The corresponding depth map is obtained through depth estimation. ; The corresponding normal map is obtained through normal estimation. ; Colmap calculates the pose using camera extrinsic parameters. ; The intrinsic parameters for calibrating the camera are ; Where: N is the number of images.

3. The NeRF-based hierarchical object modeling and fast reconstruction method as described in claim 1, characterized in that, In the spatial partitioning stage, the entire space is divided into multiple independent object regions using the semantic segmentation map S and the depth map D. The steps are as follows: Extract the pixel set of each object in the semantic segmentation image ,in: Let n be a certain pixel, and n be the number of pixels. Extract the depth value at the corresponding pixel position. ; Construct and align the 3D point cloud of the object; Using the camera's intrinsic parameter matrix and extrinsic parameter matrix Transform pixel coordinates to world coordinates. Where: K is the camera's intrinsic parameter matrix, consisting of the focal length and the coordinates of the principal point. Let R be the camera extrinsic matrix, R be the rotation matrix, and T be the translation vector. Let be the homogeneous coordinates of the pixel. This is the depth value; Obtaining 3D bounding boxes: Calculate the 3D bounding boxes for each type of point cloud using library functions such as Open3D. Where: C represents the center of the 3D bounding box; W, H, and D represent the width, height, and depth of the bounding box, respectively.

4. The NeRF-based hierarchical object modeling and fast reconstruction method as described in claim 1, characterized in that, The specific steps of the hierarchical vectorized SDF reconstruction are as follows: Semantic class sampling points: Obtain the set of sampling points corresponding to each semantic class in the scene. and number of sampling points , in: Represents the three-dimensional coordinates of the sampling point; The NeRF hierarchical processing divides NeRF into n levels, each corresponding to a different resolution. The NeRF hierarchical formula is as follows: in: Represents the NeRF of level l; Vectorized training, using implicit functions of MLP for each level of NeRF. Calculate the SDF and color values ​​of the sampling points, and perform vector training using vmap, as shown in the following formula: in: Sampling points SDF value and color value, For the batch input sampling point matrix, This is for batch output of SDF and color values; SDF to Mesh: Use the Marching Cubes algorithm to analyze the zero-crossing points of the SDF to generate triangular meshes of isosurfaces.

Images