Hierarchical object modeling and rapid reconstruction method based on NeRF

Abstract

The invention discloses a hierarchical object modeling and rapid reconstruction method based on NeRF, and the method comprises the following steps: image preprocessing: initially obtaining an image video frame through a camera, carrying out the related processing of an input image, including semantic segmentation, depth estimation, normal estimation and internal and external parameter calibration, and generating a corresponding semantic graph, a depth graph, a normal graph and camera parameters; space partitioning: partitioning the scene according to the semantic graph information and defining a bounding box; adaptive sampling: performing adaptive sampling by combining the depth map, the normal map and the camera parameters, and determining a sampling point set; and performing hierarchical vectorization SDF reconstruction, performing hierarchical vectorization SDF reconstruction by using the sampling points to obtain hierarchical SDF representation describing the shape of the object, converting the hierarchical SDF into a triangular mesh model, and completing three-dimensional object modeling and rapid reconstruction. The invention provides a hierarchical object modeling and rapid reconstruction method based on NeRF, objects in a scene are rapidly reconstructed, and downstream editing tasks, interaction and other applications are supported.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D modeling, and in particular to a NeRF-based hierarchical object modeling and rapid reconstruction method. Background Art

[0002] NeRF (Neura l Radiation Fields) is a 3D scene representation method based on neural networks. It renders high-quality images from any perspective by learning a continuous 3D radiation field. The core idea of ​​NeRF is to use an MLP to encode the radiation field in 3D space. The input is the 3D coordinate position (x, y, z) and the viewing direction (θ, φ), and the output is the color c and volume density σ at that position. Integrate along each ray to calculate the color value of each pixel. NeuS and Vol SDF are two 3D surface reconstruction technologies based on NeRF. They introduce SDF (Signed Difference Function) as an implicit expression of the three-dimensional surface and propose a volume rendering method based on SDF, but each adopts different strategies to optimize the reconstruction quality and efficiency.

[0003] SDF provides a technical solution for building object-combined neural implicit representations with high-fidelity 3D reconstruction and object representation. The framework simulates scenes by combining the SDFs of individual objects to impose explicit surface constraints. It transforms semantic information into a function of the object SDF and develops a unified and compact representation for scenes and objects.

[0004] As a new paradigm for new perspective synthesis, NeRF has shown great potential in generating high-quality 3D scenes and objects. However, due to the computational requirements of MLP and large-scale sampling, fast object-level reconstruction has limitations, especially in complex scenes, which restricts many downstream task applications. Summary of the invention

[0005] The purpose of the present invention is to provide a NeRF-based hierarchical object modeling and rapid reconstruction method, which can quickly reconstruct each object in the scene and support downstream editing tasks, interactive applications, etc.

[0006] The technical solution adopted by the NeRF-based hierarchical object modeling and rapid reconstruction method disclosed in the present invention is:

[0007] A NeRF-based hierarchical object modeling and rapid reconstruction method comprises the following steps:

[0008] Image preprocessing: initially, the image video frame is obtained through the camera, and the input image will undergo relevant processing including semantic segmentation, depth estimation, normal estimation, and internal and external parameter calibration to generate the corresponding semantic map, depth map, normal map, and camera parameters;

[0009] Spatial partitioning: divide the scene and define the bounding box based on the semantic graph information;

[0010] Adaptive sampling, which performs adaptive sampling by combining the depth map, normal map and camera parameters to determine the key set of sampling points;

[0011] Hierarchical vectorized SDF reconstruction uses these sampling points to perform hierarchical vectorized SDF reconstruction to obtain a hierarchical SDF representation that describes the shape of the object. The MarchingCubes algorithm is used to convert the hierarchical SDF into a triangular mesh model to complete the final three-dimensional object modeling and rapid reconstruction.

[0012] As a preferred solution, the image preprocessing specifically includes the following:

[0013] Through semantic segmentation, we obtain the corresponding instance semantic segmentation mask S = {S1, S2, ..., S N};

[0014] The corresponding depth map D = {D1, D2, ..., D N};

[0015] The corresponding normal map N = {N1, N2, ..., N N};

[0016] Colmap calculates the camera external parameters to obtain the pose P = {P1, P2, ..., P N};

[0017] The intrinsic parameter of the calibrated camera is K;

[0018] Where: N is the number of pictures.

[0019] As a preferred solution, in the space partitioning stage, the semantic segmentation map S and the depth map D are used to divide the entire space into multiple independent object regions. The steps are as follows:

[0020] Extract the pixel set of each object in the semantic segmentation image Where: i is a certain pixel, n is the number of pixels;

[0021] Extract the depth value d corresponding to the pixel position i ;

[0022] Construct a 3D point cloud of the object and align it;

[0023] Use the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|T] to transform the pixel coordinates to the world coordinate system.

[0024] P world =R·K -1 ·(u i ,v i ,1)·d i +T

[0025] Where: K is the camera's intrinsic parameter matrix, which is composed of the focal length and the principal point coordinates, [R|T] is the camera's extrinsic parameter matrix, R represents the rotation matrix, and T represents the translation vector; (u i ,v i ,1) is the homogeneous coordinate of the pixel, d i is the depth value;

[0026] Get 3D bounding box: Use Open3D and other library functions to calculate the 3D bounding box B of each class point cloud j =

[0027] [C, W, H, D], where C represents the center of the 3D bounding box; W, H, D represent the width, height, and depth of the bounding box, respectively.

[0028] As a preferred solution, the adaptive sampling strategy based on curvature estimation and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows:

[0029] Curvature estimation, calculating the first-order derivative of the normal vector in the u and v directions by difference method and Construct the Weingarten map and calculate the principal curvatures k1 and k2 through the eigenvalues ​​of the Weingarten map;

[0030] Calculate the curvature. The Gaussian curvature K is the product of the two principal curvatures, and the mean curvature H is the algorithmic average of the two principal curvatures.

[0031] Define the sampling density function, the sampling density function ρ(x) is composed of the curvature K(x) of the object surface and the maximum depth value D max It is decided that the generation of sampling points has a higher density in areas with high curvature or farther from the viewpoint;

[0032] Sampling point generation, for each object, first perform initial uniform sampling to generate uniformly distributed sampling points and obtain the sampling point set S initial , based on the initial sampling, the sampling points are adjusted in combination with the sampling density function ρ(x).

[0033] As a preferred solution, the specific steps of the hierarchical vectorized SDF reconstruction are as follows:

[0034] Semantic class sampling points, obtain the sampling point set S corresponding to each semantic class in the scene c and the number of sampling points N c ,

[0035] S c ={x i ∈R 3 |i=1,2,...,N c}

[0036] Where: x i Represents the three-dimensional coordinates of the sampling point;

[0037] NeRF classification processing divides NeRF into n levels, each level corresponds to a different resolution. The NeRF classification formula is as follows:

[0038]

[0039] Among them: NeRF l represents the NeRF of level l;

[0040] Vectorized training, for each level of NeRF, uses the MLP implicit function f θ Calculate the SDF value and color value of the sampling point and perform vector training through vmap. The formula is as follows:

[0041]

[0042] Y=vmap(f θ )(X)

[0043] Where:y i is the sampling point x i SDF value and color value, X=[x1,x2,...,x m ] is the sampling point matrix of batch input, Y=[y1,y2,...,y c ]=[SDF1,SDF2,...,SDF c ,C1,C2,...,C c ] is the SDF and color value of batch output;

[0044] Convert SDF to Mesh, and use the Marching Cubes algorithm to analyze the zero crossing points of SDF to generate a triangular mesh of the isosurface.

[0045] The beneficial effects of a NeRF-based hierarchical object modeling and fast reconstruction method disclosed in the present invention are: by improving reconstruction accuracy, accelerating training speed, optimizing sampling efficiency and enhancing complex scene processing capabilities, the actual performance and applicability of 3D scene reconstruction are significantly improved, and an object-combined implicit 3D representation method is adopted, which can not only capture the 3D shape and appearance of the entire scene, but also independently represent each object in the scene. This combined representation method enhances the understanding and processing capabilities of complex scenes, and dynamically adjusts the sampling density based on an adaptive sampling strategy based on curvature estimation and depth estimation to ensure that key feature areas are sufficiently sampled. This not only reduces invalid sampling, but also improves reconstruction efficiency. By using a small MLP (multi-layer perceptron) for each object separately for representation, the geometric details and texture information of a single object can be captured more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a framework diagram of a NeRF-based hierarchical object modeling and rapid reconstruction method of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described and illustrated below in conjunction with specific embodiments and accompanying drawings:

[0048] Please refer to Figure 1 , a NeRF-based hierarchical object modeling and fast reconstruction method, comprising the following steps:

[0049] Image preprocessing

[0050] Initially, the image video frames are obtained through cameras such as monocular, binocular, RGBD, etc., and the input image will undergo relevant processing including: semantic segmentation, depth estimation, normal estimation, and internal and external parameter calibration to generate the corresponding semantic map, depth map, normal map and camera parameters.

[0051] Image preprocessing specifically includes the following:

[0052] Through semantic segmentation, we obtain the corresponding instance semantic segmentation mask S = {S1, S2, ..., S N};

[0053] The corresponding depth map D = {D1, D2, ..., D N};

[0054] The corresponding normal map N = {N1, N2, ..., N N};

[0055] Colmap calculates the camera external parameters to obtain the pose P = {P1, P2, ..., P N};

[0056] The intrinsic parameter of the calibrated camera is K;

[0057] Where: N is the number of pictures.

[0058] Space partition

[0059] The scene is divided and the bounding box is defined based on the semantic graph information.

[0060] In the space partitioning stage, the semantic segmentation map S and the depth map D are used to divide the entire space into multiple independent object regions. The steps are as follows:

[0061] Extract the pixel set of each object in the semantic segmentation image Where: i is a certain pixel, n is the number of pixels;

[0062] Extract the depth value d corresponding to the pixel position i ;

[0063] Construct a 3D point cloud of the object and align it;

[0064] Use the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|T] to transform the pixel coordinates to the world coordinate system.

[0065] P world =R·K -1 ·(u i ,v i ,1)·d i +T

[0066] Where: K is the camera's intrinsic parameter matrix, which is composed of the focal length and the principal point coordinates, [R|T] is the camera's extrinsic parameter matrix, R represents the rotation matrix, and T represents the translation vector; (u i ,v i ,1) is the homogeneous coordinate of the pixel, d i is the depth value;

[0067] Get 3D bounding box: Use Open3D and other library functions to calculate the 3D bounding box B of each class point cloud j =

[0068] [C, W, H, D], where C represents the center of the 3D bounding box; W, H, D represent the width, height, and depth of the bounding box, respectively.

[0069] Adaptive Sampling

[0070] Adaptive sampling is performed by combining the depth map, normal map and camera parameters to determine the key set of sampling points.

[0071] The adaptive sampling strategy based on curvature estimation and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows:

[0072] Curvature estimation, calculating the first-order derivative of the normal vector in the u and v directions by difference method and

[0073]

[0074] Where: Δu and Δv are small step sizes.

[0075] Constructing the Wei ngarten map,

[0076]

[0077] Where: r u and r v are the partial derivatives of the surface in the u and v directions.

[0078] The principal curvatures k1 and k2 are calculated by the eigenvalues ​​of the Weingarten map, det(W-kI)=0

[0079] Where: det is the determinant, I is the unit matrix;

[0080] Calculate the curvature. Gaussian curvature K is the product of two principal curvatures K = K1·K2.

[0081] The mean curvature H is the algorithmic average of the two principal curvatures

[0082] Define the sampling density function, the sampling density function ρ(x) is composed of the curvature K(x) of the object surface and the maximum depth value D max The sampling points are generated with higher density in areas with high curvature or farther from the viewpoint.

[0083] The formula is defined as follows:

[0084]

[0085] Where: α is the curvature weight coefficient, β is the curvature depth weight coefficient, K(x) is the curvature weight coefficient, D(x) is the curvature weight coefficient, D max is the maximum depth value of the category;

[0086] Sampling point generation, for each object, first perform initial uniform sampling to generate uniformly distributed sampling points and obtain the sampling point set S initial ,

[0087] S initial ={x1,x2,...,x N}

[0088] Where: x i are sampling points evenly distributed on the surface of the object, Sinitial is the original set of sampling points;

[0089] Based on the initial sampling, the sampling points are adjusted in combination with the sampling density function ρ(x). i

[0090]

[0091] Where: S refined is the adjusted sampling point set, and θ is the density function control threshold.

[0092] Hierarchical vectorized SDF reconstruction

[0093] These sampling points are used to perform hierarchical vectorized SDF reconstruction to obtain a hierarchical SDF representation that describes the object shape. The MarchingCubes algorithm is used to convert the hierarchical SDF into a triangular mesh model to complete the final three-dimensional object modeling and rapid reconstruction.

[0094] The specific steps of hierarchical vectorized SDF reconstruction are as follows:

[0095] Semantic class sampling points, obtain the sampling point set S corresponding to each semantic class in the scene c and the number of sampling points N c ,

[0096] S c ={x i ∈R 3 |i=1,2,...,N c}

[0097] Where: x i Represents the three-dimensional coordinates of the sampling point;

[0098] NeRF's hierarchical processing divides NeRF into n levels, each level corresponds to a different resolution. The NeRF grading formula is as follows:

[0099] Among them: NeRF l represents the NeRF of level l;

[0100] Vectorized training, for each level of NeRF, uses the MLP implicit function f θ Calculate the SDF value and color value of the sampling point and perform vector training through vmap. The formula is as follows:

[0101]

[0102] Y=vmap(f θ )(X)

[0103] Where:yi is the sampling point x i SDF value and color value, X=[x1,x2,...,x m ] is the sampling point matrix of batch input, Y=[y1,y2,...,y c ]=[SDF1,SDF2,...,SDF c ,C1,C2,...,C c ] is the SDF and color value of batch output;

[0104] Convert SDF to Mesh, and use the Marching Cubes algorithm to analyze the zero crossing points of SDF to generate a triangular mesh of the isosurface.

[0105] The present invention provides a NeRF-based hierarchical object modeling and rapid reconstruction method, which significantly improves the actual performance and applicability of 3D scene reconstruction by improving reconstruction accuracy, accelerating training speed, optimizing sampling efficiency and enhancing complex scene processing capabilities.

[0106] The implicit 3D representation method using object combination can not only capture the 3D shape and appearance of the entire scene, but also independently represent each object in the scene. This combined representation method enhances the ability to understand and process complex scenes.

[0107] Based on the adaptive sampling strategy of curvature estimation and depth estimation, the sampling density is dynamically adjusted to ensure that key feature areas are adequately sampled. This not only reduces invalid sampling, but also improves reconstruction efficiency.

[0108] By using a small MLP (multi-layer perceptron) to represent each object separately, the geometric details and texture information of a single object can be captured more accurately.

[0109] These improvements have enabled the technology to demonstrate excellent results in a variety of practical application scenarios, including virtual reality, augmented reality, architectural design, cultural heritage protection, and autonomous driving. For example, in the AR&VR field, by providing a higher quality and smoother virtual environment, more users can be attracted, and the subscription fees and advertising revenue of the platform can be increased. Although the direct economic benefits of cultural heritage protection are limited, more tourists can be attracted through digital display, which can indirectly increase tourism revenue.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A NeRF-based hierarchical object modeling and rapid reconstruction method, characterized in that: The following steps are involved: Image preprocessing: initially, the image video frame is obtained through the camera, and the input image will undergo relevant processing including semantic segmentation, depth estimation, normal estimation, and internal and external parameter calibration to generate the corresponding semantic map, depth map, normal map, and camera parameters; Spatial partitioning: divide the scene and define the bounding box based on the semantic graph information; Adaptive sampling, which performs adaptive sampling by combining the depth map, normal map and camera parameters to determine the key set of sampling points; Hierarchical vectorized SDF reconstruction uses these sampling points to perform hierarchical vectorized SDF reconstruction to obtain a hierarchical SDF representation that describes the shape of the object. The MarchingCubes algorithm is used to convert the hierarchical SDF into a triangular mesh model to complete the final three-dimensional object modeling and rapid reconstruction.

2. The NeRF-based hierarchical object modeling and rapid reconstruction method according to claim 1, characterized in that: The image preprocessing specifically includes the following: Through semantic segmentation, we obtain the corresponding instance semantic segmentation mask S = {S1, S2, ..., S N }; The corresponding depth map D = {D1, D2, ..., D N }; The corresponding normal map N = {N1, N2, ..., N N }; Colmap calculates the camera external parameters to obtain the pose P = {P1, P2, ..., P N }; The intrinsic parameter of the calibrated camera is K; Where: N is the number of pictures.

3. The NeRF-based hierarchical object modeling and rapid reconstruction method according to claim 1, characterized in that: In the space partitioning stage, the semantic segmentation map S and the depth map D are used to divide the entire space into multiple independent object regions. The steps are as follows: Extract the pixel set of each object in the semantic segmentation image Where: i is a certain pixel, n is the number of pixels; Extract the depth value d corresponding to the pixel position i ; Construct a 3D point cloud of the object and align it; Use the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|T] to transform the pixel coordinates to the world coordinate system. P world =R·K -1 ·(u i ,v i ,1)·d i +T Where: K is the camera's intrinsic parameter matrix, which is composed of the focal length and the principal point coordinates, [R|T] is the camera's extrinsic parameter matrix, R represents the rotation matrix, and T represents the translation vector; (u i ,v i ,1) is the homogeneous coordinate of the pixel, d i is the depth value; Get 3D bounding box: Use Open3D and other library functions to calculate the 3D bounding box B of each class point cloud j =[C,W,H,D], where: C represents the center of the 3D bounding box; W, H, D represent the width, height, and depth of the bounding box respectively.

4. The NeRF-based hierarchical object modeling and rapid reconstruction method according to claim 1, characterized in that: The adaptive sampling strategy based on curvature estimation and depth estimation aims to dynamically adjust the sampling density. The specific steps are as follows: Curvature estimation, calculating the first-order derivative of the normal vector in the u and v directions by difference method and Construct the Weingarten map and calculate the principal curvatures k1 and k2 through the eigenvalues ​​of the Weingarten map; Calculate the curvature. The Gaussian curvature K is the product of the two principal curvatures, and the mean curvature H is the algorithmic average of the two principal curvatures. Define the sampling density function, the sampling density function ρ(x) is composed of the curvature K(x) of the object surface and the maximum depth value D max It is decided that the generation of sampling points has a higher density in areas with high curvature or farther from the viewpoint; Sampling point generation, for each object, first perform initial uniform sampling to generate uniformly distributed sampling points and obtain the sampling point set S initial , based on the initial sampling, the sampling points are adjusted in combination with the sampling density function ρ(x).

5. The NeRF-based hierarchical object modeling and rapid reconstruction method according to claim 1, characterized in that: The specific steps of hierarchical vectorized SDF reconstruction are as follows: Semantic class sampling points, obtain the sampling point set S corresponding to each semantic class in the scene c and the number of sampling points N c , S c ={x i ∈R 3 ∣i=1,2,...,N c } Where: x i Represents the three-dimensional coordinates of the sampling point; NeRF classification processing divides NeRF into n levels, each level corresponds to a different resolution. The NeRF classification formula is as follows: Among them: NeRF l represents the NeRF of level l; Vectorized training, for each level of NeRF, uses the MLP implicit function f θ Calculate the SDF value and color value of the sampling point and perform vector training through vmap. The formula is as follows: Y=vmap(f θ )(X) Where:y i is the sampling point x i SDF value and color value, X=[x1,x2,...,x m ] is the sampling point matrix of batch input, Y=[y1,y2,...,y c ]=[SDF1,SDF2,...,SDF c ,C1,C2,...,C c ] is the SDF and color value of batch output; Convert SDF to Mesh, and use the Marching Cubes algorithm to analyze the zero crossing points of SDF to generate a triangular mesh of the isosurface.

Images