A NeRF-based 3D object model reconstruction method based on semantic segmentation

Abstract

The present invention discloses a method for reconstructing a three-dimensional target object model based on NeRF with semantic segmentation, which uses neural radiation field (NeRF) to reconstruct a 3D car model. Compared with traditional manual modeling, physical modeling, and modeling algorithms based on laser point clouds or images, the efficiency of three-dimensional car reconstruction is greatly improved. Compared with traditional NeRF, the three-dimensional car reconstruction method based on NeRF with semantic segmentation combines MaskR-CNN semantic segmentation and multi-level feature grid construction and matching multi-level MLP sensing deviation to improve the accuracy of three-dimensional car modeling. At the same time, the multi-level feature grid that only focuses on the modeling target is compared with the original image feature grid, and the amount of processed data is greatly reduced, which improves the training and reconstruction efficiency of NeRF in 3D car models.

Description

Technical Field

[0001] The present invention relates to image information of a static target object reconstructed from a 3D automobile model, including RGB-D image data with depth information. Background Art

[0002] 3D car modeling is an important application direction in many fields such as automobile design and manufacturing, autonomous driving, etc. Its high-precision 3D reconstruction has always been the focus of research in industry and academia. In the past, car modeling was mainly based on hand-drawn sketches and physical models, which took a lot of time and resources. With the development of computer image processing technology and lidar, more and more emerging technologies are being developed and applied to 3D modeling. However, traditional 3D reconstruction based on laser point clouds and images can only handle static or simple scenes, and cannot achieve reasonable geometric modeling of unobserved areas, and the modeling accuracy is difficult to meet current application requirements.

[0003] Neural Radiance Fields is a neural rendering algorithm used to generate high-quality rendered images in 3D scenes. It can generate high-precision, high-fidelity 3D scenes from image data from a single perspective. With the help of multi-layer perceptrons (MLP), NeRF can well learn the information in image data and achieve scene reconstruction of complex geometric shapes and textures, and the reconstruction effect is highly detailed. With the increase in the number of researchers in recent years, NeRF has developed very rapidly and has attracted the attention of scholars in the field of 3D reconstruction. However, due to the huge amount of data in 3D reconstruction, the traditional NeRF is very slow in model training and rendering, and requires high-performance computing equipment, which limits NeRF in 3D reconstruction. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for reconstructing a three-dimensional car model based on NeRF with semantic segmentation, which overcomes the above problems or at least partially solves the above problems. The reconstruction target and the background are separated, so that the amount of NeRF data during reconstruction is greatly reduced, thereby achieving high-efficiency and high-precision three-dimensional car reconstruction.

[0005] A NeRF-based 3D car model reconstruction method based on semantic segmentation includes the following steps:

[0006] Step 1: Using an RGB-D depth camera to obtain image information of a static target object at different viewing angles, including RGB-D image data with depth information;

[0007] Step 2: Perform semantic segmentation on the RGB-D image with depth information based on Mask R-CNN, and obtain the spatial information of the static target object in the image;

[0008] Step 3: Use the spatial information of the static target object from different perspectives to construct a hierarchical feature grid to represent the geometric shape and appearance of the static target object;

[0009] Step 4: Import the image feature information of the hierarchical feature grid into the neural radiation field generation model;

[0010] Step 5: Use volume rendering technology to render the generated model to obtain depth information and RGB, compare it with the RGB-D image data with depth information to calculate the reconstruction loss, minimize and optimize the rendering loss function, and complete the reconstruction of the feature mesh three-dimensional static target object.

[0011] Preferably, the hierarchical feature grid in step 3 includes three resolutions, and the three resolutions decrease from large to small.

[0012] Optionally, step 1 includes:

[0013] Step 1.1, collect image data of static target objects through a data acquisition device equipped with an RGB-D depth camera;

[0014] Step 1.2: Arrange the collected image dataset of static target objects to obtain an RGB-D image dataset with depth information.

[0015] Optionally, the step 2 includes:

[0016] Step 2.1, extract features from the RGB-D image with depth information obtained in step 1 through a convolutional neural network to obtain a feature map;

[0017] Step 2.2: Extract candidate target regions on the feature map based on the RPN algorithm, and output the bounding box coordinates and confidence score of each target region;

[0018] Step 2.3: Use the classifier and regressor to classify the features in each target area and regress the bounding box to obtain the category and location information of the target area.

[0019] Step 2.4: Use the segmentation mask prediction network to output the segmentation mask of each target area;

[0020] Step 2.5: Separate the static target object from the feature map based on the segmentation mask generated by Mask R-CNN for each target area, and obtain the spatial information of the static target object: image plane position, depth information, and shape information.

[0021] The target region is a set of pixels on the feature map.

[0022] Optionally, the step 3 includes:

[0023] Step 3.1, construct the acquired spatial information of the static target object into a kd-tree structure;

[0024] Step 3.2, based on the kd-tree structure, the acquired spatial information is constructed into three feature grids with different resolutions through the octree map method, where the coarse grid resolution is 2m×2m, the middle grid resolution is 32cm×32cm, and the fine grid resolution is 16cm×16cm;

[0025] Step 3.3: Use the trilinear interpolation algorithm to extract feature information from three grids of different resolutions.

[0026] Optionally, step 4 includes:

[0027] Step 4.1. Import the image feature information of the hierarchical feature grids: coarse grid, middle grid, middle grid and fine grid, and fine grid into the four discriminators of the neural radiation field, coarse MLP ( )、middleMLP( )、fine MLP( )、 ;

[0028] It is a 5×32 pre-trained fully connected neural network that obtains feature information from the coarse grid and outputs the occupancy value of the coarse grid , the formula is as follows:

[0029] ;

[0030] in Represents any point in the coarse grid, , Indicates the coarse grid at point Trilinear interpolation is performed at , express Neural network function; R 3 is a three-dimensional space set.

[0031] It is a 5×32 pre-trained fully connected neural network that obtains feature information from the middle grid and outputs the occupancy value of the middle grid , the formula is as follows:

[0032] ;

[0033] in Represents any point in the middle grid, , Indicates that the middle grid is at point Trilinear interpolation is performed at , express Neural network function;

[0034] It is a 5×32 pre-trained fully connected neural network that obtains feature information from middle and fine grids and outputs feature grid occupancy values ​​in the form of residuals. , the formula is as follows:

[0035] ;

[0036] in Represents any point in the middle and fine grids, , Indicates that the fine grid is at point Trilinear interpolation is performed at , express Neural network function;

[0037] It is a 5×32 fully connected neural network that obtains feature information from the fine grid and outputs the color RGB value of the fine grid , the formula is as follows:

[0038] ;

[0039] in represents fine grid, Indicates the corresponding decoder. Step 4.2: The output and The output of the fine-level feature decoder is summed up as the result , the formula is as follows: ;

[0040] The pre-trained MLP model parameters in steps 4.1 and 4.2 are obtained by ConvONet training. ConvONet includes a CNN encoder and an MLP decoder. The encoder-decoder is pre-trained using a 3D static target object model dataset, and the MLP decoder is used after training.

[0041] Optionally, step 5 includes:

[0042] Step 5.1: Volume rendering obtains The possession value , and of as well as RGB value of

[0043] Step 5.2: Use volume rendering technology to obtain the predicted depth value of the generated model at different viewing angles, and calculate the depth information of the RGB-D image data at the corresponding viewing angle. and predicted depth information The geometric loss between

[0044] ;

[0045] Where M represents the number of pixels, c represents the coarse layer, Represents the fine layer, l Step 5.3: Get the predicted RGB value of the generated model at different viewing angles through volume rendering technology, and calculate the RGB value of the RGB-D image data at the corresponding viewing angle. and predicted RGB values Color loss between ;

[0046] Step 5.4: Sum the geometric error and color error, update the feature grid scene expression, minimize the loss, and finally achieve 3D reconstruction of the static target object model.

[0047] ;

[0048] in Indicates color error The weight of They are spatial angle, pigment, spatial collection and temporal collection.

[0049] The static target object is a car.

[0050] The beneficial effects of the present invention are:

[0051] 1. The present invention uses Neural Radiance Field (NeRF) to reconstruct 3D car models, which greatly improves the efficiency of 3D car reconstruction compared to traditional manual modeling, physical modeling, and modeling algorithms based on laser point clouds or images.

[0052] 2. Compared with traditional NeRF, the NeRF-based 3D reconstruction method of automobiles combines MaskR-CNN semantic segmentation and multi-level feature grid construction with matching multi-level MLP sensing deviation to improve the accuracy of 3D automobile modeling.

[0053] At the same time, the multi-level feature grid that only focuses on the modeling target greatly reduces the amount of processed data compared to the original image feature grid, thereby improving the training and reconstruction efficiency of NeRF in 3D car models. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a NeRF-based 3D vehicle reconstruction method of the present invention based on semantic segmentation;

[0055] Figure 2 It is a flowchart of Mask R-CNN of the present invention;

[0056] Figure 3 It is a flow chart of the NeRF algorithm model based on semantic segmentation of the present invention. DETAILED DESCRIPTION

[0057] The specific implementation of the present invention is described in detail in conjunction with the following drawings. This specific embodiment is exemplary and is intended to illustrate and explain the present invention, but does not limit the present invention. The following describes the conventional algorithm used in the specific implementation of the present invention in conjunction with the prior art:

[0058] Kd-tree is an extension of binary tree index in multidimensional space. The construction steps of kd-tree include:

[0059] K01. Set a threshold, which is the maximum number of points allowed to exist in the space represented by the leaf node;

[0060] K02. Establish a root node, store the number of original point cloud points, calculate the variance of the point cloud data in three dimensions, take the dimension direction with the largest variance as the normal direction of the segmentation surface of the root node, that is, the initial discriminator, and the discriminators of subsequent layers increase with the number of layers;

[0061] K03. Determine whether the number of points contained in the space represented by the node is less than the threshold set in K01. If it is less than the threshold, store all point data in the node, otherwise proceed to K04;

[0062] K04. According to the discriminator of the layer where the node is located, determine the median, that is, the position of the splitting plane, split the space into two subspaces, establish two new nodes to represent the corresponding two subspaces, and perform K03 on the two nodes in turn;

[0063] K05. After the number of points contained in all subspaces is less than the threshold, the kd-tree index is completed.

[0064] The data is not moved during the indexing process. The index is only created based on the data. After the index is created, the corresponding data is stored in the corresponding leaf node. Each node of the KD tree is located on the subspace partition plane represented by the node. The partition plane will divide the space into two subspaces. The two child nodes of the node represent the two subspaces. All points in the space will also be assigned to the two subspaces.

[0065] The octree map divides the outer hexahedron containing the point cloud data according to the octree division rules (the depth of the division is determined by the set threshold). Each division is assigned to different subspaces by comparing the point coordinates with the center coordinates of the space. As the recursive division deepens, when the number of points in the subspace is less than the threshold, the data of the points contained in the subspace will be stored in the leaf node, and the "intermediate point" in the middle of the division does not store point cloud data. Finally, the point cloud data is assigned to a certain number of subspaces.

[0066] B1. Set the index threshold, which is the maximum number of points allowed to exist in the space represented by the leaf node;

[0067] B2. Establish the root node R, calculate the side length, center point and number of point cloud data contained in the outer hexahedron of the original point cloud and store them in the root node;

[0068] B3. If the number of point data contained in the hexahedral space represented by the node R is less than the threshold, then the part of the point cloud data is stored in the node, otherwise proceed to B4;

[0069] B4. Establish eight child nodes of node R, distribute the point cloud data to the eight child nodes according to the position of the point cloud data points relative to the center point of node R, and store the variable length, center point and number of data points of the subspace represented by each node, and execute B3 for each child node in turn;

[0070] B5. End index creation after all points are assigned.

[0071] Trilinear interpolation is an algorithm that calculates the center point based on the eight surrounding vertices of a cube.

[0072] The NeRF-based 3D car model reconstruction method for semantic segmentation includes the following steps:

[0073] Step 1: Use an RGB-D depth camera to obtain image information of a static target object (car) at different viewing angles;

[0074] Step 1.1, acquiring image data of a static target object through a data acquisition device equipped with an RGB-D depth camera;

[0075] Step 1.2: The collected car image data set is sorted to obtain an RGB-D image data set without depth information. The RGB-D image data with depth information obtained at the same time will be used as the reference truth value. The model reconstruction algorithm of the present invention will predict the image data with depth information with the help of neural radiation field and volume rendering technology, and minimize the loss between the image data and the reference truth value.

[0076] Step 2: Based on Mask R-CNN, semantic segmentation is performed on the RGB-D image with depth information, and spatial information of static target objects in the image is obtained at the same time; the present invention aims to perform three-dimensional reconstruction on static target objects, and the static target objects can be well extracted from the image background through the technical means of semantic segmentation, thereby saving unnecessary calculation and improving the efficiency of modeling.

[0077] Specifically include:

[0078] Step 2.1, extract features from the RGB-D image without depth information obtained in step 1 through a convolutional neural network to obtain a feature map;

[0079] Step 2.2: Extract candidate target regions on the feature map based on the RPN algorithm, and output the bounding box coordinates and confidence score of each region;

[0080] Step 2.3: Use the classifier and regressor to classify the features in each region and regress the bounding box to obtain the category and location information of the target.

[0081] Step 2.4: Use the segmentation mask prediction network to output the segmentation mask of each region target;

[0082] Step 2.5: Separate the static target object (car) from the image based on the segmentation mask generated by Mask R-CNN for each target region, and obtain the spatial information of the static target object: image plane position, depth information, shape information, and viewing direction. The target region is a set of pixels on the feature map.

[0083] Step 3: Construct a hierarchical feature grid to represent the geometric shape and appearance of the static target object in the image acquired from different perspectives. The hierarchical feature grid combines the pre-trained decoder to predict the grid occupancy. In the entire reconstruction algorithm model, the 3D model of the static target object is finally reconstructed by optimizing the grid occupancy.

[0084] Step 3.1, construct the acquired spatial information of the car into a kd-tree structure;

[0085] Step 3.2: Based on the kd-tree structure, the acquired spatial information is constructed into three feature grids with different resolutions through the octree map method: coarse grid, middle grid, and fine grid. The resolution of the coarse grid is 2m×2m, the resolution of the middle grid is 32cm×32cm, and the resolution of the fine grid is 16cm×16cm. The features of the static target object are extracted through hierarchical feature grids of different resolutions to obtain feature information at different resolutions. (The resolution of the coarse grid is significantly larger than the other two grid resolutions, so it will be used for prediction of larger areas, while the middle and fine grids are used for prediction updates of local areas).

[0086] The occupancy of the coarse feature grid is randomly initialized. The goal of the coarse feature grid is to capture a large range of geometric information in the scene (such as the body of the vehicle), while it can also predict unobserved parts of the scene, reducing scene hole information. The coarse feature grid will be optimized and updated independently of the middle and fine grids.

[0087] The occupancy of the middle feature grid is also randomly initialized, while the occupancy of the fine feature grid is initialized to zero. The goal of the middle and fine feature grids is to capture small local high-frequency details in the scene. During the reconstruction process, the geometry is first reconstructed by optimizing the middle feature grid, and then the fine feature grid is used for refinement to capture smaller high-frequency geometric details.

[0088] By constructing a hierarchical feature grid, image feature information of different dimensions is obtained based on different classifiers in subsequent steps to perform coarse MLP, middle MLP, fine MLP and predict color training.

[0089] Step 4: Import image feature information at different resolutions of the hierarchical feature grid into the neural radiation field generation model;

[0090] The neural radiation field in the present invention is composed of four multi-layer perceptrons (MLPs), which are used to combine different discriminators coarse MLP ( )、middle MLP( )、fine MLP( ) and predicted color ;

[0091] Step 4.1. Import the image feature information of the hierarchical feature grids coarse grid, middle grid, middle grid and fine grid, and fine grid into the four discriminators coarse MLP ( )、middle MLP( )、fine MLP( )、 ;

[0092] It is a 5×32 pre-trained fully connected neural network that obtains feature information from the coarse grid and outputs the occupancy value of the coarse grid , the formula is as follows:

[0093] ;

[0094] in Represents any point in the coarse grid, , Indicates the coarse grid at point Trilinear interpolation is performed at , express Neural network function; R 3 is a three-dimensional space set, and the scheme of the present invention belongs to a voxel grid. The grid formed by trilinear interpolation is The stride and size of the grid are the same.

[0095] It is a 5×32 pre-trained fully connected neural network that obtains feature information from the middle grid and outputs the occupancy value of the middle grid , the formula is as follows:

[0096] ;

[0097] in Represents any point in the middle grid, , Indicates that the middle grid is at point Trilinear interpolation is performed at , express Neural network function; the grid formed by trilinear interpolation and The stride and size of the grid are the same.

[0098] It is a 5×32 pre-trained fully connected neural network that obtains feature information from middle and fine grids and outputs feature grid occupancy values ​​in the form of residuals. , the formula is as follows:

[0099] ;

[0100] in Represents any point in the middle and fine grids, , Indicates that the fine grid performs trilinear interpolation at the point. express The neural network function of ; the grid formed by trilinear interpolation has the same step size and size as the grid it is in.

[0101] It is a 5×32 fully connected neural network that obtains feature information from the fine grid and outputs the color RGB value of the fine grid , the formula is as follows:

[0102] ;

[0103] in represents fine grid, Indicates the corresponding decoder.

[0104] Step 4.2: The output and The output of the fine-level feature decoder is summed up as the result , the formula is as follows:

[0105] ;

[0106] The parameters of the pre-trained MLP model in steps 4.1 and 4.2 are obtained by ConvONet training. ConvONet includes a CNN encoder and an MLP decoder. The encoder-decoder is pre-trained using a 3D car model dataset. After training, only the MLP decoder is used. The decoder of the model therefore has fixed parameters (the parameters will not be updated by loss back propagation, which only optimizes the feature grid occupancy. Fixed parameters of the MLP model help to optimize the feature grid stably, and there will be no instability caused by changes in MLP parameters). ConvONet is a grid convolution operation, and based on sensors of different resolutions, feature information occupancy values ​​of different neural network functions can be obtained. , , and RGB values , where the occupancy value is the grid proportion under different neural networks.

[0107] Step 5: Use volume rendering technology to render the generated model to obtain depth information and RGB, compare it with the RGB-D image data with depth information to calculate the reconstruction loss, minimize the rendering loss function, and update the multi-layer feature grid to complete the three-dimensional vehicle target reconstruction.

[0108] Step 5.1: Volume rendering obtains The possession value , and of as well as RGB value of

[0109] Step 5.2: Use volume rendering technology to obtain the predicted depth value of the generated model at different viewing angles, and calculate the depth information of the RGB-D image data at the corresponding viewing angle. and predicted depth information The geometric loss between them (optimized by L1 loss, which is used in coarse and fine layers);

[0110] ;

[0111] Where M represents the number of pixels, c represents the coarse layer, Represents the fine layer, l represents the predicted depth;

[0112] Step 5.3: Use volume rendering technology to obtain the predicted RGB values ​​of the generated model at different viewing angles, and calculate the RGB values ​​of the RGB-D image data at the corresponding viewing angles. and predicted RGB values Color loss between (optimized by L1 loss, which is used in the color layer)

[0113] ;

[0114] Step 5.4: Sum the geometric error and color error, update the feature grid scene expression, minimize the loss, and finally achieve 3D reconstruction of the car model.

[0115] ;

[0116] in Indicates color error The weight of Respectively spatial angle, color depth, spatial set and time set;

[0117] Volume rendering technology belongs to the existing 3D space reconstruction technology and belongs to the prior art, but its combination with semantic image segmentation belongs to the inventive point of the present invention. The car information obtained by volume rendering is optimized by RGB images with depth information, and the car 3D model is reconstructed based on the least squares convex optimization method, which can improve the reconstruction efficiency of the car target.

Claims

1. A method for reconstructing a three-dimensional target object model based on NeRF of semantic segmentation, comprising the following steps: Step 1: Use an RGB-D depth camera to obtain image information of a static target object at different viewing angles, including RGB-D image data with depth information; Step 2: Perform semantic segmentation on the RGB-D image with depth information based on Mask R-CNN, and obtain the spatial information of the static target object in the image; Step 3: Use the spatial information of the static target object from different perspectives to construct a hierarchical feature grid to represent the geometric shape and appearance of the static target object; Step 4: Import the image feature information of the hierarchical feature grid into the neural radiation field generation model; Step 5: Use volume rendering technology to render the generated model to obtain depth information and RGB, compare it with the RGB-D image data with depth information to calculate the reconstruction loss, minimize and optimize the rendering loss function, and complete the reconstruction of the feature mesh three-dimensional static target object; Step 2 includes: Step 2.1, extract features from the RGB-D image with depth information obtained in step 1 through a convolutional neural network to obtain a feature map; Step 2.2: Extract candidate target regions on the feature map based on the RPN algorithm, and output the bounding box coordinates and confidence score of each target region; Step 2.3: Use the classifier and regressor to classify the features in each target area and regress the bounding box to obtain the category and location information of the target area. Step 2.4: Use the segmentation mask prediction network to output the segmentation mask of each target area; Step 2.5, separate the static target object from the feature map according to the segmentation mask generated by Mask R-CNN for each target area, and obtain the spatial information of the static target object: image plane position, depth information and shape information; The target area is a set of pixels on the feature map; Step 4 includes: Step 4.1: Import the image feature information of the hierarchical feature grids: coarse grid, middle grid, middle grid and fine grid, and fine grid into the four discriminators of the neural radiation field, coarse MLP (MLP c )、middle MLP(MLP m )、fine MLP(MLP f ), MLP color ; MLP c It is a 5×32 pre-trained fully connected neural network that obtains feature information from the coarse grid and outputs the occupancy value of the coarse grid The formula is as follows: Where p represents any point in the coarse grid, p∈R 3 , Indicates that the coarse grid performs trilinear interpolation at point p, f 0 () indicates MLP c Neural network function; R 3 is a three-dimensional space set; MLP m It is a 5×32 pre-trained fully connected neural network that obtains feature information from the middle grid and outputs the occupancy value of the middle grid The formula is as follows: Where p represents any point in the middle grid, p∈R 3 , Indicates that the middle grid performs trilinear interpolation at point p, f 1 () indicates MLP m Neural network function; MLP f It is a 5×32 pre-trained fully connected neural network that obtains feature information from middle and fine grids and outputs feature grid occupancy values ​​in the form of residuals. The formula is as follows: Where p represents any point in the middle and fine grids, p∈R 3 , Indicates that the fine grid performs trilinear interpolation at point p, f 2 () indicates MLP f Neural network function; MLP color It is a 5×32 fully connected neural network that obtains feature information from the fine grid and outputs the color RGB value c of the fine grid p , the formula is as follows: where ψ ω (p) represents fine grid, g ω Indicates the corresponding decoder; Step 4.2: MLP m The output and MLP f The output of the fine-level feature decoder is summed up as the result o P , the formula is as follows: The pre-trained MLP model parameters in steps 4.1 and 4.2 are obtained by ConvONet training. ConvONet includes a CNN encoder and an MLP decoder. The encoder-decoder is pre-trained using a 3D target object model dataset, and the MLP decoder is used after training.

2. The method for reconstructing a three-dimensional target object model based on NeRF of semantic segmentation according to claim 1, characterized in that: The hierarchical feature grid in step 3 includes three resolutions, and the three resolutions decrease from large to small.

3. The method for reconstructing a three-dimensional target object model based on NeRF of semantic segmentation according to claim 1, characterized in that: The step 1 comprises: Step 1.1, collect image data of static target objects through a data acquisition device equipped with an RGB-D depth camera; Step 1.2: Arrange the collected image dataset of static target objects to obtain an RGB-D image dataset with depth information.

4. The method for reconstructing a three-dimensional target object model based on NeRF based on semantic segmentation according to claim 1, characterized in that: The step 3 comprises: Step 3.1, construct the acquired spatial information of the static target object into a kd-tree structure; Step 3.2, based on the kd-tree structure, the acquired spatial information is constructed into three feature grids with different resolutions through the octree map method: coarse grid, middle grid, and fine grid. The resolution of the coarse grid is 2m×2m, the resolution of the middle grid is 32cm×32cm, and the resolution of the fine grid is 16cm×16cm; Step 3.3: Extract the features of the static target object through hierarchical feature grids of different resolutions to obtain feature information at different resolutions.

5. The method for reconstructing a three-dimensional target object model based on NeRF of semantic segmentation according to claim 1, characterized in that: The step 5 comprises: Step 5.

1. Volume rendering acquisition from MLP c The possession value MLP m and MLP f o P and MLP color RGB value of Step 5.2: Use volume rendering technology to obtain the predicted depth value of the generated model at different viewing angles, and calculate the depth information D of the RGB-D image data at the corresponding viewing angle. m and predicted depth information The geometric loss between Where M represents the number of pixels, c represents the coarse layer, f represents the fine layer, and l represents the predicted depth; Step 5.3: Use volume rendering technology to obtain the predicted RGB values ​​of the generated model at different viewing angles, and calculate the RGB value I of the RGB-D image data at the corresponding viewing angle. m and predicted RGB values Color loss between Step 5.4, sum the geometric error and the color error, update the feature grid scene expression, minimize the loss, and finally achieve 3D reconstruction of the static target object model; where λ p Indicates color error The weights of θ, ω, {R i ,t i }Spatial angle, color depth, spatial collection and time collection respectively.

6. The method for reconstructing a three-dimensional target object model based on NeRF based on semantic segmentation according to any one of claims 1 to 5, characterized in that: The static target object is a car.

Images