Three-dimensional scene interactive semantic segmentation method based on online feature distillation

Abstract

The invention discloses a three-dimensional scene interactive semantic segmentation method based on online feature distillation, which does not need semantic truth value data, and is characterized in that features of a pre-trained three-dimensional scene semantic segmentation large model are distilled into a three-dimensional grid for coding, and semantic features of any view angle of a three-dimensional scene are directly obtained through rendering. A three-dimensional scene semantic segmentation large model encoder does not need to be operated, and a high-quality semantic segmentation result can be obtained only by operating a decoder; the method can be used for tasks such as positioning and single object extraction based on a 3D space, and belongs to the technical field of computer vision. By adopting the technical scheme of the invention, the processing speed can be greatly improved while the semantic segmentation quality is kept.

Description

Technical Field

[0001] The present invention provides a technology for real-time interactive semantic segmentation of three-dimensional scene data such as neural radiance fields (NeRF) and three-dimensional meshes, and specifically relates to a three-dimensional scene interactive semantic segmentation method based on online feature distillation. The present invention utilizes a perception model to realize real-time interactive semantic segmentation of zero-sample three-dimensional scene data in any NeRF model, and can be used for tasks such as positioning and single object extraction in 3D space, and belongs to the field of computer vision technology. Background Art

[0002] Neural Radiance Fields (NeRF) in 3D scenes is a new 3D modeling technology in recent years (Ben Mildenhall et al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis". In: ECCV. 2020). It has received widespread attention since it was proposed in 2020. Initially, NeRF was used to solve the problem of novel view synthesis based on multi-view RGB image data. For the first time, NeRF achieved the generation effect of camera realism and can recover accurate depth information only from RGB images. From the perspective of graphics, NeRF combines implicit neural representation and volume rendering to learn a plenoptic function using a deep neural network model. Compared with traditional 3D reconstruction technology, the NeRF model has the following key advantages: (1) It only uses multi-view RGB images and their camera poses as training data, and does not rely on data such as depth images or point clouds that require additional acquisition equipment. In practical applications, users only need to provide multi-view images or videos taken by an RGB camera, and the camera pose can be estimated by existing Structure from Motion (SfM) algorithms such as COLMAP. (2) New view images with camera realism can be generated. Compared with other 3D scene reconstruction and new view synthesis technologies, NeRF has achieved unprecedented rendering quality. By modeling the observation perspective, NeRF can also distill perspective-dependent Non-lambertian effects, such as metallic luster, to a certain extent. Since NeRF itself can act as a bridge from 2D data to 3D data, it has attracted a lot of follow-up research after it was proposed, and has been applied in more and more new tasks. More and more 3D modeling and generation technologies have begun to use NeRF and have made unprecedented progress. However, existing NeRF applications mainly focus on learning the geometry and appearance information of images, and cannot handle the semantic information in the 3D space of videos. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned technologies, the present invention proposes a three-dimensional scene interactive semantic segmentation method based on online feature distillation, which realizes zero-sample semantic segmentation on three-dimensional scenes represented by NeRF or Mesh, reconstructs the segmentation process by directly rendering semantic features and only applying the decoder of the perception model, and projects the learned semantics onto the extracted mesh surface to achieve real-time interaction, which can greatly improve the processing speed while maintaining the quality of semantic segmentation.

[0004] The present invention introduces semantics into NeRF to expand its application potential. Although NeRF has been proven to be effective in practical applications such as virtual reality and digital creation, the lack of semantics limits the interaction with objects in complex scenes. The present invention proposes to distill the backbone features of the perceptual model to achieve zero-shot semantic segmentation through NeRF. The method of the present invention eliminates the need for large backbone models and helps maintain 3D consistency by directly rendering semantic features and only applying the decoder of the perceptual model to reconstruct the segmentation process. In addition, the learned semantics can be projected onto the extracted mesh surface to achieve real-time interaction. By distilling the features of the Segment Anything Model (SAM), the present invention is accelerated by 16 times while maintaining similar segmentation quality. Experimental results demonstrate the effectiveness and computational advantages of the method of the present invention.

[0005] The technical solution of the present invention is:

[0006] A 3D scene interactive semantic segmentation method based on online feature distillation distills the features of a pre-trained 3D scene semantic segmentation model into a multi-resolution hashed 3D grid for encoding without the need for semantic truth data. This allows the semantic features of any perspective to be directly obtained during the rendering process, avoiding the need to run the expensive large model encoder each time. Instead, only the decoder needs to be run to obtain semantic segmentation results of similar quality.

[0007] The interactive semantic segmentation method of three-dimensional scenes based on online feature distillation includes the following steps:

[0008] A. Based on the provided multi-view image dataset, the NeRF model is trained and the corresponding feature grid is extracted to reconstruct the 3D scene.

[0009] B. Perform online feature distillation training;

[0010] B1. According to the camera pose of the data set, the 3D scene model reconstructed in step A is rendered to obtain an RGB image of the corresponding perspective, and the camera pose is randomly interpolated as data enhancement;

[0011] B2. Run the encoder of the existing perception model (such as SAM) to encode these RGB images of different perspectives and obtain the semantic features of RGB images of different perspectives as the true semantic features. At the same time, a cache pool is maintained to record the perspective information and the corresponding true semantic features to avoid repeated calculations of the same perspective to improve training efficiency; the cache pool stores camera views, RGB images and corresponding feature maps;

[0012] B3. The semantic features predicted by RGB images of different perspectives are obtained through NeRF rendering, and the error between them and the true semantic features is reduced through the optimization process, completing the feature distillation from the existing perception model to our feature grid.

[0013] C. Interactive segmentation reasoning process;

[0014] C1. The user adjusts the 3D scene to a specific perspective and marks the object to be segmented by clicking on a part of it or describing it verbally as input;

[0015] C2. Obtain the RGB image and semantic features of a specific perspective by rendering, run the decoder of the perception model, decode the semantic features, and obtain the 2D semantic segmentation result under the corresponding perspective.

[0016] C3. Project the 2D semantic segmentation result onto the Mesh surface of the reconstructed 3D scene to obtain the 3D semantic segmentation result, so that a single object can be extracted from the 3D scene.

[0017] In step A, we use NeRF to reconstruct the geometry and appearance of a 3D scene from multi-view RGB images. We support any NeRF method that can render RGB images from any viewpoint.

[0018] In step B, we perform online feature distillation training. This step mainly involves distilling the features of existing perceptual models such as SAM (semantic segmentation model) into the feature grid obtained in step A, so that the semantic features of the new perspective can be directly obtained through rendering.

[0019] In step C, we use the feature grid distilled in B to perform interactive semantic segmentation and object extraction on the 3D scene.

[0020] Compared with the prior art, the beneficial technical effects of the present invention are:

[0021] The present invention provides a neural radiation field interactive semantic segmentation method based on online feature distillation, which can realize zero-sample semantic segmentation in NeRF by distilling the backbone features of the existing perception model. This method allows users to perform fine semantic segmentation of the reconstructed three-dimensional scene in real time through interactive operations, thereby realizing accurate positioning and operation of specific objects or areas. The present invention designs an efficient online feature distillation training algorithm, including camera pose enhancement and the use of a cache pool. By increasing the different camera perspectives in the training data, the robustness of the model to multi-angle observation is improved. The overhead of repeated calculations is also reduced by caching, further improving the speed of training. The method of the present invention obtains 3D scene segmentation results by projecting 2D segmentation results, which can extract the Mesh of each object in the scene (the object Mesh after the scene Mesh is segmented), providing a basis for subsequent rendering, animation and other applications, making the processing of specific objects more convenient.

[0022] The present invention avoids running expensive large model encoders through online feature distillation, increases the image semantic segmentation speed by 16 times, reaches the level of real-time segmentation, and supports the extraction of mesh of a single object for subsequent application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall process of this method.

[0024] Figure 2 It is a schematic diagram of the online feature distillation training process of the present invention.

[0025] Figure 3 is a schematic diagram of interactive segmentation via a graphical user interface (GUI). DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0027] First, we will focus on NeRF training and Mesh extraction. We use a NeRF model based on multi-resolution hash grids to improve training and rendering efficiency. Given the coordinate x of any point in three-dimensional space, its spatial density and color are obtained through two multi-layer perceptrons (MLPs) and hash grid encoders in the NeRF model:

[0028] σ=Φ(MLP(E geo (x)))

[0029] c=Ψ(MLP(E rgb (x)))

[0030] Where Φ is the exponential activation function and Ψ is the Sigmoid activation function. geoand E rgb They are the learnable hash grid encoders mentioned above. After that, we obtain the depth and RGB color of any light through volume rendering, and optimize the NeRF model through the loss function between the real data. After completing the NeRF training, we can extract the Mesh surface corresponding to the NeRF voxel grid through the Marching Cubes algorithm.

[0031] The second step is to train the distillation features. The overall process is as follows: Figure 2 As shown in Figure 2. Our proposed feature distillation method can be fully inserted into the original RGB and density fields of NeRF, which are fixed during the semantic feature distillation training. Given a camera view, we first render the RGB image I using NeRF. Then, we use the encoder Enc of the perceptual model to extract semantic features F:

[0032]

[0033] Where C is the semantic feature dimension, h, w are the height and width of the semantic feature (usually smaller than the image resolution). These semantic features (or feature maps) can be used to supervise the 3D feature grid, just like RGB information. We reuse the density information of points in 3D space and use numerical integration to render the semantic features F(r) along each ray:

[0034]

[0035] Where E sem is a learnable semantic feature grid, and T i =Π j＜i (1-α i ) are the positions x i Transparency and transmittance at i is the step size of the ray step. Since the feature dimension C is usually much higher than 3-channel RGB, we put the nonlinear MLP after the integration to save computation. In addition, after RGB rendering, we directly perform feature rendering on a smaller feature resolution h, w, further saving computation. For the case of using Mesh, we only need to sample one surface point x for each ray s , and simplify the rendering equation to:

[0036] F(r)=MLP(E sem (x i ))

[0037] Different perceptual models may require different numbers of feature maps for the decoder. For a single-scale decoder like the one used in SAM, we only need to distill one feature map F. Other models like X-Decoder use multi-scale feature maps {F i}, i∈{0, 1, 2}, in this case we share the feature grid E sem , only using different MLP i to predict multi-scale features, which helps exploit cross-scale correlations and reduce the total number of parameters. In the inference phase, we render all necessary semantic feature maps F for the test image and apply the decoder of the perceptual model to generate mask predictions M = Dec(F).

[0038] For different feature maps, we use different loss functions for supervision. For the single-scale decoder model, we directly minimize the rendered semantic features And the mean square error (MSE) loss between the true semantic feature F:

[0039]

[0040] For the multi-scale decoder model, feature maps at different scales are usually correlated and are used to represent the same underlying image.

[0041] We introduce an additional loss term, called cross-scale correlation loss. This loss term helps learn the correlation of semantic features between different scales and helps capture contextual information. We first calculate the i-th feature map F i and the jth feature map F j The cosine similarity mapping between them obtains the similarity of the true value features, which is expressed as a mapping matrix This cosine similarity map can be used as an additional supervisory signal by optimizing the similarity of the predicted features. Similarity S with the true feature ij The error between them, the cross-scale correlation loss we proposed can be expressed as:

[0042]

[0043] This cross-scale correlation loss can improve the quality of feature distillation and accelerate convergence.

[0044] In addition, we also adopt camera pose data augmentation and feature caching to improve training effect and efficiency. Since we can use pre-trained NeRF to synthesize RGB images from arbitrary camera poses, we increase the samples of the training dataset by interpolating between the original training camera poses in feature distillation training. By capturing a more diverse range of viewing angles, we find that camera augmentation helps to enhance the robustness of our semantic feature distillation and produce smoother segmentation prediction results. During training, we render new RGB images in real time and extract corresponding semantic features as supervision through the perception model. However, the above-mentioned real-time feature extraction training will lead to slower training speed because we need to run complex perception models in each training step. In order to reduce the large computational cost associated with the segmentation backbone network, we adopt a caching mechanism. During training, we maintain a cache pool that stores camera views, RGB images, and corresponding feature maps. In the initial step, we input the new view image into the segmentation backbone network and cache the generated feature map. In the subsequent steps, we use a randomization method to determine whether to use the cached feature map or sample a new feature map as supervision. By randomly choosing between cached feature maps and online generated feature maps, we strike a balance between reusing cached information and incorporating new data during training. To maintain efficiency and manage cache memory, we make the cache follow a first-in-first-out (FIFO) rule, where the oldest data is replaced by new data when the cache reaches capacity. By adopting this caching mechanism, we significantly reduce the computational cost and speed up the training process while maintaining a similar level of performance.

[0045] Finally, the interactive real-time segmentation process. We implemented a graphical user interface (GUI) to verify the user interaction process, such as Figure 3 As shown. The GUI allows the user to drag and view the 3D scene in real time. For any camera view, the user can enter prompt information (such as click) and our model will render a 2D segmentation mask on the RGB image. The user can then choose to project the 2D mask onto the 3D mesh surface, so that the mask can be rendered from other camera views. For the clicked point, we also project it into 3D space so that the position of the clicked point can be automatically tracked when the camera view is changed. By repeating the above process from multiple camera views, it is relatively easy to completely segment the target object from 3D space. Finally, the user can export the segmented mesh and texture map.

[0046] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A 3D scene interactive semantic segmentation method based on online feature distillation, characterized in that: Without semantic truth data, the features of the pre-trained 3D scene semantic segmentation model are distilled into a 3D grid for encoding. The semantic features of any perspective of the 3D scene are directly obtained through rendering. There is no need to run the 3D scene semantic segmentation model encoder, and only the decoder needs to be run to obtain high-quality semantic segmentation results. The steps are as follows: A. Extract the 3D feature grid from the multi-view image dataset and reconstruct the 3D scene; B. Perform online feature distillation training; including: B1. Render the 3D scene model reconstructed in step A according to the camera pose of the multi-view image dataset to obtain the corresponding view RGB image, and perform random interpolation on the camera pose as data enhancement; B2. Run the encoder of the perception model to encode RGB images of different viewing angles, and obtain semantic features of RGB images of different viewing angles as true value semantic features; at the same time, maintain a buffer pool to record the viewing angle information and the corresponding true value semantic features, so as to avoid repeated calculation of the same viewing angle to improve training efficiency; the buffer pool stores the camera view, RGB image and the corresponding feature map; B3. Obtain the semantic features predicted by RGB images from different perspectives through rendering, and optimize the error between the predicted semantic features and the true semantic features to complete the feature distillation from the perception model to the feature grid; C. Perform interactive segmentation reasoning; the process includes: C1. Adjust the 3D scene to a specific perspective and mark the objects to be segmented as input data; C2. Decode the semantic features obtained in step B3 by running the decoder of the perception model to obtain a two-dimensional semantic segmentation result under the corresponding perspective; C3. Project the 2D semantic segmentation result onto the mesh surface of the reconstructed 3D scene to obtain the 3D semantic segmentation result; Through the above steps, interactive semantic segmentation of three-dimensional scenes based on online feature distillation is achieved.

2. The three-dimensional scene interactive semantic segmentation method based on online feature distillation as claimed in claim 1, characterized in that: The obtained 3D semantic segmentation results are used to extract individual objects from the 3D scene.

3. The interactive semantic segmentation method for three-dimensional scenes based on online feature distillation as claimed in claim 1, characterized in that In step A, the three-dimensional scene is reconstructed by training the neural radiance field NeRF, extracting the corresponding three-dimensional mesh according to the multi-view RGB image, and reconstructing the geometry and appearance of the three-dimensional scene.

4. The three-dimensional scene interactive semantic segmentation method based on online feature distillation as claimed in claim 1, characterized in that: In step B, the perception model for online feature distillation training specifically adopts the SAM semantic segmentation model, which directly obtains the semantic features of the new perspective through rendering.

5. The three-dimensional scene interactive semantic segmentation method based on online feature distillation as claimed in claim 4, characterized in that: In step C, the distilled 3D feature grid is used to perform interactive semantic segmentation and object extraction on the 3D scene.

6. The three-dimensional scene interactive semantic segmentation method based on online feature distillation as claimed in claim 1, characterized in that: The encoder of the perceptual model is used to extract semantic features, which can be expressed as: Among them, F is the semantic feature; I is the RGB image; Enc is the encoder of the perception model; C is the feature dimension, h, w are the height and width of the feature; The density information of points in three-dimensional space is used, and numerical integration is used to render semantic features along each ray, expressed as: Where F(r) is the characteristic of distillation; E sem It is a learnable semantic feature grid; MLP is a multi-layer perceptron.

7. The method for interactive semantic segmentation of a three-dimensional scene based on online feature distillation as claimed in claim 6, characterized in that: For the case of a 3D mesh, we only need to sample one surface point for each ray and simplify the rendering equation to: F(r)=MLP(E sem (x i )) Among them, E sem is the feature grid.

8. The three-dimensional scene interactive semantic segmentation method based on online feature distillation as claimed in claim 6, characterized in that: By defining a cross-scale correlation loss, it is used to learn the correlation of semantic features between different scales and help capture contextual information, improve the quality of feature distillation and accelerate convergence; Cross-scale correlation loss It is expressed as: Among them, calculate the i-th feature map F i and the jth feature map F j The cosine similarity mapping between them obtains the similarity of the true value features, which is expressed as a mapping matrix is the similarity of the predicted features.

Images