Open vocabulary object-level neural radiation field mapping method

Abstract

The invention discloses an open vocabulary object-level neural radiation field mapping method, which relates to the technical field of robots, and comprises the following steps: firstly, an object-level segmentation and component-level understanding module focuses on instance segmentation and description feature and visual language feature extraction from a two-dimensional image, and simultaneously extracts SAM mask scale component-level features; secondly, a cross-view instance mask clustering module is responsible for establishing consistency association for masks of the same object from different views, so that the masks are used for supervising and training an object level NeRF; and finally, the neural radiation field rendering and training module receives object mask information and feature information from different views, and a final open vocabulary object-level neural radiation field map is obtained through volume rendering loss supervision training. According to the method, the semantic reasoning capability of objects and parts can be enhanced, and meanwhile, the scene is efficiently represented by using the neural radiation field so as to be used for open vocabulary accurate navigation and operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and particularly relates to an open-vocabulary object-level neural radiance field mapping method. BACKGROUND

[0002] Accurate reconstruction and comprehensive understanding of three-dimensional scenes are crucial for guiding robots to perform various downstream tasks. Traditional geometric mapping methods focus on constructing the geometric dimensions of the actual scene, thereby being used for point-to-point navigation and obstacle avoidance of mobile robots. Deep learning further introduces semantic information into the map space, realizing semantic-level spatial understanding and intelligent navigation. However, such closed-set semantic understanding is difficult to generalize to complex and diverse real-world scenes.

[0003] In recent years, visual language models (VLMs) pre-trained on large-scale network data have gradually gained attention due to their rich visual language understanding and strong generalization ability. Through contrast training on a large number of image-text pairs, the access of VLMs can endow traditional maps with open-vocabulary reasoning capabilities similar to human thinking. Thus, high-level cognitive navigation tasks such as “please help me find a place to rest” can be realized.

[0004] Existing open-vocabulary mapping methods mainly have the following defects:

[0005] Firstly, there is a lack of object-level scene understanding. Most existing methods directly project the pixel-scale open-vocabulary features on the image to obtain a point cloud map using depth information. The discrete point cloud structure is simple and direct, but it lacks instance concepts in three-dimensional space.

[0006] Secondly, the precision of part-level understanding is insufficient. With the help of “segmentation-all model” (SAM), some methods first extract object or part-level segmentation masks on the image, then use VLMs to extract open-vocabulary features at the mask scale, and cluster the point clouds that may belong to the same object or part under the supervision of multi-view constraints to obtain an approximate object-level scene representation. However, the mask scale of such methods is limited by the predefined granularity of SAM, and is limited to general object-level understanding, thus lacking the part-level understanding necessary for robot operation.

[0007] Thirdly, the map representation is scarce and inefficient. The initial open-vocabulary point cloud map and the improved open-vocabulary object-level map are still based on the discrete and inefficient point cloud representation. The map occupies a large amount of memory, and there is no connection or consistency constraint between feature point clouds belonging to the same object. Although some subsequent methods use a structured and compact neural radiance field to learn open-vocabulary features in the scene, they all lack object-level and part-level understanding and recognition.

[0008] Therefore, there is an urgent need for an open vocabulary mapping method for object-level component understanding. SUMMARY

[0009] Therefore, the present application provides an open vocabulary object-level neural radiation field mapping method, which realizes object-level component understanding, enhances semantic reasoning ability of objects and components, and uses neural radiation field to efficiently represent the scene for open vocabulary accurate navigation and operation.

[0010] To achieve the above purpose, the open vocabulary object-level neural implicit mapping method provided by the present application includes the following technical solutions:

[0011] First, the object-level segmentation and component-level understanding module performs instance segmentation and feature description and visual language feature extraction from two-dimensional images, while extracting SAM mask scale component-level features.

[0012] Second, the cross-view instance mask clustering module is responsible for establishing consistent association of the same object mask from different views, thereby being used for supervised training of object-level NeRF.

[0013] Finally, the neural radiation field rendering and training module accepts object mask information and feature information from different views, and obtains the final open vocabulary object-level neural radiation field map through volume rendering loss supervised training.

[0014] Further, after obtaining the final open vocabulary object-level neural radiation field map, the following steps are further included:

[0015] The constructed open vocabulary object-level neural radiation field map is used for robot global path planning and local component-level object operation, and the application process is as follows:

[0016] The global path planning is specifically as follows: the open vocabulary object-level neural radiation field map is projected to a 2D plane to obtain a 2D navigation obstacle map and a Vino topology navigation map; a natural language navigation instruction is issued, and an open vocabulary scene object similarity index confirms the navigation target position; and a mobile robot autonomously navigates and avoids obstacles.

[0017] The local component operation is specifically as follows: after reaching the target position, the best operation component of the target object is decided by a large language model LLM, the open vocabulary object-level neural radiation field further highlights the local point cloud on the object, and the grasping manipulator generates a grasping pose and performs grasping operation.

[0018] Further, the object-level segmentation and component-level understanding module specifically performs the following steps:

[0019] First, a pre-trained instance segmentation model CropFormer is used to extract a 2D mask without instance and semantic label category from each color image The extracted 2D mask without instance and semantic label category is used as input of the SAM model For the number of extracted 2D masks, each 2D mask extracted is considered as a complete instance on the image; then, the extracted 2D mask is input into the CLIP model to extract open-vocabulary visual features Further, the 2D detection bounding box of the 2D mask is provided to the TAP model, which describes the main object in the bounding box to obtain the text cap t,i ; the text will be further encoded by the large language model SBERT to generate high-dimensional embedding features Finally, the mask-scale CLIP features and description features are used in the next step of open-vocabulary cross-view instance mask clustering to assist in generating cross- Figure 1 view instance segmentation results to supervise object-level neural radiance field training.

[0020] Using the built-in part segmentation capability of the SAM model, part-level masks are extracted from each input color image For the total number of extracted part-level masks, the CLIP features extracted on each mask are weighted and fused on the image scale through mask distribution to obtain the final image pixel-level part feature image

[0021] After the above object-level instance segmentation and feature extraction, candidate object 2D masks and corresponding CLIP features and description features and part-level feature images for supervised part-level understanding are obtained on each frame of image pixel-level part feature image

[0022] Further, the cross-view instance mask clustering module, specifically:

[0023] Clustering the instance masks extracted by CropFormer that lack visual Figure 1 consistency, the clustering process includes a coarse clustering stage and a fine clustering stage, specifically:

[0024] First, in the coarse clustering stage: each instance mask is regarded as a node in the mask clustering graph , and the edge weight between each instance node is composed of the following 4 similarity parameters: 1. Geometric similarity S geo ​​Defined as the intersection-over-union (IOU) value between the three-dimensional point clouds obtained by projecting each mask through the depth image; 2. Optical similarity S pho Defined as the inner product between the color histograms on each mask; 3. CLIP feature similarity S clip Similarity S with 4. Description feature cap are defined as the feature cosine similarity between masks respectively; the final weighted matrix S is defined as the weighted sum of the above similarity matrices:

[0025] S=ω geo S geo +ω pho S pho +ω clip S clip +ω cap S cap

[0026] Among them, ω geo ,ω pho ,ω clip ,ω cap S geo 、S pho 、S clip 、S cap The weighted coefficients of , and the sum is 1.

[0027] For each pair of similarities in S that exceeds the threshold θ mask A similarity weighted edge is constructed between the masks, and then the Louvain graph clustering algorithm is used to perform community clustering on the undirected weighted graph to obtain a rough result.

[0028] The fine clustering stage: for the objects located at the edge of the image during camera observation, the global point cloud and average color histogram on each cluster block are counted. Whenever the intersection-over-union ratio and color similarity of two cluster blocks exceed the threshold θ pc and θ pho When , the two are further integrated, and cluster blocks with mask numbers less than N / 500 are treated as outliers and filtered out, thus obtaining the final instance mask clustering result:

[0029]

[0030] in For objects, is a cluster of object masks, is an object The mask clustering belongs to clustering All object masks constitutes an object The mask set Cross-vision Figure 1The consistent instance segmentation masks are used to supervise the subsequent object-level NeRF training.

[0031] Furthermore, the object-level neural radiance field rendering and training module specifically performs the following steps:

[0032] Each object Encoded as an object-level neural radiance field

[0033]

[0034] Where {x, y, z} is the input three-dimensional coordinate, {c, o, f} is the output color, occupancy probability and visual language feature vector respectively, θ k Then it is the optimizable network weight corresponding to MLP.

[0035] The object-level neural radiance field rendering and training is performed incrementally in the order of the picture sequence. and the 2D mask corresponding to the specific object instance in the image Each object level neural radiation field

[0036] Only in n containing the object k 2D bounding box on keyframes Internal random sampling ray r [u,v] And the corresponding ray sampling point p m ; The coordinates of these ray sampling points first pass through Predict the color c at the corresponding output position m , occupation probability o m , eigenvector f m Estimated value, obtained through the ray body rendering process, the corresponding 2D rendering value occupies color depth feature

[0037] After object-level neural radiance field rendering and training, the scene is finally represented as a set of object-level open vocabulary NeRFs, where each object The overall visual language feature of is taken as the corresponding largest cluster block in the cross-view clustering and

[0038] This holistic visual-linguistic feature is used for open-vocabulary object indexing, while the object-level open-vocabulary neural radiance field is used for Used for fine-grained component-level queries within objects.

[0039] Furthermore, the corresponding 2D rendering value occupies color depth feature The rendering process is as follows:

[0040]

[0041] Where T m =o m Π n＜m (1-o n ) represents ray r [u,v] At every point p m The termination probability at .

[0042] Furthermore, each object-level neural radiation field The loss function for training is defined as follows:

[0043]

[0044] in The corresponding loss function for 2D rendering value occupancy, is the loss function corresponding to the depth, I d [u,v] is the depth image input, is the loss function corresponding to the color, I c [u,v] is the color image input,

[0045] is the loss function corresponding to the feature, I f [u,v] is the object component-level CLIP encoding feature image, where u and v are the pixel coordinate positions corresponding to the ray in the neural body rendering process;

[0046] The final supervised training loss function is the weighted sum of the loss functions of all objects:

[0047]

[0048] Among them, λ1, λ2, λ3, and λ4 are The corresponding weight.

[0049] Beneficial effects:

[0050] 1:In view of the problem that the current open-vocabulary mapping method lacks object-level / part-level fine-grained understanding ability, the application proposes a novel open-vocabulary object-level neural radiance field mapping method. First, the object-level segmentation and part-level understanding module focuses on instance segmentation and feature description from two-dimensional images, visual language feature extraction, and SAM mask scale part-level feature extraction. Second, the cross-view instance mask clustering module is responsible for establishing consistency association of the same object mask from different views, thereby being used for supervised training of object-level NeRF. Finally, the neural radiance field rendering and training module accepts object mask information and feature information from different views, and obtains the final open-vocabulary object-level neural radiance field map through body rendering loss supervised training. The application can enhance the object and part semantic reasoning ability, and use neural radiance field to efficiently represent the scene for open-vocabulary accurate navigation and operation.

[0051] 2:Compared with the existing open-vocabulary mapping method, the application uses the latest instance segmentation model and VLM to extract part-level visual language features, and learns and understands the part-level knowledge inside the object through the object-level NeRF method.

[0052] 3:Compared with the existing open-vocabulary NeRF method, the application proposes to use object-level NeRF to supervise and train the visual language features of the scene object, which improves the interactivity of the map object. Compared with the existing object-level NeRF method, a two-stage instance mask clustering method based on open-vocabulary mask clustering is proposed, which does not need to rely on real instance segmentation label input. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flow chart of the open-vocabulary object-level neural radiance field construction method of the application;

[0054] Figure 2 The open-vocabulary object-level neural radiance field map constructed by the application can be used for robot global path planning and local part-level object operation application flow. DETAILED DESCRIPTION

[0055] The application will be described in detail below in combination with the drawings and examples.

[0056] Open-vocabulary: In traditional deep learning computer tasks, a fixed length and category of semantic label set is usually defined on the training set, that is, closed-vocabulary. However, such models are difficult to directly apply to other new scenarios with different semantic categories. To address this challenge, researchers have introduced the concept of open-vocabulary. That is, existing models do not need or only need simple fine-tuning to be applied to completely new scenarios or tasks, without being limited to predefined label categories. In recent years, VLM and LLM have the ability to understand and recognize approximately open-vocabulary in the training process of massive network data.

[0057] In view of the problem that the current open-vocabulary mapping method lacks object-level / part-level fine-grained understanding ability, the present application proposes a novel open-vocabulary object-level neural radiance field mapping method to improve the open-vocabulary object-level understanding and part-level operation ability of the constructed map.

[0058] The present application obtains a series of two-dimensional RGB images and Depth depth images and pose As input, incrementally construct an open-vocabulary object-level map Wherein the map is composed of a series of object-level neural radiance fields These neural radiance fields (NeRF, Neural Radiance Field) use a shallow multilayer perception (MLP, Multilayer Perception) to receive three-dimensional point position information x, y, z, output color values c, occupancy probability values o, and feature vector values f at any point in the encoding space, to achieve fine optical geometric modeling and part-level understanding at the object level.

[0059] NeRF: Neural Radiance Field, is a scene modeling method based on rendering principles for image supervision. It uses a simple shallow multilayer perception (MLP) to encode the body density, occupancy probability, signed distance field (SDF), and other geometric properties of the directional ray sampling points in three-dimensional space. RGB color values. The basic idea is to use volume rendering principles to synthesize RGB or depth values on each sampling ray, and construct a rendering loss between the actual observation to supervise the training of MLP weights, to achieve compact modeling and rendering of three-dimensional space.

[0060] Figure 1The method flowchart is shown, and the overall framework is composed of three main modules. First, the object-level segmentation and part-level understanding module focuses on instance segmentation and feature description from two-dimensional images, visual language feature extraction, and SAM mask scale part-level feature extraction. Second, the cross-view instance mask clustering module is responsible for establishing consistency association of the same object mask from different views, thereby being used for supervised training of object-level NeRF. Finally, the neural radiation field rendering and training module accepts object mask information and feature information from different views, and obtains the final open vocabulary object-level neural radiation field map through volume rendering loss supervised training.

[0061] As shown in Figure 2 , the open vocabulary object-level neural radiation field map constructed by the application can be used for robot global path planning and local part-level object operation, and the application process is as follows:

[0062] Global path planning: the open vocabulary object-level neural radiation field map is projected to a 2D plane to obtain a 2D navigation obstacle map and a Vino topology navigation map; natural language navigation instructions are given, such as “please find a white frying pan”; the open vocabulary scene object similarity index confirms the navigation target position; the mobile robot autonomously navigates and avoids obstacles.

[0063] Local part operation: after reaching the target position, the Large Language Model (LLM) decides the best operation part of the target object, such as “the handle of the frying pan”; the open vocabulary object-level neural radiation field further highlights the local point cloud on the object, guiding the generation of the grasping pose of the grasping manipulator and performing grasping operation.

[0064] The specific modules involved in the application are as follows:

[0065] Object-level segmentation and part-level understanding:

[0066] This module first uses the pre-trained instance segmentation model CropFormer to extract 2D masks without instance and semantic label categories from each color image The number of extracted 2D masks, due to the superiority of the CropFormer model, each 2D mask extracted can be regarded as a complete instance on the image. Subsequently, the extracted 2D mask is input into the CLIP model to extract open vocabulary visual features Further, in order to improve the semantic reasoning ability of the CLIP feature, the 2D detection bounding box of the 2D mask is provided to the TAP model, which will describe the main object in the bounding box to obtain the text cap t,i . The text will be further encoded by the Large Language Model SBERT to generate high-dimensional embedding features​ The final mask-scale CLIP feature and descriptive features It will be used in the next step of open vocabulary cross-view instance mask clustering to assist in generating cross-view Figure 1 Consistent instance segmentation results are obtained for supervised object-level neural radiance field training.

[0067] However, in order to obtain semantic understanding at the component level, it is not enough to integrate visual language features only at the instance scale. Similarly, feature extraction and fusion at the component level mask scale are required. Therefore, the present invention utilizes the built-in component level segmentation capability of the SAM model to extract the image from each input color image. Extracting dense component-level masks is the total number of component-level masks extracted, and the CLIP features extracted on each mask are The final image pixel-level component feature image is obtained by weighted fusion of mask distribution on the image scale

[0068]

[0069] After the above object-level instance segmentation and feature extraction, each frame image can be obtained Candidate 2D masks on and the corresponding CLIP features and descriptive features and part-level feature images for supervised part-level understanding

[0070] Cross-view instance mask clustering:

[0071] In order to obtain the multi-view instance segmentation images of supervised training object-level neural radiance fields, it is necessary to extract the non-visual images extracted by CropFormer. Figure 1 To address the object confusion problem caused by previous methods that greedily fuse instances based on similarity, this paper proposes a two-stage mask clustering method that comprehensively considers all observation frames.

[0072] First, in the coarse clustering stage: the present invention masks each instance Considered as a mask cluster map The edge weights between instance nodes are composed of the following four similarity parameters: 1. Geometric similarity S geo Defined as the intersection-over-union (IOU) value between the three-dimensional point clouds obtained by projecting each mask through the depth image; 2. Optical similarity S phoDefined as the inner product between the color histograms on each mask; 3. CLIP feature similarity S clip Similarity S with 4. Description feature cap are defined as the feature cosine similarity between masks. The final weighted matrix is ​​defined as the weighted sum of the above similarity matrices:

[0073] S=ω geo S geo +ω pho S pho +ω clip S clip +ω cap S cap (2)

[0074] Among them, ω geo ,ω pho ,ω clip ,ω cap S geo 、S pho 、S clip 、S cap The weighted coefficients of S are summed to 1. The present invention is for each pair of similarities in S that exceeds the threshold θ mask A similarity weighted edge is constructed between the masks, and then the Louvain graph clustering algorithm is used to perform community clustering on the undirected weighted graph to obtain a rough result.

[0075] Then comes the fine clustering stage: for special objects that are difficult to cluster at the edge of the image during camera observation, the fine stage counts the global point cloud and average color histogram on each cluster block. Whenever the intersection-over-union ratio and color similarity of two cluster blocks exceed the threshold θ pc and θ pho When , the two are further integrated, and cluster blocks with mask numbers less than N / 500 are treated as outliers and filtered out, thus obtaining the final instance mask clustering result:

[0076]

[0077] in For objects,

[0078] is a cluster of object masks, is an object The mask clustering belongs to clustering All object masks constitutes an object The mask set Cross-vision Figure 1 The consistent instance segmentation masks are used to supervise the subsequent object-level NeRF training.

[0079] Object-level neural radiance field rendering and training:

[0080] In this invention, in order to consistently and efficiently represent the internal geometry, color and visual language feature attributes of an object, each object is The encoding is represented as a NeRF neural network

[0081]

[0082] Where {x, y, z} is the input three-dimensional coordinate, {c, o, f} is the output color, occupancy probability and visual language feature vector respectively, θ k Then it is the optimizable network weight corresponding to MLP.

[0083] The object-level neural radiance field rendering and training is performed incrementally in the order of the picture sequence. and the 2D mask corresponding to the specific object instance in the image Per-object level NeRF Only in n containing the object k 2D bounding box on keyframes Internal random sampling ray r [u,v] And the corresponding ray sampling point p m The coordinates of these ray sampling points first pass through Predict the color c at the corresponding output position m , occupation probability o m , eigenvector f m Estimated value, the corresponding 2D rendering value can be obtained through the ray volume rendering process color depth feature The rendering process is as follows:

[0084]

[0085] Where T m =o m Π n＜m (1-o n ) represents ray r [u,v] At every point p m The termination probability at .

[0086] Further each object level neural radiation field The loss function for training can be defined as follows:

[0087]

[0088]

[0089] wherein is the 2D rendered value, and is the loss function corresponding to depth, I d [u,v] is the depth image input, is the loss function corresponding to color, I c [u,v] is the color image input,

[0090] is the loss function corresponding to feature, I f [u,v] is the object part-level CLIP encoded feature image, u,v are the pixel coordinate positions corresponding to the rays in the neural body rendering process;

[0091] The final supervised training loss function is the weighted sum of the loss functions of all objects:

[0092]

[0093] wherein λ1, λ2, λ3, λ4 are corresponding weights.

[0094] After the above object-level neural radiance field rendering and training, the present application finally represents the scene as a set of object-level open vocabulary NeRF, wherein each object takes the overall visual language feature of the object as the and The overall feature can be used Figure 2 in the open vocabulary object index, and the object-level open vocabulary neural radiance field is used for Figure 2 the fine-grained part-level query inside the object.

[0095] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An open vocabulary object-level neural implicit mapping method, characterized by: include: First, the object-level segmentation and component-level understanding module performs instance segmentation and description features, visual language feature extraction from the two-dimensional image, and simultaneously extracts SAM mask-scale component-level features; Secondly, the cross-view instance mask clustering module is responsible for establishing consistent associations between the masks of the same object from different views, which is used to supervise the training of object-level NeRF. Finally, the neural radiance field rendering and training module accepts object mask information and feature information from different views, and obtains the final open-vocabulary object-level neural radiance field map through volume rendering loss supervised training.

2. The open vocabulary object-level neural implicit mapping method according to claim 1, characterized in that: After obtaining the final open vocabulary object-level neural radiation field map, it also includes: The constructed open vocabulary object-level neural radiation field map is used for robot global path planning and local component-level object manipulation. The application process is as follows: Global path planning involves projecting the open-vocabulary object-level neural radiation field map onto a 2D plane to obtain a 2D navigation obstacle map and a Voronoi topology navigation map. Natural language navigation instructions are issued, and the open-vocabulary scene object similarity index confirms the navigation target location. The mobile robot then autonomously navigates and avoids obstacles. The specific operation of local components is as follows: after reaching the target position, the large language model LLM decides the best operating component of the target object, and the open vocabulary object-level neural radiation field further highlights the local point cloud on the object, guiding the grasping robot to generate the grasping posture and perform the grasping operation.

3. The open vocabulary object-level neural implicit mapping method according to claim 1, characterized in that: The object-level segmentation and component-level understanding module specifically performs the following steps: First, we use the pre-trained instance segmentation model CropFormer to extract the Extract 2D masks without instance and semantic label categories is the number of extracted 2D masks, and each extracted 2D mask is regarded as a complete instance on the image; then, the extracted 2D mask is input into the CLIP model to extract open vocabulary visual features Furthermore, the 2D detection bounding box of the 2D mask is provided to the TAP model. Under the guidance of the bounding box, the TAP model describes the main objects in it and obtains the text cap t,i ; The text will be further encoded by the large language model SBERT to produce high-dimensional embedding features The final mask-scale CLIP feature and descriptive features The next step is open vocabulary cross-view instance mask clustering, which helps generate consistent instance segmentation results across views in a semantic prior manner for supervised object-level neural radiance field training; Using the built-in component-level segmentation capability of the SAM model, from each input color image Extract component-level mask is the total number of component-level masks extracted, and the CLIP features extracted on each mask are The final image pixel-level component feature image is obtained by weighted fusion of mask distribution on the image scale After the above object-level instance segmentation and feature extraction, the pixel-level component feature image of each frame is obtained. Candidate 2D masks on and the corresponding CLIP features and descriptive features and part-level feature images for supervised part-level understanding 4. The open vocabulary object-level neural implicit mapping method according to claim 3, characterized in that: The cross-view instance mask clustering module is specifically: Cluster the instance masks that lack view consistency extracted by CropFormer. The clustering process includes coarse clustering and fine clustering stages, specifically: First, in the coarse clustering stage: each instance mask Considered as a mask cluster map The edge weights between instance nodes are composed of the following four similarity parameters:

1. Geometric similarity S geo It is defined as the intersection-over-union (IOU) value between the three-dimensional point clouds obtained by projecting each mask onto the depth image.

2. Optical similarity S pho Defined as the inner product between the color histograms on each mask; 3. CLIP feature similarity S clip Similarity S with 4. Description feature cap are defined as the feature cosine similarity between masks respectively; the final weighted matrix S is defined as the weighted sum of the above similarity matrices: S=ω geo S geo +oh pho S pho +oh clip S clip +oh cap S cap Among them, ω geo ,ω pho ,ω clip ,ω cap S geo 、S pho 、S clip 、S cap The weighted coefficients of , and the sum is 1; For each pair of similarities in S that exceeds the threshold θ mask A similarity weighted edge is constructed between the masks, and then the Louvain graph clustering algorithm is used to perform community clustering on the undirected weighted graph to obtain a rough result; The fine clustering stage: for the objects located at the edge of the image during camera observation, the global point cloud and average color histogram on each cluster block are counted. Whenever the intersection-over-union ratio and color similarity of two cluster blocks exceed the threshold θ pc and θ pho When , the two are further integrated, and cluster blocks with mask numbers less than N / 500 are treated as outliers and filtered out, thus obtaining the final instance mask clustering result: in For objects, is a cluster of object masks, is an object The mask clustering belongs to clustering All object masks constitutes an object The mask set The instance segmentation masks that are consistent across views are used to supervise the subsequent object-level NeRF training.

5. The open vocabulary object-level neural implicit mapping method according to claim 1, characterized in that: The object-level neural radiance field rendering and training module specifically performs the following steps: Each object Encoded as an object-level neural radiance field Where {x, y, z} is the input three-dimensional coordinate, {c, o, f} is the output color, occupancy probability and visual language feature vector respectively, θ k Then it is the optimizable network weight corresponding to MLP; The object-level neural radiance field rendering and training is performed incrementally in the order of the picture sequence. and the 2D mask corresponding to the specific object instance in the image Per-object level neural radiance field Only in n containing the object k 2D bounding box on keyframes Internal random sampling ray r [u,v] And the corresponding ray sampling point p m ; The coordinates of these ray sampling points first pass through Predict the color c at the corresponding output position m , occupation probability o m , eigenvector f m Estimated value, obtained through the ray body rendering process, the corresponding 2D rendering value occupies color depth feature After object-level neural radiance field rendering and training, the scene is finally represented as a set of object-level open vocabulary NeRFs, where each object The overall visual language feature of is taken as the corresponding largest cluster block in the cross-view clustering and This holistic visual-linguistic feature is used for open-vocabulary object indexing, while the object-level open-vocabulary neural radiance field is used for Used for fine-grained component-level queries within objects.

6. The open vocabulary object-level neural implicit mapping method according to claim 5, characterized in that: The corresponding 2D rendering value occupies the color depth feature The rendering process is as follows: Where T m =o m Π n＜m (1-o n ) represents ray r [u,v] At every point p m The termination probability at .

7. The open vocabulary object-level neural implicit mapping method according to claim 5, characterized in that: Per-object level neural radiance field The loss function for training is defined as follows: in The corresponding loss function for 2D rendering value occupancy, is the loss function corresponding to the depth, I d [u,v] is the depth image input, is the loss function corresponding to the color, I c [u,v] is the color image input, is the loss function corresponding to the feature, I f [u,v] is the object component-level CLIP encoding feature image, where u and v are the pixel coordinate positions corresponding to the ray in the neural body rendering process; The final supervised training loss function is the weighted sum of the loss functions of all objects: Among them, λ1, λ2, λ3, and λ4 are The corresponding weight.

Images