Open-set semantic segmentation method and system based on neural radiance fields

Abstract

The present disclosure relates to the technical field of computer vision, and proposes a neural radiance field based open set semantic segmentation method and system, which comprises the following steps: performing feature extraction on the obtained to-be-recognized image based on neural radiance field to obtain global perspective features of known class objects; calculating the hybrid distance between the global perspective features of the known class objects and the corresponding prototypes to constrain all known class features to be close to the corresponding class prototypes in the metric space, and obtaining a learned semantic field; and performing segmentation on unknown objects in a scene according to the learned semantic field by using a maximum Logits strategy based on metrics combined with an open operation method of morphology. The present disclosure uses an implicit neural scene expression method for open set segmentation, improves the feature expression capability and open set segmentation performance, and can realize accurate segmentation of unknown class objects in a scene while maintaining accurate segmentation of known class objects in the scene.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, in particular, to a neural radiance fields based open-set semantic segmentation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] Open-set semantic segmentation is one of the important research directions of computer vision, which is used for the identification and processing of unknown class objects suddenly appearing in the actual scene. Existing open-set semantic segmentation methods mainly fall into two categories: discriminative methods and generative methods. Discriminative methods mainly distinguish known class and unknown class pixels based on uncertainty, while generative methods usually use autoencoders or generative adversarial networks to reconstruct images from segmentation maps, mainly relying on the difference between reconstructed images and original inputs to detect unknown class pixels; however, the training of generative models is usually unstable, and they usually have complex network structures, in addition, the closed-set segmentation accuracy of generative models is usually lower than that of discriminative models, which limits the application of generative methods. Existing discriminative and generative methods mainly rely on convolutional neural networks to extract image features, these methods can obtain local context information of images, but lack the relevance of global perspective features of objects in the scene, therefore, they cannot achieve accurate segmentation of unknown objects in the scene.

[0004] In recent years, neural radiance fields based scene representation methods have attracted widespread attention in the field of computer vision, this method learns the representation information of three-dimensional scenes, such as geometric structure, color, texture features, etc., and implicitly establishes a model of the scene that can learn perspective-consistent features, but existing neural scene representation methods are trained on known category datasets, and can only segment known category objects in the scene, once unknown category objects appear in the scene, the semantic segmentation performance of this method will be greatly reduced. It can be seen that existing methods cannot achieve accurate segmentation of unknown objects in the scene. SUMMARY

[0005] In order to solve the above problems, the present disclosure proposes a neural radiance fields based open-set semantic segmentation method and system, which uses implicit neural scene representation method for open-set segmentation, improves the feature expression ability, and also improves the open-set segmentation performance, which can achieve accurate segmentation of unknown category objects in the scene while maintaining accurate segmentation of known category objects in the scene.

[0006] In order to achieve the above purpose, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a neural radiance fields based open-set semantic segmentation method, comprising the following steps:

[0008] extracting features of the acquired image to be identified based on the neural radiance field to obtain global perspective features of the known class objects;

[0009] calculating a hybrid distance between the global perspective features of the known class objects and corresponding prototypes to constrain all known class features to be close to corresponding class prototypes in a metric space, to obtain a learned semantic field;

[0010] According to the learned semantic field, an unknown object in a scene is segmented by using a maximum Logits strategy based on metrics combined with a morphological opening operation method.

[0011] One or more embodiments provide an open set semantic segmentation system based on a neural radiance field, comprising:

[0012] a neural radiance field feature extraction module configured to extract features of the acquired image to be identified based on the neural radiance field to obtain global perspective features of the known class objects;

[0013] a metric-driven feature constraint module configured to calculate a hybrid distance between the global perspective features of the known class objects and corresponding prototypes to constrain all known class features to be close to corresponding class prototypes in a metric space, to obtain a learned semantic field;

[0014] an unknown class segmentation module configured to segment an unknown object in a scene by using a maximum Logits strategy based on metrics combined with a morphological opening operation method according to the learned semantic field.

[0015] An electronic device includes a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of the open set semantic segmentation method based on the neural radiance field described above are completed.

[0016] A computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the steps of the open set semantic segmentation method based on the neural radiance field described above are completed.

[0017] Compared with the prior art, the beneficial effects of the present disclosure are:

[0018] This disclosure uses neural radiation fields to extract global viewpoint features of known object classes in a scene. Then, it learns a semantic field by calculating the mixture distance between the feature vectors and their corresponding prototypes. Based on this, it employs a metric-based maximization of Logits (MaxLogits) strategy combined with morphological opening operations to achieve accurate segmentation of unknown objects in the scene. This invention effectively meets the computer vision needs of complex real-world environments, improving the recognition and semantic understanding capabilities of unknown objects in a scene.

[0019] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0020] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0021] Fig. 1 This is a flowchart of the open set semantic segmentation method of Embodiment 1 of this disclosure;

[0022] Fig. 2 This is a system block diagram of the open set semantic segmentation system based on neural radiation field according to Embodiment 1 of this disclosure. Detailed Implementation

[0023] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0025] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0026] Example 1

[0027] In one or more of the technical solutions disclosed in the embodiments, such as Figs. 1-2 As shown, the open-set semantic segmentation method based on neural radiation fields includes the following steps:

[0028] Step 1: Extract features from the acquired image to be identified based on the neural radiation field to obtain global view features of known object classes;

[0029] Step 2: Calculate the mixing distance between the global view features of known class objects and their corresponding prototypes to constrain all known class features to be close to their corresponding class prototypes in the metric space, thus obtaining the learned semantic field.

[0030] Step 3: Based on the semantic field obtained after learning, the unknown objects in the scene are segmented by using a metric-based maximization of Logits strategy combined with morphological opening operation.

[0031] The method in this embodiment uses neural radiation fields to extract global viewpoint features of known object classes in a scene. Then, it learns a semantic field by calculating the mixture distance between the feature vectors and their corresponding prototypes. Based on this, it employs a maximization of Logits (MaxLogits) strategy combined with morphological opening operations to achieve accurate segmentation of unknown objects in the scene. This invention effectively meets the computer vision needs of complex real-world environments, improving the ability to recognize and understand the semantics of unknown objects in a scene.

[0032] It should be noted that segmentation tasks where all objects in the image dataset belong to a known specific category (with semantic labels) are closed-set segmentation tasks. Segmentation tasks where, in addition to known categories, some objects also belong to unknown categories (without semantic labels) are open-set segmentation tasks.

[0033] In step 1, a neural radiation field feature extraction module can be constructed. This module performs feature extraction based on the neural radiation field, and its structural diagram is shown below. Fig. 1 As shown, the feature extraction process includes the following steps:

[0034] Step 11: Based on a given ray sampling point x in the space, map it to a signed distance d(x) and a feature vector v using an MLP network. i ;

[0035] The symbolic distance d(x) is defined as:

[0036] d(x)=F θ (x)

[0037] Among them, F θ It refers to the MLP network, which stands for Multi-Layer Perceptron, a type of feedforward neural network model.

[0038] The signed distance function d(x) is used to determine the distance from a point to the boundary of a finite region in space, and to define the sign of the distance. Specifically, the distance is positive when the point is inside the boundary, negative when it is outside the boundary, and zero when it is on the boundary.

[0039] Step 12: Convert the signed distance d(x) into the volume density σ(x);

[0040] Step 13: Based on the obtained volume density σ(x), use volume rendering methods to extract the feature vector v in three-dimensional space. i Mapping to a two-dimensional image space yields the global viewpoint feature v(r);

[0041] For each pixel corresponding to a ray r, its feature vector can be obtained by the following equation (1):

[0042]

[0043] Among them, v i It is the feature vector of the i-th sampling point on the camera ray r, δ ii =||x i+1 -x i ||2 is the distance between adjacent sampling points This represents the cumulative transmittance along the light ray at each sampling point. σ j This represents the volume density at the j-th sampling point on the i-th ray.

[0044] Furthermore, the process includes a step of training the neural radiation field feature extraction module before use, during which a procedural loss function is incorporated. To constrain the MLP to satisfy the SDF property, a two-dimensional image supervision loss is also employed. Sparse depth supervision loss and geometric loss To optimize the neural radiation field, specifically by optimizing the weight parameters of the MLP, we obtain the trained neural radiation field feature extraction module.

[0045] Cheng Han's loss

[0046]

[0047] in, This represents a combination of points sampled from a random, uniform space and surface points of pixels.

[0048] Two-dimensional image supervision loss

[0049]

[0050] in, Let C(r) be the color of the pixel corresponding to the camera light, and C(r) be the color of the actual pixel. It is a set of camera rays that have passed through the sampled pixels.

[0051] Sparse depth supervision

[0052]

[0053] Where D(r) is the sparse depth generated using Colmap. This represents a set of camera rays that pass through sparse depth value pixels.

[0054] Geometric loss

[0055] Using semantic labels as priors, the normals of the walls and the ground are constrained based on the Manhattan hypothesis:

[0056]

[0057] in, and The normal loss for the ground and walls is calculated separately, and the specific design is consistent with the Manhattan-SDF model. and It refers to a set of camera rays that belong to the ground and walls, as determined by semantic tags.

[0058] Step 2, implement metric-driven feature constraint construction, which can be achieved by building a metric-driven feature constraint module. Specifically, the method for learning the semantic field includes the following steps:

[0059] Step 21: Calculate the feature vector v(r) and the category prototype m in the metric space. k The mixing distance;

[0060] Optionally, the Euclidean distance combining cosine angles can be used as the feature vector v(r) and the class prototype m of the metric space. k The mixing distance;

[0061] In this embodiment, the traditional semantic segmentation classifier is replaced by Euclidean distance in the metric space, which incorporates cosine angles. The feature vector v(r) and the category prototype m in the metric space are... k The mixing distance d(v(r),m k ), can be obtained through the following formula (2):

[0062]

[0063] Among them, the category prototype m kLet be an N-dimensional one-hot vector, k∈{1,...,N}, where N refers to the number of known category labels.

[0064] Step 22: Based on the mixing distance obtained in Step 21, calculate the class probability P of the feature vector v(r) belonging to the corresponding class K. k (r);

[0065] The probability that the pixel corresponding to each ray r belongs to category K can be obtained by the following formula (3):

[0066]

[0067] Step 23: Set the category probability P k The category corresponding to the largest value of r is taken as the category of the feature vector v(r). The correspondence between the feature vector v(r) and its category is constructed to obtain the closed-set semantic segmentation graph, as follows:

[0068]

[0069] A further technical solution, to ensure that known class feature vectors are closer to their corresponding prototypes and farther from other prototypes, while enhancing the compactness of feature vectors of the same class in the metric space, involves adjusting the class probabilities P... k In the process of classifying the feature vector v(r) into its corresponding category with the largest value (r), a classification loss function is set up by fusing the cross-entropy loss based on the mixture distance and the attraction loss function. The corresponding feature vector is then classified into the classification loss function L. m The smallest prototype category.

[0070] In this embodiment, a cross-entropy loss L based on mixed distance is used in the metric-driven feature constraint module. d :

[0071]

[0072] Where R is a set of camera rays passing through the sampled pixel.

[0073] Furthermore, to enhance the compactness of feature vectors of the same category in the metric space, an attraction loss function L is constructed. a :

[0074]

[0075] Therefore, the feature constraint loss in the metric-driven module is as follows:

[0076] L m =L d +λ a L a (7)

[0077] Where λ a This is a hyperparameter.

[0078] In step 3, the unknown objects in the scene are segmented, which can be achieved by constructing an unknown class segmentation module, including the following steps:

[0079] Step 31: Stitch together the blending distances corresponding to each camera ray to obtain the Logits of the entire image;

[0080] The model, including the neural radiation field feature extraction module and the metric-driven feature constraint module, is fully trained using a training dataset. The parameters in the network are continuously updated so that the features of known object categories in the scene are distributed near their corresponding prototypes. The final integrated distance Logits (feature vector) output by the network can be denoted as:

[0081]

[0082] Where D is composed of the mixing distance Logits corresponding to each camera ray under the target viewpoint, K represents the number of known categories, H represents the height of the image under the target viewpoint, and W represents the width of the image under the target viewpoint.

[0083] The blending distance obtained in step 2 is the distance Logits of the corresponding pixel point for each ray. Each time, the network outputs a batch of rays, which is the Logits of a batch of pixels. For example, if 1 batch = 1024, then 300 1024 pixels are stitched together in sequence to obtain the required 480*640 image.

[0084] Logits refer to the output of the last fully connected layer in a neural network; in this example, it refers to v(r). Here, distance Logits refers to the combined distance Logits, which is calculated by retrieving v(r) and m. k The mixed distance between them is obtained, that is, d(v(r),m k ).

[0085] Step 32: For the stitched image, use the metric-based MaxLogits strategy to obtain the probability map of the unknown class.

[0086] Specifically, first maximize the Logits transformation value D. i,j Then, for the maximum Logits transformation value D i,j Normalization is performed to obtain the probability map of the unknown class.

[0087] For each pixel (i,j), Maxlogits:D i,jIt can be obtained from the following formula:

[0088]

[0089] D i,j Normalization is performed to obtain the probability graph of the unknown class.

[0090]

[0091] Step 33: Use morphological opening operations to correct the edges of the categories in the probability graph of the unknown class;

[0092] The edges between different categories in the probability graph of unknown classes are ambiguous. This embodiment uses the opening operation method from morphology to solve this problem. The opening operation is performed by setting a structuring element B and then sequentially performing erosion and dilation operations on the obtained probability graph of unknown classes. The calculation formula is as follows:

[0093]

[0094] In this function, Opening(P,B) = Dilation(Erosion(P,B),B), where B is the structuring element (or kernel), Erosion is the erosion operation, and Dilation is the dilation operation.

[0095] Step 34: Set a pixel threshold λ, and segment unknown objects in the scene based on the obtained closed-set semantic segmentation map and the corrected unknown class probability map, as follows:

[0096] The final open set segmentation result image Y i,j for:

[0097]

[0098] Where λ is the threshold for distinguishing pixels from known and unknown classes, and y unseen The unknown class label is k+1.

[0099] To illustrate the technical effectiveness of the method in this embodiment, relevant experiments were conducted on the Replica dataset. The results were compared with existing methods such as MSP, MSP+Crf, DMLNet, and PEBAL. A certain class of objects in the scene was set as an unknown class. The performance of open set segmentation was evaluated using AUROC (area under the ROC curve) and AUPR (area under the precision-recall curve), and the performance of closed set segmentation was evaluated using Miou (mean intersection-union ratio of closed sets).

[0100] Among the existing methods: MSP is a discriminative semantic segmentation method that uses maximum softmax probability; MSP+Crf introduces a fully connected conditional random field on the basis of MSP; DMLNet uses a deep metric learning network with contrastive clustering for open set segmentation; and PEBAL identifies unknown classes by jointly optimizing pixel-level anomaly abstention learning and pixel-level energy model.

[0101] Table 1: Comparison Results of Open Set Partitioning

[0102]

[0103] As shown in Table 1, the method in this embodiment achieves the best results in both the open set segmentation metrics AUPR and AUROC compared to the other four open set segmentation methods. In particular, the AUPR metric shows a significant improvement, exceeding the current leading method PEBAL by 86.4%, indicating that the method in this embodiment enhances feature representation and possesses stronger open set segmentation performance. Furthermore, the method in this embodiment also achieves the best results in the closed set metric Miou compared to other methods, demonstrating that the method in this embodiment maintains good performance in both open set and closed set segmentation while achieving good performance in the latter.

[0104] Example 2

[0105] Based on Embodiment 1, this embodiment provides an open-set semantic segmentation system based on neural radiation fields, such as... Fig. 2 As shown, it includes:

[0106] Neural radiation field feature extraction module: configured to extract features from the acquired image to be identified based on the neural radiation field, and obtain global view features of known object classes;

[0107] The metric-driven feature constraint module is configured to calculate the mixture distance between the global view features of known class objects and their corresponding prototypes, so as to constrain all known class features to be close to the corresponding class prototypes in the metric space, thereby obtaining the learned semantic field.

[0108] Unknown Class Segmentation Module: Configured to segment unknown objects in the scene based on the learned semantic field, using a metric-based maximization of Logits strategy combined with morphological opening operation.

[0109] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0110] Example 3

[0111] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps in the open set semantic segmentation method based on neural radiation fields in Embodiment 1.

[0112] Example 4

[0113] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the open set semantic segmentation method based on neural radiation fields in Embodiment 1.

[0114] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0115] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. An open-set semantic segmentation method based on neural radiation fields, characterized in that, Includes the following steps: Based on the neural radiation field, feature extraction is performed on the acquired image to be identified to obtain the global view features of known object classes; The feature extraction of the acquired image to be identified based on the neural radiation field includes the following steps: Given a ray sampling point in a space, it is mapped to a symbolic distance and a feature vector through an MLP network; Convert the symbolic distance to volume density; Based on the obtained volume density, the volume rendering method is used to map the feature vectors in the three-dimensional space to the two-dimensional image space to obtain the global view features of known object classes. Calculate the mixture distance between the global view features of known class objects and their corresponding prototypes to constrain all known class features to be close to their corresponding class prototypes in the metric space, thus obtaining the learned semantic field; Based on the semantic field obtained after learning, a metric-based maximization of Logits strategy combined with morphological opening operation method is used to segment unknown objects in the scene; The segmentation of unknown objects in the scene includes the following steps: The Logits of the entire image are obtained by stitching together the blending distances corresponding to each camera ray; For the stitched image, the probability map of the unknown class is obtained by using the metric-based maximization of Logits transformation. The edge of the category in the probability graph of the unknown class is corrected by using the opening operation in morphology; Set pixel threshold Based on the obtained closed-set semantic segmentation map and the corrected unknown class probability map, the unknown objects in the scene are segmented.

2. The open-set semantic segmentation method based on neural radiation fields as described in claim 1, characterized in that, The method for learning semantic fields includes the following steps: Calculate the mixture distance between the feature vector and the class prototype in the metric space; Based on the obtained mixing distance, calculate the probability that the feature vector belongs to the corresponding class K; The category with the highest category probability is taken as the category of the feature vector. The correspondence between the feature vector and its category is constructed to obtain the closed-set semantic segmentation graph.

3. The open set semantic segmentation method based on neural radiation field as described in claim 2, characterized in that: The category with the highest category probability is taken as the category of the feature vector. Specifically: Set a classification loss function obtained by fusing cross-entropy loss based on mixed distance and attraction loss function, and classify the corresponding feature vector into the prototype category with the smallest classification loss function.

4. The open set semantic segmentation method based on neural radiation field as described in claim 2, characterized in that: We use Euclidean distance combined with cosine angles to calculate the mixed distance between the feature vector and the class prototype in the metric space.

5. The open set semantic segmentation method based on neural radiation field as described in claim 1, characterized in that: The opening operation involves performing erosion and dilation operations sequentially on the resulting probability graph of unknown classes by setting a structuring element B.

6. An open-set semantic segmentation system based on neural radiation fields, characterized in that, include: Neural radiation field feature extraction module: configured to extract features from the acquired image to be identified based on the neural radiation field, and obtain global view features of known object classes; The feature extraction of the acquired image to be identified based on the neural radiation field includes the following steps: Given a ray sampling point in a space, it is mapped to a symbolic distance and a feature vector through an MLP network; Convert the symbolic distance to volume density; Based on the obtained volume density, the volume rendering method is used to map the feature vectors in the three-dimensional space to the two-dimensional image space to obtain the global view features of known object classes. The metric-driven feature constraint module is configured to calculate the mixture distance between the global view features of known class objects and their corresponding prototypes, so as to constrain all known class features to be close to the corresponding class prototypes in the metric space, thereby obtaining the learned semantic field. Unknown Class Segmentation Module: Configured to segment unknown objects in the scene based on the learned semantic field, using a metric-based maximization of Logits strategy combined with morphological opening operation method; The segmentation of unknown objects in the scene includes the following steps: The Logits of the entire image are obtained by stitching together the blending distances corresponding to each camera ray; For the stitched image, the probability map of the unknown class is obtained by using the metric-based maximization of Logits transformation. The edge of the category in the probability graph of the unknown class is corrected by using the opening operation in morphology; Set pixel threshold Based on the obtained closed-set semantic segmentation map and the corrected unknown class probability map, the unknown objects in the scene are segmented.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps in the open set semantic segmentation method based on neural radiation fields as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps in the open set semantic segmentation method based on neural radiation fields as described in any one of claims 1-5.

Images