Unsupervised new class discovery method, device and equipment based on implicit neural representation

Abstract

The application provides a kind of unsupervised new class discovery method, device and equipment based on implicit neural representation, wherein the method comprises: obtaining an image to be segmented, and segmenting the image to be segmented to obtain an initial sub-instance segmentation set;Based on neural radiance field model, render the implicit neural representation of the image to be segmented, use the implicit neural representation as the semantic feature of the image to be segmented, and calculate the cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented;Fuse the semantic feature and the entropy feature to obtain the fusion feature;Based on the coverage range of each sub-instance in the initial sub-instance segmentation set, extract the sub-instance feature set from the fusion feature;Cluster the sub-instance feature set to obtain the semantic segmentation result of the image to be segmented.The method, device and equipment provided by the application use neural radiance field model to realize implicit encoding of 3D space, significantly reduce memory consumption, and improve the recognition accuracy of new class.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and equipment for unsupervised new category discovery based on implicit neural representation. Background Art

[0002] In recent years, with the development of computer vision technology, the paradigm of scene perception algorithms has gradually transitioned from supervised artificial intelligence (Supervised Artificial Intelligence) to embodied artificial intelligence (Embodied Artificial Intelligence). Although supervised AI has achieved excellent performance in various visual perception tasks, it relies on large amounts of manually labeled, limited-category datasets to train models, and its recognition performance for new, unlabeled categories of objects is poor, making it difficult to apply on a large scale in open environments. The embodied AI paradigm leverages the interaction between the intelligent agent and the surrounding environment to continuously accumulate new knowledge, incrementally learning environmental geometry and semantic information, thereby enabling the discovery of new categories of objects, allowing the intelligent agent to still perform tasks in open environments. It can be widely used in fields such as indoor mobile robots and autonomous driving.

[0003] Most existing new category discovery methods identify new category objects by manually designing explicit feature descriptors or constructing 3D maps.

[0004] However, the above new category discovery methods not only require a large amount of memory to store 3D models, but also cannot render high-resolution recognition results. Summary of the Invention

[0005] The present invention provides an unsupervised new category discovery method, device and equipment based on implicit neural representation, which is used to solve the defects of existing new category discovery methods that most of them identify new category objects through manually designed explicit feature descriptors or constructing 3D maps, which not only requires a large amount of memory to store 3D models, but also cannot render high-resolution recognition results.

[0006] The present invention provides an unsupervised new category discovery method based on implicit neural representation, comprising:

[0007] Acquire an image to be segmented, and segment the image to be segmented to obtain an initial sub-instance segmentation set;

[0008] Rendering an implicit neural representation of the image to be segmented based on a neural radiation field model, using the implicit neural representation as a semantic feature of the image to be segmented, and calculating a cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented;

[0009] Fusing the semantic feature with the entropy feature to obtain a fused feature;

[0010] Extracting a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set;

[0011] Clustering the sub-instance feature sets to obtain a semantic segmentation result of the image to be segmented.

[0012] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, the training steps of the neural radiation field model include:

[0013] Extracting sample semantic features from a sample image, and obtaining the three-dimensional coordinates and viewing angle direction of a first sampling ray of the sample image, as well as a label pixel rendering result;

[0014] Obtaining an initial model, the initial model including a first model and a second model, the first model and the second model having the same model structure, the first model including a first trunk model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first trunk model, the second model including a second trunk model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second trunk model;

[0015] Inputting the three-dimensional coordinates and viewing direction of the first sampling ray into the first model, the first model performs sampling along the first sampling ray, and obtains and outputs a coarse pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs a first implicit neural representation feature based on the first implicit neural representation branch;

[0016] Determining the three-dimensional coordinates and viewing direction of a second sampling ray of the sample image based on the density value, inputting the three-dimensional coordinates and viewing direction of the second sampling ray into the second model, causing the second model to sample along the second sampling ray, and obtaining and outputting a fine pixel rendering result based on the second pixel rendering branch, and obtaining and outputting a second implicit neural representation feature based on the second implicit neural representation branch;

[0017] Based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, and the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result, the initial model is iterated on parameters to obtain the neural radiation field model.

[0018] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, the parameter iteration of the initial model is performed based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, and the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result, including:

[0019] Determining a feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature;

[0020] Determining a luminosity loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result;

[0021] Parameter iteration is performed on the initial model based on the feature loss and the photometric loss.

[0022] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, determining feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature includes:

[0023] Determining a first feature loss based on the first implicit neural representation feature and the sample semantic feature;

[0024] Determining a second feature loss based on the second implicit neural representation feature and the sample semantic feature;

[0025] The feature loss is determined based on the first feature loss and the second feature loss.

[0026] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, determining the photometric loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result includes:

[0027] Determining a first luminosity loss based on the coarse pixel rendering result and the label pixel rendering result;

[0028] Determining a second luminosity loss based on the fine pixel rendering result and the label pixel rendering result;

[0029] The luminosity loss is determined based on the first luminosity loss and the second luminosity loss.

[0030] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, clustering the sub-instance feature set to obtain the semantic segmentation result of the image to be segmented includes:

[0031] Obtaining a candidate sub-instance feature set based on similarities between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set;

[0032] Based on the candidate sub-instance feature set, a semantic segmentation result of the image to be segmented is obtained.

[0033] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, obtaining a semantic segmentation result of the image to be segmented based on the candidate sub-instance feature set includes:

[0034] The candidate sub-instance feature sets of sub-instances belonging to the same sub-instance and the same semantics are aggregated to obtain multiple aggregation results, and the multiple aggregation results are merged to obtain a semantic segmentation result of the image to be segmented.

[0035] According to an unsupervised new category discovery method based on implicit neural representation provided by the present invention, the cross entropy is calculated based on the semantic features to obtain the entropy features of the image to be segmented, including:

[0036] Based on the following formula, the entropy feature of the image to be segmented is obtained:

[0037]

[0038] in, represents the probability that the pixel category in the i-th frame to be segmented is o, ε i Represents the entropy feature, and the dimension of the entropy feature is N×H×W×1.

[0039] The present invention also provides an unsupervised new category discovery device based on implicit neural representation, comprising:

[0040] an acquisition unit, configured to acquire an image to be segmented, and segment the image to be segmented to obtain an initial sub-instance segmentation set;

[0041] a rendering unit, configured to render an implicit neural representation of the image to be segmented based on a neural radiance field model, use the implicit neural representation as a semantic feature of the image to be segmented, and calculate a cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented;

[0042] A feature fusion unit, configured to fuse the semantic feature and the entropy feature to obtain a fused feature;

[0043] an extraction unit, configured to extract a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set;

[0044] A clustering unit is used to cluster the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for discovering new categories based on implicit neural representation as described above is implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described unsupervised new category discovery methods based on implicit neural representation.

[0047] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-described unsupervised new category discovery methods based on implicit neural representation.

[0048] The present invention provides an unsupervised new category discovery method, device and equipment based on implicit neural representation, which obtains an image to be segmented, segments the image to be segmented, and obtains an initial sub-instance segmentation set; based on a neural radiation field model, renders the implicit neural representation of the image to be segmented, uses the implicit neural representation as the semantic feature of the image to be segmented, and calculates cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented; fuses the semantic feature and the entropy feature to obtain a fused feature; based on the coverage of each sub-instance in the initial sub-instance segmentation set, extracts a sub-instance feature set from the fused feature; clusters the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented. This process uses the neural radiation field model to realize implicit encoding of 3D space, which significantly reduces memory consumption. On the one hand, the neural radiation field model ensures the global temporal consistency of the semantic feature, and on the other hand, suppresses the noise semantics brought by the open environment, effectively solving the problems of poor fusion effect and weak robustness of the existing explicit feature method, and improving the accuracy of identifying and discovering new category semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is one of the flowcharts of the unsupervised new category discovery method based on implicit neural representation provided by the present invention;

[0051] Figure 2 1 is a flow chart of the training steps of the neural radiation field model provided by the present invention;

[0052] Figure 3 This is the second flowchart of the unsupervised new category discovery method based on implicit neural representation provided by the present invention;

[0053] Figure 4 Schematic diagram of the structure of the unsupervised new category discovery device based on implicit neural representation provided by the present invention;

[0054] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0056] The terms "first," "second," and the like in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type.

[0057] The present invention provides an unsupervised new category discovery method based on implicit neural representation. Figure 1 This is one of the flow charts of the unsupervised new category discovery method based on implicit neural representation provided by the present invention, such as Figure 1 As shown, the method includes:

[0058] Step 110 : Acquire an image to be segmented, and segment the image to be segmented to obtain an initial sub-instance segmentation set.

[0059] Specifically, an image to be segmented can be obtained. The image to be segmented, that is, the image that needs to be semantically segmented, can be pre-captured by an image acquisition device, or captured in real time, or downloaded or scanned from the Internet. The embodiment of the present invention does not make specific limitations on this.

[0060] Here, the image to be segmented may be an RGBD (RGB+Depth) image or an RGB image, etc., which is not specifically limited in the embodiment of the present invention.

[0061] After obtaining the image to be segmented, the image to be segmented can be segmented to obtain an initial sub-instance segmentation set. For example, the image to be segmented can be used to extract the edges of the image to be segmented using texture, geometry, and other information, and then the image can be segmented to generate an over-segmented initial sub-instance segmentation set.

[0062] That is, for the i-th frame to be segmented, if the input is an RGBD image, an RGBD-based image segmentation method, such as SceneCut, is used. Alternatively, a depth image segmentation method based on geometric constraints, such as GSM (Generalized Space-Mapping), is used to perform superpixel segmentation on the image to be segmented according to image texture and depth. If the input is an RGB image, an RGB-based image segmentation method is used to segment the image and generate an initial sub-instance segmentation set:

[0063]

[0064] in, Indicates the jth sub-instance segmentation of the i-th frame image to be segmented, Denotes the pre-segmentation function, I is the input RGB(D) image, O is defined as the set of all categories in the image to be segmented, N is the set of all instance segmentations in the image to be segmented, and represents the qth instance segmentation in the i-th frame of the image to be segmented, Represents the set of all instances of category p in the i-th frame image to be segmented, then there should be

[0065] Step 120: Render an implicit neural representation of the image to be segmented based on a neural radiation field model, use the implicit neural representation as a semantic feature of the image to be segmented, and calculate cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented.

[0066] Specifically, after obtaining the image to be segmented, the implicit neural representation of the image to be segmented can be rendered based on the neural radiance field model, and the implicit neural representation can be used as the semantic feature of the image to be segmented. The neural radiance field model here is pre-trained. The neural radiance field model can include two Embedding-NeRF models with the same model structure, that is, an Embedding branch (implicit neural representation branch) is added on the basis of the NeRF (Neural Radiance Fields) model. The NeRF model represents the scene as a neural radiation field synthesized by the visual field, and uses a neural network to fit the radiation field to achieve an implicit representation of the three-dimensional scene. The semantic features here reflect the feature information at the semantic level in the image to be segmented.

[0067] The embedding branch can render the appearance, geometry and implicit representation of the image simultaneously. The embedding branch is a Multi-Layer Perceptron (MLP) network parallel to the RGB branch.

[0068] After obtaining the semantic feature of the image to be segmented, the cross entropy can be calculated based on the semantic feature to obtain the entropy feature of the image to be segmented. For example, the semantic feature of the image to be segmented is input into the last two layers (the up-sampling layer and the regression layer) of the semantic feature extraction model, and the cross entropy is calculated on the output feature to obtain the entropy feature of the image to be segmented.

[0069] That is, the entropy feature of the image to be segmented is obtained based on the following formula:

[0070]

[0071] wherein, represents the probability of the pixel class o in the i-th frame of the image to be segmented, ε i represents the entropy feature, and the dimension of the entropy feature is N x H x W x 1.

[0072] First, a semantic segmentation model ESANet is trained on a pre-training dataset SUN-RGBD, and after the training is completed, the last two layers of the model, such as the up-sampling layer and the softmax regression layer, are removed, and the remaining model is used as the semantic feature extraction model f w , and w is the weight parameter of the semantic feature extraction model. During the training process, 9 classes in all classes of the SUN-RGBD dataset are selected as known classes, and the other 4 classes are unknown classes, and the finally extracted semantic feature E i is:

[0073]

[0074] wherein, E i is the high-dimensional semantic feature of the i-th frame of the image to be segmented, and the dimension is N x H x W x S, S is the number of channels of the last layer of the semantic feature extraction model, and when the image to be segmented is an RGBD image, S can be 37, that is, 37 is the number of channels of the last layer of the feature extraction model.

[0075] Let the known class set in the pre-training dataset be O known , and the unknown new class set in the target dataset (i.e. the dataset used for new class discovery) be O novel Obviously, f w has good feature extraction performance for instances belonging to O known , and has poor feature extraction performance for instances belonging to O novel .

[0076] Step 130: Fusing the semantic feature and the entropy feature to obtain a fused feature.

[0077] Specifically, after obtaining the semantic features and the entropy features, the semantic features and the entropy features may be fused to obtain fused features.

[0078] Here, the semantic features and entropy features may be fused, which may be concatenated, or weighted using an attention mechanism before concatenating the semantic features and entropy features. This is not specifically limited in the embodiment of the present invention.

[0079] Step 140 : extracting a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set.

[0080] Specifically, after obtaining the fused features, the sub-instance feature set can be extracted from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set. The j-th sub-instance feature set is:

[0081]

[0082] in represents the splicing operation, E i Represents the semantic features of the i-th frame of the image to be segmented, represents the fusion feature, is an N×H×W×(S+1)-dimensional vector, represents the entropy feature of the i-th frame to be segmented, and j represents the sub-instance.

[0083] Step 150: clustering the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented.

[0084] Specifically, after obtaining the sub-instance feature set, the sub-instance feature set can be clustered to obtain a semantic segmentation result of the image to be segmented.

[0085] For example, sub-instance feature sets of sub-instances belonging to the same sub-instance and the same semantics may be aggregated to obtain multiple aggregation results, and then the multiple aggregation results may be merged to obtain a semantic segmentation result of the image to be segmented.

[0086] The semantic segmentation results here refer to the semantic segmentation results that include known categories and new categories.

[0087] The method provided by the embodiment of the present invention obtains an image to be segmented, and segments the image to be segmented to obtain an initial sub-instance segmentation set; based on a neural radiation field model, renders an implicit neural representation of the image to be segmented, uses the implicit neural representation as the semantic feature of the image to be segmented, and calculates cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented; performs feature fusion on the semantic feature and the entropy feature to obtain a fusion feature; based on the coverage of each sub-instance in the initial sub-instance segmentation set, extracts a sub-instance feature set from the fusion feature; clusters the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented. This process uses the neural radiation field model to realize implicit encoding of 3D space, which significantly reduces memory consumption. The neural radiation field model can be used to render the implicit neural representation corresponding to the image to be segmented at any perspective, that is, the semantic feature of the image to be segmented, and the neural radiation field model fuses semantic information from multiple perspectives. The semantic feature has good spatiotemporal consistency, that is, it can render image semantic features of any scale for discovering new categories of objects, thereby improving the recognition accuracy of new categories.

[0088] Based on the above embodiments, Figure 2 This is a flow chart of the training steps of the neural radiation field model provided by the present invention, such as Figure 2 As shown, the training steps of the neural radiation field model include:

[0089] Step 210: extracting sample semantic features from the sample image, and obtaining the three-dimensional coordinates and viewing angle direction of the first sampling ray of the sample image, as well as the label pixel rendering result;

[0090] Step 220: Obtain an initial model, where the initial model includes a first model and a second model. The first model and the second model have the same model structure. The first model includes a first backbone model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first backbone model. The second model includes a second backbone model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second backbone model.

[0091] Step 230: Input the three-dimensional coordinates and viewing direction of the first sampling ray into the first model. The first model performs sampling along the first sampling ray, obtains and outputs a coarse pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs a first implicit neural representation feature based on the first implicit neural representation branch.

[0092] Step 240: determining the three-dimensional coordinates and viewing direction of a second sampling ray of the sample image based on the density value, inputting the three-dimensional coordinates and viewing direction of the second sampling ray into the second model, causing the second model to sample along the second sampling ray, and obtaining and outputting a fine pixel rendering result based on the second pixel rendering branch, and obtaining and outputting a second implicit neural representation feature based on the second implicit neural representation branch;

[0093] Step 250, based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, and the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result, the initial model is parameter iterated to obtain the neural radiation field model.

[0094] Specifically, we first train a semantic segmentation model ESANet on the pre-training dataset SUN-RGBD, and remove the last two layers of the model after training, such as the upsampling layer and the softmax regression layer, and use the retained model as the semantic feature extraction model f w , w is the weight parameter of the semantic feature extraction model. During the training process, 9 categories of all categories in the SUN-RGBD dataset are selected as known categories, and the other 4 categories are unknown categories. The final extracted sample semantic features E i for:

[0095]

[0096] Among them, E i is the sample semantic feature of the i-th frame of the image to be segmented, with a dimension of N×H×W×S. S is the number of channels in the last layer of the semantic feature extraction model. When the image to be segmented is an RGBD image, S can be 37, that is, 37 is the number of channels in the last layer of the feature extraction model.

[0097] That is, the semantic feature extraction model f can be used w The sample semantic features in the sample image are extracted, and the three-dimensional coordinates and viewing direction of the first sampling light of the sample image and the label pixel rendering result are collected in advance.

[0098] In addition, an initial model can also be constructed. The initial model here includes a first model and a second model. The first model and the second model have the same model structure. The first model includes a first trunk model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first trunk model. The second model includes a second trunk model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second trunk model. Both the first trunk model and the second trunk model are NeRF trunk models.

[0099] The first implicit neural representation branch and the second implicit neural representation branch have the same structure, and the first pixel rendering branch and the second pixel rendering branch have the same structure.

[0100] That is, both the first and second models are Embedding-NeRF models, which add an Embedding branch (implicit neural representation branch) to the NeRF model. The implicit neural representation branch can simultaneously render the appearance, geometry, and implicit representation of the image. The implicit neural representation branch is a multi-layer perceptron network that runs in parallel with the pixel rendering branch.

[0101] After building the initial model, you can use the pre-collected sample semantic features in the sample image, the 3D coordinates and viewing direction of the first sampling ray of the sample image, and the label pixel rendering results to train the initial model:

[0102] First, the three-dimensional coordinates and viewing direction of the first sampling ray of the sample image are input into the first model in the initial model. The first model samples along the first sampling ray intervals, and obtains and outputs the rough pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs the first implicit neural representation feature based on the first implicit neural representation branch.

[0103] Here, the three-dimensional coordinates of the first sampling ray can be expressed as o(x, y, z), and the viewing direction can be expressed as d(θ, φ). All rays to be rendered are generated by the three-dimensional coordinates and the viewing direction. r(t)=o+td, t is the point on the first sampling ray, and the nearest point t on the first sampling ray is t. n To the farthest point t f Sample K points between The RGB values ​​and density values ​​predicted by the K sampling points are integrated through the volume rendering equation to calculate the rough pixel rendering result (pixel RGB value) and density value corresponding to the first sampling light.

[0104] Then, the three-dimensional coordinates and viewing direction of the second sampling ray of the sample image are determined based on the density value, and the three-dimensional coordinates and viewing direction of the second sampling ray are input into the second model in the initial model. The second model samples along the second sampling ray, and obtains and outputs a fine pixel rendering result based on the second pixel rendering branch, and obtains and outputs a second implicit neural representation feature based on the second implicit neural representation branch.

[0105] Among them, the rendering process of the first model is as follows:

[0106] First, the first pixel rendering branch renders the rough pixel rendering results of each ray (RGB value) is:

[0107]

[0108]

[0109] Among them, α(x)=1-exp(-x), δ k =t k+1 -t k is the distance between adjacent sampling points, c(t k ) is the sampling point t k The rendered pixel RGB value, σ(t k ) represents the sampling point t k The density value of .

[0110] The first implicit neural representation branch renders the first implicit neural representation features of each ray for:

[0111]

[0112] Among them, e(t k ) is the sampling point t k The rendered S-dimensional implicit neural representation, where S dimension can be 37 dimensions, σ(t k ) represents the sampling point t k The advantage of using the first implicit neural representation branch to render implicit representation is that it implicitly integrates semantic information from multiple perspectives, making the rendered semantic features more globally consistent and robust.

[0113] The rendering process of the second model is the same as that of the first model, and will not be repeated here.

[0114] After obtaining the first implicit neural representation feature output by the first implicit neural representation branch, the second implicit neural representation feature output by the second implicit neural representation branch, the coarse pixel rendering result output by the first pixel rendering branch, and the fine pixel rendering result output by the second pixel rendering branch, the initial model can be iterated based on the first implicit neural representation feature and the second implicit neural representation feature and the sample semantic feature, as well as the coarse pixel rendering result, the fine pixel rendering result and the label pixel rendering result, and the initial model after completing the parameter iteration is used as the neural radiation field model.

[0115] The neural radiation field model after parameter iteration has the same structure as the initial model, so the neural radiation field model can be divided into two parts, namely the implicit neural representation part and the pixel rendering part.

[0116] Based on the above embodiment, step 250 includes:

[0117] Step 251, determining a feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature;

[0118] Step 252, determining a photometric loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result;

[0119] Step 253, performing parameter iteration on the initial model based on the feature loss and the photometric loss.

[0120] Specifically, after obtaining the first implicit neural representation feature and the second implicit neural representation feature, the first implicit neural representation feature, the second implicit neural representation feature, and the pre-extracted sample semantic feature can be compared respectively to obtain a feature loss.

[0121] After obtaining the coarse pixel rendering result and the fine pixel rendering result, the coarse pixel rendering result, the fine pixel rendering result, and the pre-collected label pixel rendering result can be compared respectively to obtain a photometric loss.

[0122] After obtaining the feature loss and the photometric loss, the initial model can be iterated based on the feature loss and the photometric loss.

[0123] Here, the initial model can be iterated based on the sum of the feature loss and the photometric loss, or based on the weighted sum of the feature loss and the photometric loss, which is not specifically limited by the embodiments of the present application.

[0124] Wherein, the formula for iterating the initial model based on the weighted sum of the feature loss and the photometric loss is as follows:

[0125] L = L p + λL e

[0126] Wherein, L represents the total loss, L p represents the photometric loss, L e represents the feature loss, and λ is a weight factor, which can be set to 1.0, or 0.8, 0.5, etc.

[0127] Based on the above embodiments, step 251 includes:

[0128] Step 310, determining a first feature loss based on the first implicit neural representation feature and the sample semantic feature;

[0129] Step 320, determining a second feature loss based on the second implicit neural representation feature and the sample semantic feature;

[0130] Step 330 determines the feature loss based on the first feature loss and the second feature loss.

[0131] Specifically, the first feature loss can be calculated based on the difference between the first implicit neural representation feature and the sample semantic feature. The first feature loss L ec The formula is as follows:

[0132]

[0133] Among them, L ec KL (Kullback-Leibler Divergence) divergence loss is used to constrain the implicit neural representation features of the rendering The consistency between the semantic features E(r) of the samples extracted by the semantic feature extraction model.

[0134] It can be understood that the greater the difference between the first implicit neural representation feature and the sample semantic feature, the greater the first feature loss; the smaller the difference between the first implicit neural representation feature and the sample semantic feature, the smaller the first feature loss.

[0135] Then, the second feature loss can be calculated based on the difference between the second implicit neural representation feature and the sample semantic feature. The second feature loss L ef The formula is as follows:

[0136]

[0137] Among them, L ef KL divergence loss is used to constrain the implicit neural representation features of the rendering The consistency between the semantic features E(r) of the samples extracted by the semantic feature extraction model.

[0138] It can be understood that the greater the difference between the second implicit neural representation feature and the sample semantic feature, the greater the second feature loss; the smaller the difference between the second implicit neural representation feature and the sample semantic feature, the smaller the second feature loss.

[0139] After obtaining the first feature loss and the second feature loss, the feature loss can be determined based on the first feature loss and the second feature loss. The formula of the feature loss is as follows:

[0140] L e =L ec +L ef

[0141] Among them, L e represents feature loss, L ec represents the first feature loss, L efrepresents the second feature loss.

[0142] Based on the above embodiment, step 252 includes:

[0143] Step 410: determining a first luminosity loss based on the coarse pixel rendering result and the label pixel rendering result;

[0144] Step 420 , determining a second luminosity loss based on the fine pixel rendering result and the label pixel rendering result;

[0145] Step 430: Determine the luminosity loss based on the first luminosity loss and the second luminosity loss.

[0146] Specifically, the first photometric loss can be calculated based on the difference between the rough pixel rendering result and the label pixel rendering result. The formula of the first photometric loss is as follows:

[0147]

[0148] Among them, L p1 represents the first photometric loss, where R is the total number of rays sampled in the current training batch image, C(r), They represent the label pixel rendering results (real RGB value) and the rough pixel rendering results respectively.

[0149] It can be understood that, the greater the difference between the coarse pixel rendering result and the labeled pixel rendering result, the greater the first photometric loss; the smaller the difference between the coarse pixel rendering result and the labeled pixel rendering result, the smaller the first photometric loss.

[0150] Then, the second photometric loss can be calculated based on the difference between the fine pixel rendering result and the label pixel rendering result. The formula of the second photometric loss is as follows:

[0151]

[0152] Among them, L p2 represents the second photometric loss, where R is the total number of rays sampled from the current training batch image, C(r), They represent the label pixel rendering results (real RGB value) and fine pixel rendering results respectively.

[0153] It can be understood that, the greater the difference between the fine pixel rendering result and the label pixel rendering result, the greater the second photometric loss; the smaller the difference between the fine pixel rendering result and the label pixel rendering result, the smaller the second photometric loss.

[0154] After obtaining the first luminosity loss and the second luminosity loss, the luminosity loss may be determined based on the first luminosity loss and the second luminosity loss. The formula for the luminosity loss is as follows:

[0155]

[0156] Among them, L p represents the photometric loss, where R is the total number of rays sampled in the current training batch image, C(r), They represent the label pixel rendering result (real RGB value), coarse pixel rendering result, and fine pixel rendering result respectively.

[0157] Based on the above embodiment, step 150 includes:

[0158] Step 151, obtaining a candidate sub-instance feature set based on the similarity between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set;

[0159] Step 152: Obtain a semantic segmentation result of the image to be segmented based on the candidate sub-instance feature set.

[0160] Specifically, a candidate sub-instance feature set can be obtained based on the similarity between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set. The similarity here can be calculated using methods such as cosine similarity and Pearson correlation coefficient, and the embodiments of the present invention do not specifically limit this.

[0161] Among them, the sub-instance feature set Segment set S with the initial sub-instance i The calculation formula for the similarity between each sub-instance in is as follows:

[0162]

[0163] in, represents the sub-instance feature set, S i Represents the initial sub-instance segmentation set.

[0164] After obtaining the candidate sub-instance feature set, the candidate sub-instance feature set can be clustered based on the Markov Cluster Algorithm (MCL) to obtain the semantic segmentation result of the image to be segmented.

[0165] Since the MCL algorithm is flexible in the number of cluster centers and has efficient clustering, the merged semantic segmentation results can be obtained in the end, thereby improving the accuracy and reliability of the semantic segmentation results.

[0166] Based on the above embodiment, step 152 includes:

[0167] The candidate sub-instance feature sets of sub-instances belonging to the same sub-instance and the same semantics are aggregated to obtain multiple aggregation results, and the multiple aggregation results are merged to obtain a semantic segmentation result of the image to be segmented.

[0168] Specifically, based on the Markov clustering algorithm, the candidate sub-instance feature sets belonging to the same sub-instance and the same semantic sub-instance can be aggregated to obtain multiple aggregation results, and the multiple aggregation results can be merged to obtain the semantic segmentation result of the image to be segmented.

[0169] Based on any of the above embodiments, Figure 3 This is the second flow chart of the unsupervised new category discovery method based on implicit neural representation provided by the present invention, such as Figure 3 As shown, the method includes:

[0170] The first step is to obtain the image to be segmented and segment it to obtain an initial sub-instance segmentation set. The image to be segmented can be an RGBD image.

[0171] The RGBD image can be input into the RGB encoding module and the depth encoding module respectively to obtain RGB encoding features and depth encoding features respectively, and then the RGB encoding features and depth encoding features are input into the content encoding module to obtain content encoding features, and then the content encoding features are input into the neural radiation field model.

[0172] In the second step, based on the neural radiation field model, the implicit neural representation of the image to be segmented is rendered, the implicit neural representation is used as the semantic feature of the image to be segmented, and the cross entropy is calculated based on the semantic feature to obtain the entropy feature of the image to be segmented.

[0173] The training steps for the neural radiation field model here include:

[0174] Extract the sample semantic features in the sample image, and obtain the three-dimensional coordinates and viewing direction of the first sampling ray of the sample image, as well as the label pixel rendering result.

[0175] Obtain an initial model, the initial model includes a first model and a second model, the model structure of the first model and the second model is the same, the first model includes a first implicit neural representation branch and a first pixel rendering branch, and the second model includes a second implicit neural representation branch and a second pixel rendering branch.

[0176] The three-dimensional coordinates and viewing direction of the first sampling light are input into the first model, and the first model samples along the first sampling light intervals, and obtains and outputs the rough pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs the first implicit neural representation feature based on the first implicit neural representation branch.

[0177] The three-dimensional coordinates and the perspective direction of the second sampling light ray of the sample image are determined based on the density value, the three-dimensional coordinates and the perspective direction of the second sampling light ray are input into the second model, the second model is sampled along the second sampling light ray, and a fine pixel rendering result is obtained and output based on the second pixel rendering branch, and a second implicit neural representation feature is obtained and output based on the second implicit neural representation branch.

[0178] The first feature loss is determined based on the first implicit neural representation feature and the sample semantic feature.

[0179] The second feature loss is determined based on the second implicit neural representation feature and the sample semantic feature.

[0180] The feature loss is determined based on the first feature loss and the second feature loss.

[0181] The first photometric loss is determined based on the coarse pixel rendering result and the label pixel rendering result.

[0182] The second photometric loss is determined based on the fine pixel rendering result and the label pixel rendering result.

[0183] The photometric loss is determined based on the first photometric loss and the second photometric loss.

[0184] The initial model is iterated in parameters based on the feature loss and the photometric loss.

[0185] In the third step, the semantic feature and the entropy feature are fused to obtain a fusion feature.

[0186] In the fourth step, a sub-instance feature set is extracted from the fusion feature based on the coverage range of each sub-instance in the initial sub-instance segmentation set.

[0187] In the fifth step, a candidate sub-instance feature set is obtained based on the similarity between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set.

[0188] In the sixth step, the candidate sub-instance feature sets belonging to the same sub-instance and the same semantic are aggregated to obtain multiple aggregation results, and the multiple aggregation results are merged to obtain a semantic segmentation result of the image to be segmented.

[0189] The unsupervised new class discovery device based on the implicit neural representation provided by the application is described below. The unsupervised new class discovery device based on the implicit neural representation described below can be mutually corresponding and referred to with the unsupervised new class discovery method based on the implicit neural representation described above.

[0190] Based on the above embodiment, the application provides an unsupervised new class discovery device based on an implicit neural representation, Figure 4is a schematic diagram of the structure of the unsupervised new category discovery device based on implicit neural representation provided by the present invention, such as Figure 4 As shown, the device includes:

[0191] An acquisition unit 410 is configured to acquire an image to be segmented and segment the image to be segmented to obtain an initial sub-instance segmentation set;

[0192] a rendering unit 420 configured to render an implicit neural representation of the image to be segmented based on a neural radiance field model, use the implicit neural representation as a semantic feature of the image to be segmented, and calculate a cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented;

[0193] A feature fusion unit 430 is configured to fuse the semantic feature and the entropy feature to obtain a fused feature;

[0194] An extraction unit 440 is configured to extract a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set;

[0195] The clustering unit 450 is configured to cluster the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented.

[0196] The device provided by the embodiment of the present invention obtains an image to be segmented, and segments the image to be segmented to obtain an initial sub-instance segmentation set; based on a neural radiation field model, renders an implicit neural representation of the image to be segmented, uses the implicit neural representation as the semantic feature of the image to be segmented, and calculates cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented; fuses the semantic feature and the entropy feature to obtain a fused feature; based on the coverage of each sub-instance in the initial sub-instance segmentation set, extracts a sub-instance feature set from the fused feature; clusters the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented. This process uses the neural radiation field model to realize implicit encoding of 3D space, which significantly reduces memory consumption. The neural radiation field model can be used to render the implicit neural representation corresponding to the image to be segmented at any perspective, that is, the semantic feature of the image to be segmented, and the neural radiation field model fuses semantic information from multiple perspectives. The semantic feature has good spatiotemporal consistency, that is, it can render image semantic features of any scale for discovering new categories of objects, thereby improving the recognition accuracy of new categories.

[0197] Based on any of the above embodiments, the training step of the neural radiation field model includes:

[0198] A semantic feature extraction unit is used to extract sample semantic features from a sample image, and obtain the three-dimensional coordinates and viewing angle direction of a first sampling ray of the sample image, as well as a label pixel rendering result;

[0199] an initial model acquisition unit, configured to acquire an initial model, the initial model including a first model and a second model, the first model and the second model having the same model structure, the first model including a first trunk model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first trunk model, the second model including a second trunk model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second trunk model;

[0200] a first model unit, configured to input the three-dimensional coordinates and viewing direction of the first sampling ray into the first model, wherein the first model performs sampling along the first sampling ray, obtains and outputs a coarse pixel rendering result and a density value based on the first pixel rendering branch, and obtains and outputs a first implicit neural representation feature based on the first implicit neural representation branch;

[0201] a second model unit, configured to determine the three-dimensional coordinates and viewing angle direction of a second sampling ray of the sample image based on the density value, input the three-dimensional coordinates and viewing angle direction of the second sampling ray into the second model, cause the second model to sample along the second sampling ray, obtain and output a fine pixel rendering result based on the second pixel rendering branch, and obtain and output a second implicit neural representation feature based on the second implicit neural representation branch;

[0202] A parameter iteration unit is used to perform parameter iteration on the initial model based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result to obtain the neural radiation field model.

[0203] Based on any of the above embodiments, the parameter iteration unit is specifically configured to:

[0204] a feature loss determining unit, configured to determine a feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature;

[0205] a luminosity loss determining unit, configured to determine a luminosity loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result;

[0206] A parameter iteration subunit is used to iterate the parameters of the initial model based on the feature loss and the photometric loss.

[0207] Based on any of the above embodiments, the feature loss determination unit is specifically configured to:

[0208] Determining a first feature loss based on the first implicit neural representation feature and the sample semantic feature;

[0209] Determining a second feature loss based on the second implicit neural representation feature and the sample semantic feature;

[0210] The feature loss is determined based on the first feature loss and the second feature loss.

[0211] Based on any of the above embodiments, the luminosity loss determination unit is specifically configured to:

[0212] Determining a first luminosity loss based on the coarse pixel rendering result and the label pixel rendering result;

[0213] Determining a second luminosity loss based on the fine pixel rendering result and the label pixel rendering result;

[0214] The luminosity loss is determined based on the first luminosity loss and the second luminosity loss.

[0215] Based on any of the above embodiments, the clustering unit 450 is specifically configured to:

[0216] a similarity calculation unit, configured to obtain a candidate sub-instance feature set based on similarities between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set;

[0217] The semantic segmentation unit is used to obtain a semantic segmentation result of the image to be segmented based on the candidate sub-instance feature set.

[0218] Based on any of the above embodiments, the semantic segmentation unit is specifically used to:

[0219] The candidate sub-instance feature sets of sub-instances belonging to the same sub-instance and the same semantics are aggregated to obtain multiple aggregation results, and the multiple aggregation results are merged to obtain a semantic segmentation result of the image to be segmented.

[0220] Based on any of the above embodiments, the rendering unit 420 is specifically configured to:

[0221] Based on the following formula, the entropy feature of the image to be segmented is obtained:

[0222]

[0223] in, represents the probability that the pixel category in the i-th frame to be segmented is o, ε i Represents the entropy feature, and the dimension of the entropy feature is N×H×W×1.

[0224] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute an unsupervised new category discovery method based on implicit neural representation, the method comprising: obtaining an image to be segmented, and segmenting the image to be segmented to obtain an initial sub-instance segmentation set; rendering an implicit neural representation of the image to be segmented based on a neural radiance field model, using the implicit neural representation as a semantic feature of the image to be segmented, and calculating cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented; fusing the semantic feature and the entropy feature to obtain a fused feature; extracting a sub-instance feature set from the fused feature based on the coverage of each sub-instance in the initial sub-instance segmentation set; and clustering the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented.

[0225] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the unsupervised new category discovery method based on implicit neural representation provided by the above methods, the method including: obtaining an image to be segmented, and segmenting the image to be segmented to obtain an initial sub-instance segmentation set; rendering the implicit neural representation of the image to be segmented based on a neural radiation field model, using the implicit neural representation as the semantic feature of the image to be segmented, and calculating the cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented; fusing the semantic feature and the entropy feature to obtain a fused feature; extracting a sub-instance feature set from the fused feature based on the coverage of each sub-instance in the initial sub-instance segmentation set; clustering the sub-instance feature set to obtain the semantic segmentation result of the image to be segmented.

[0227] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the unsupervised new category discovery method based on implicit neural representation provided by the above-mentioned methods, the method comprising: obtaining an image to be segmented, and segmenting the image to be segmented to obtain an initial sub-instance segmentation set; rendering the implicit neural representation of the image to be segmented based on a neural radiation field model, using the implicit neural representation as the semantic feature of the image to be segmented, and calculating the cross entropy based on the semantic feature to obtain the entropy feature of the image to be segmented; fusing the semantic feature and the entropy feature to obtain a fused feature; extracting a sub-instance feature set from the fused feature based on the coverage of each sub-instance in the initial sub-instance segmentation set; clustering the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented.

[0228] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0229] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An unsupervised new category discovery method based on implicit neural representation, characterized by: include: Acquire an image to be segmented, and segment the image to be segmented to obtain an initial sub-instance segmentation set; Rendering an implicit neural representation of the image to be segmented based on a neural radiation field model, using the implicit neural representation as a semantic feature of the image to be segmented, and calculating a cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented; Fusing the semantic feature with the entropy feature to obtain a fused feature; Extracting a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set; Clustering the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented; The training steps of the neural radiation field model include: Extracting sample semantic features from a sample image, and obtaining the three-dimensional coordinates and viewing angle direction of a first sampling ray of the sample image, as well as a label pixel rendering result; Obtaining an initial model, the initial model including a first model and a second model, the first model and the second model having the same model structure, the first model including a first trunk model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first trunk model, the second model including a second trunk model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second trunk model; Inputting the three-dimensional coordinates and viewing direction of the first sampling ray into the first model, the first model performs sampling along the first sampling ray, and obtains and outputs a coarse pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs a first implicit neural representation feature based on the first implicit neural representation branch; Determining the three-dimensional coordinates and viewing direction of a second sampling ray of the sample image based on the density value, inputting the three-dimensional coordinates and viewing direction of the second sampling ray into the second model, causing the second model to sample along the second sampling ray, and obtaining and outputting a fine pixel rendering result based on the second pixel rendering branch, and obtaining and outputting a second implicit neural representation feature based on the second implicit neural representation branch; Based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, and the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result, the initial model is iterated on parameters to obtain the neural radiation field model.

2. The unsupervised new category discovery method based on implicit neural representation according to claim 1 is characterized in that The performing parameter iteration on the initial model based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result includes: Determining a feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature; Determining a luminosity loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result; Parameter iteration is performed on the initial model based on the feature loss and the photometric loss.

3. The unsupervised new category discovery method based on implicit neural representation according to claim 2 is characterized in that The determining of feature loss based on the first implicit neural representation feature, the second implicit neural representation feature, and the sample semantic feature includes: Determining a first feature loss based on the first implicit neural representation feature and the sample semantic feature; Determining a second feature loss based on the second implicit neural representation feature and the sample semantic feature; The feature loss is determined based on the first feature loss and the second feature loss.

4. The unsupervised new category discovery method based on implicit neural representation according to claim 2, characterized in that The determining of the luminosity loss based on the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result includes: Determining a first luminosity loss based on the coarse pixel rendering result and the label pixel rendering result; Determining a second luminosity loss based on the fine pixel rendering result and the label pixel rendering result; The luminosity loss is determined based on the first luminosity loss and the second luminosity loss.

5. The unsupervised new category discovery method based on implicit neural representation according to any one of claims 1 to 4, characterized in that: Clustering the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented includes: Obtaining a candidate sub-instance feature set based on similarities between the sub-instance feature set and each sub-instance in the initial sub-instance segmentation set; Based on the candidate sub-instance feature set, a semantic segmentation result of the image to be segmented is obtained.

6. The unsupervised new category discovery method based on implicit neural representation according to claim 5, characterized in that Obtaining a semantic segmentation result of the image to be segmented based on the candidate sub-instance feature set includes: The candidate sub-instance feature sets of sub-instances belonging to the same sub-instance and the same semantics are aggregated to obtain multiple aggregation results, and the multiple aggregation results are merged to obtain a semantic segmentation result of the image to be segmented.

7. The unsupervised new category discovery method based on implicit neural representation according to any one of claims 1 to 4, characterized in that: The calculating cross entropy based on the semantic features to obtain the entropy features of the image to be segmented includes: Based on the following formula, the entropy feature of the image to be segmented is obtained: in, represents the probability that the pixel category in the i-th frame to be segmented is o, ε i Represents the entropy feature, and the dimension of the entropy feature is N×H×W×1.

8. An unsupervised new category discovery device based on implicit neural representation, characterized in that: include: an acquisition unit, configured to acquire an image to be segmented, and segment the image to be segmented to obtain an initial sub-instance segmentation set; a rendering unit, configured to render an implicit neural representation of the image to be segmented based on a neural radiance field model, use the implicit neural representation as a semantic feature of the image to be segmented, and calculate a cross entropy based on the semantic feature to obtain an entropy feature of the image to be segmented; A feature fusion unit, configured to fuse the semantic feature and the entropy feature to obtain a fused feature; an extraction unit, configured to extract a sub-instance feature set from the fused features based on the coverage of each sub-instance in the initial sub-instance segmentation set; A clustering unit, configured to cluster the sub-instance feature set to obtain a semantic segmentation result of the image to be segmented; The training unit is further included, and the training unit is specifically used to: Extracting sample semantic features from a sample image, and obtaining the three-dimensional coordinates and viewing angle direction of a first sampling ray of the sample image, as well as a label pixel rendering result; Obtaining an initial model, the initial model including a first model and a second model, the first model and the second model having the same model structure, the first model including a first trunk model, and a first implicit neural representation branch and a first pixel rendering branch respectively connected to the first trunk model, the second model including a second trunk model, and a second implicit neural representation branch and a second pixel rendering branch respectively connected to the second trunk model; Inputting the three-dimensional coordinates and viewing direction of the first sampling ray into the first model, the first model performs sampling along the first sampling ray, and obtains and outputs a coarse pixel rendering result and density value based on the first pixel rendering branch, and obtains and outputs a first implicit neural representation feature based on the first implicit neural representation branch; Determining the three-dimensional coordinates and viewing direction of a second sampling ray of the sample image based on the density value, inputting the three-dimensional coordinates and viewing direction of the second sampling ray into the second model, causing the second model to sample along the second sampling ray, and obtaining and outputting a fine pixel rendering result based on the second pixel rendering branch, and obtaining and outputting a second implicit neural representation feature based on the second implicit neural representation branch; Based on the first implicit neural representation feature, the second implicit neural representation feature, the sample semantic feature, and the coarse pixel rendering result, the fine pixel rendering result, and the label pixel rendering result, the initial model is iterated on parameters to obtain the neural radiation field model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the unsupervised new category discovery method based on implicit neural representation is implemented as described in any one of claims 1 to 7.

Images