Three-dimensional open vocabulary segmentation method, system and device based on consistency regularization and storage medium

Abstract

The invention provides a three-dimensional open vocabulary segmentation method, system and device based on consistency regularization and a storage medium. According to the invention, a geometric guidance consistency correction strategy and a self-adaptive guidance consistency correction strategy are introduced. Firstly, a geometric guidance consistency correction strategy can accurately explore an object proposal by means of a pre-trained geometric estimation network and SAM, unifies object features of different view angle images, and ensures the accuracy of feature reflection semantics under the different view angle images. Secondly, the self-adaptive guide consistency correction strategy fully considers the inherent multi-view consistency of NeRF, and effectively inhibits the adverse effect caused by the inconsistent knowledge of CLIP. In conclusion, through the innovative strategy and the ingenious design, the accuracy and reliability of the 3D segmentation result are remarkably improved, and compared with the prior art, the method has obvious advantages.

Description

Technical Field

[0001] The present invention belongs to the field of 3D open vocabulary segmentation, and relates to a three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization. Background Art

[0002] With the rapid development of computer vision, understanding 3D scenes has become an important research topic. Among many applications, 3D open vocabulary segmentation is one of the key tasks. Its goal is to segment the 3D scene into semantic regions based on a given open vocabulary text description. This helps to fully understand the composition and structure of the 3D scene, and can effectively connect 3D scenes with natural language. It has broad application prospects in fields such as autonomous driving and robot navigation. In recent years, many methods for 3D open vocabulary segmentation have emerged. Among them, Neural Radiance Field (NeRF) can be used to reconstruct 3D scenes from multi-view images; the pre-trained basic model is used to guide NeRF to perform 3D scene segmentation based on text descriptions.

[0003] A common basic model is the contrastive language-image pre-training (CLIP) model. CLIP is combined with NeRF. NeRF is used to reconstruct 3D scenes from multi-view images, while CLIP's visual and text encoders are used to map images to CLIP image features to establish a semantic relationship between 3D scenes and text descriptions at the image level during NeRF's 3D reconstruction process, thereby achieving segmentation. To further improve the segmentation results, DINO and Segment Anything Model (SAM) are also added to use the object-level cues they generate to optimize segmentation boundaries and improve the accuracy of the results. Thanks to NeRF's powerful 3D reconstruction capabilities and the rich prior knowledge hidden in the pre-trained basic models, these methods have achieved good segmentation results.

[0004] However, the existing schemes combining CLIP with NeRF have the problem of multi-view inconsistency caused by CLIP. Specifically, in a given set of multi-view images, the following situation occurs: the classification results of the same "gray sofa" in different views are very different, and it may be mistakenly identified as a "Gundam" or an "Xbox wireless controller". This clearly reveals that CLIP has a serious flaw, that is, when processing multi-view images, CLIP cannot extract consistent feature representations for the same object from different perspectives. This inconsistency brings negative interference to the radiating neural field, which greatly reduces the quality of the final generated segmentation result. Summary of the invention

[0005] In order to solve the multi-view inconsistency problem caused by CLIP in the above-mentioned prior art, the present invention provides a three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization.

[0006] The present invention is achieved through the following technical solutions:

[0007] A three-dimensional open vocabulary segmentation method based on consistency regularization, comprising:

[0008] S1, using NeRF to reconstruct the 3D scene from multi-view images, and generating an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0009] S2, using SAM and CLIP to extract the SegCLIP features of the objects in each view image, calculating the geometric correspondence between each view image and other view images, and aggregating the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain the SegCLIP guided features;

[0010] S3, based on the text features and SegCLIP guided features of each view image, the SegCLIP guided correlation map of each view image is generated by using the cosine similarity measurement method; multiple images are selected from the multi-view images as optimized view images; each optimized view image is optimized separately: the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image are optimized by using the SegCLIP guided correlation maps of all other view images except the current optimized view image, so as to obtain the optimal language radiation features of the current optimized view image;

[0011] S4, for each optimized view image, the similarity between the optimal language radiation feature and the text feature is measured, and the result with the highest score is taken as the segmentation result of the corresponding optimized view image.

[0012] Preferably, in S2, SAM and CLIP are used to extract SegCLIP features of objects in each perspective image, specifically: each perspective image is segmented into different objects using SAM; then, the CLIP visual encoder is used to map the SegCLIP features of each object separately to obtain the SegCLIP features of the objects in each perspective image.

[0013] Furthermore, SAM and CLIP are used to extract SegCLIP features of objects in each view image. Specifically, each view image is segmented into different multi-level objects using SAM. Then, the CLIP visual encoder is used to map each level of objects into SegCLIP features, and the SegCLIP features of objects at each level are combined to obtain the SegCLIP features of objects in each view image.

[0014] Preferably, in S2, the calculating of the geometric correspondence between each perspective image and other perspective images specifically comprises: calculating the geometric correspondence between each perspective image and other perspective images using a DKM network.

[0015] Preferably, in S2, the SegCLIP features of the same object in images of different viewing angles are aggregated according to the geometric correspondence to obtain SegCLIP guiding features, specifically: the SegCLIP features of the same object in images of different viewing angles are aggregated by obtaining the geometric correspondence of the images of each viewing angle to obtain aggregated SegCLIP features; then the SegCLIP features of all pixels of each object are averaged to obtain the SegCLIP guiding features.

[0016] Preferably, in S3, each optimized perspective image is optimized separately, specifically: according to the SegCLIP-guided correlation graph of all other perspective images except the current optimized perspective image, the thermal label of the corresponding perspective image is obtained, and according to the geometric correspondence between the current optimized perspective image and other perspective images, all the obtained thermal labels are transferred to the current optimized perspective image, and all the thermal labels are voted to select a most credible thermal label for the current optimized perspective image to obtain a reliable thermal label of the current optimized perspective image; under the supervision of the reliable thermal label, the initial language radiation feature in the CLIP-NeRF correlation graph of the current optimized perspective image is optimized to obtain the optimal language radiation feature of the current optimized perspective image.

[0017] Furthermore, the CLIP-NeRF correlation map of the current optimized view map is optimized under the supervision of the reliable thermal label, specifically: the CLIP-NeRF correlation map of the current optimized view map is optimized under the supervision of the reliable thermal label using the cross entropy loss.

[0018] The present invention also provides a three-dimensional open vocabulary segmentation system based on consistency regularization, comprising:

[0019] An initialization module is used to reconstruct a 3D scene based on multi-view images using NeRF, and generate an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0020] A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features;

[0021] The optimization module is used to generate the SegCLIP guided correlation graph of all view images by using the cosine similarity measurement method according to the text features and SegCLIP guided features of each view image; select multiple images from the multi-view images as optimized view images; optimize each optimized view image separately: use the SegCLIP guided correlation graph of all other view images except the current optimized view image to optimize the initial language radiation feature in the CLIP-NeRF correlation graph of the current optimized view image, and obtain the optimal language radiation feature of the current optimized view image;

[0022] The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

[0023] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization as described above when executing the computer program.

[0024] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented as described above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention proposes a multi-view Figure 1 The Multi-view ConsistencyGuided 3D Open-vocabulary Segmentation (MCGSeg) method solves the multi-view inconsistency problem caused by CLIP in the above-mentioned prior art by introducing the Geometry Guided Consistent Correction Strategy (GGCCS) and the Self-adaption Guided Consistent Correction Strategy (SGCCS). First, GGCCS uses the pre-trained geometry estimation network and SAM to accurately explore object proposals and unify the object features of images from different perspectives, ensuring the accuracy of semantics reflected by features in images from different perspectives. Secondly, SGCCS fully considers the multi-view inherent in NeRF itself. Figure 1Consistency effectively suppresses the adverse effects of inconsistent knowledge of CLIP. In summary, the present invention significantly improves the accuracy and reliability of 3D segmentation results through innovative strategies and ingenious designs, and has obvious advantages over the prior art. The MCGSeg of the present invention has broad application prospects. In the field of autonomous driving, accurate 3D scene segmentation enables vehicles to better perceive the surrounding environment, identify various objects such as roads, vehicles, pedestrians, etc., and improve the safety and reliability of autonomous driving. In robot navigation, it helps robots to more accurately understand the spatial environment in which they are located, avoid obstacles, and plan the optimal action path, so as to complete tasks more efficiently. In virtual reality and augmented reality, it can create a more realistic and immersive experience for users, making the fusion of virtual objects and real scenes more natural and accurate. In the fields of industrial design and architecture, it can accurately segment and analyze complex 3D models, assisting designers and engineers in optimization and innovation. In the medical field, for example, 3D segmentation of medical images can help doctors more clearly understand the structural relationship between diseased tissue and normal tissue, and improve the accuracy of diagnosis and treatment. In short, the present invention has huge application potential in many fields that require accurate 3D scene understanding and analysis, and is expected to promote the development and progress of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 This is the MCGSeg framework diagram of the present invention.

[0029] Figure 2 This is the bed segmentation result diagram in the example of the present invention.

[0030] Figure 3 This is a diagram of bench segmentation results in an example of the present invention.

[0031] Figure 4 This is the segmentation result diagram of the blue sofa in the example of the present invention.

[0032] Figure 5 This is the lawn segmentation result diagram in the example of the present invention.

[0033] Figure 6 This is the sofa segmentation result diagram in the example of the present invention.

[0034] Figure 7 This is the segmentation result diagram of the blue sofa in the example of the present invention.

[0035] Figure 8 This is a snacks segmentation result diagram in an example of the present invention. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0037] It should be noted that the process equipment or devices not specifically specified in the following embodiments are all conventional equipment or devices in the art.

[0038] It should be noted that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. Moreover, unless otherwise specified, the numbering of each method step is only a convenient tool for identifying each method step, and is not intended to limit the order of arrangement of each method step or to define the scope of the present invention. Changes or adjustments in their relative relationships should also be regarded as the scope of the present invention without substantially changing the technical content.

[0039] In order to alleviate the multi-view inconsistency problem of CLIP and improve the segmentation results, this paper proposes MCGSeg based on NeRF framework, such as Figure 1 . First, considering the geometric correspondence of images in multi-view scenes (i.e., the coordinates of pixels in different view images should be the same), this paper proposes GGCCS to alleviate the multi-view inconsistency problem of CLIP. GGCCS uses pre-trained geometry estimation networks and SAM to explore object proposals and unify feature representations from images of different view angles. After applying GGCCS to CLIP, the obtained features can correctly reflect the semantics of images of different views. Secondly, considering the inherent multi-view Figure 1 The present invention directly utilizes the consistency of GGCCS to develop SGCCS to suppress the influence of inconsistent knowledge of CLIP. By embedding GGCCS and SGCCS into the NeRF-based framework, accurate 3D segmentation results can be produced.

[0040] First, several models adopted by the present invention are introduced.

[0041] 1. NeRF

[0042] NeRF is a basic framework for rendering new perspective images of scenes. Given a 3D coordinate x = (x, y, z) and a 2D viewing direction It accumulates the emitted color at the given coordinates and bulk density To calculate the color of each pixel, NeRF starts from the camera origin Emit light and along the direction Propagation, expressed as r(t)=o+td, where t∈[t n ,t f ] are predefined as the nearest and farthest boundaries. NeRF then samples M points for each ray and applies volume rendering to determine the final color representation for each pixel. This process can be expressed as:

[0043]

[0044] Among them, T i represents the transmission probability, C(r) represents the final color of the pixel rendered by NeRF, and c i and σ i Represent the color and volume density of the i-th sample along the ray, δ i =t i+1 -t i Indicates the distance between adjacent sampling points.

[0045] 2. NeRF-based 3D open vocabulary segmentation basic model

[0046] The method of the present invention is developed based on the 3D open vocabulary segmentation basic model. Therefore, the present invention records it as 3D-OVS and first briefly introduces it here. 3D-OVS consists of two key components, namely NeRF and the basic model. NeRF first reconstructs the 3D scene from multi-view images. Then, based on the reconstructed 3D scene, NeRF combines additional multi-layer perceptrons to generate language radiation features for subsequent segmentation. They can be obtained by the following formula:

[0047]

[0048] in, Indicates language radiation characteristics, Represents the point features sampled at M points along the ray r.

[0049] The base model is used to guide NeRF to complete 3D scene segmentation. There are two base models in traditional 3D-OVS: CLIP and DINO. CLIP focuses on supervising NeRF to understand the semantic relationship between 3D scenes and text descriptions. To this end, it first converts text descriptions and multi-view images into text features and multi-scale visual features. Then, the CLIP correlation graph is generated based on the text features and multi-scale visual features using the cosine similarity measurement method, and the language radiation features are optimized based on the multi-scale visual features. Here, the CLIP correlation graph represents the relationship between CLIP text features and multi-scale visual features. Next, two loss functions are developed and applied. and to the above-mentioned language radiation features and CLIP correlation map to complete the segmentation. In addition, the traditional 3D-OVS introduces DINO to refine the results from the perspective of segmentation boundaries. In this part, another loss function is developed

[0050] 3. Geometry-guided consistency correction strategy

[0051] The GGCCS process can be divided into two steps. The first step is to extract SegCLIP features from multi-view images. The second step is to unify the obtained SegCLIP features by analyzing the geometric correspondence of multi-view images.

[0052] For SegCLIP feature extraction: GGCCS uses both SAM and CLIP to obtain effective feature representations from multi-view images. Here, SAM is used to divide each view image into different multi-level objects, and CLIP is used to map these objects into distinguishable SegCLIP features and convert text descriptions into text features. Preferably, GGCCS first uses SAM to segment the image into multi-level objects M = {M s ,M p ,M w}, where M s ,M p ,M w Represent different numbers of sub-parts, parts, and whole-level objects respectively. Then, GGCCS uses CLIP visual encoder to map multi-level objects into distinguishable SegCLIP features. By simply combining Where D is the dimension (fixed at 512 in the embodiment of the present invention), H and W are the height and width of the image. s is the SegCLIP feature of small-scale objects, F p is the SegCLIP feature of the mesoscale object, F wIt is the SegCLIP feature of large-scale objects. The integration of SAM and CLIP ensures that the content within the image can be fully and hierarchically explored.

[0053] Unification of SegCLIP features: Although SegCLIP features can appropriately present image content, they still face the problem of multi-view inconsistency. To solve this problem, GGCCS introduces image geometry constraints. In other words, GGCCS tends to locate the same object in multi-view images and then enhances the consistency of its SegCLIP features across different viewpoints. To this end, GGCCS applies a dense kernelized feature matching network (DKM network) to the obtained SegCLIP features. The DKM network was chosen because it can identify pixels representing the same object in different viewpoints. In addition, the DKM network also utilizes dense feature matching instead of the common sparse paradigm, which is more suitable for 3D multi-view images. The present invention uses F s Take B multi-view images of a 3D scene as an example to illustrate this step. The present invention uses the DKM network to calculate the geometric correspondence information between the b-th image and other images, as shown below:

[0054]

[0055] where DKM(·) represents the function of the DKM network, and Respectively represent the geometric correspondence set and the geometric correspondence deterministic set for the b-th image. Here, p b←j (·) represents the deformation function that transforms the jth image to the bth image, is the certainty of the deformation operation for each pixel of the b-th image.

[0056] By warping other images to the bth image, GGCCS can transfer the same object from other images to the bth image, thereby locating the same object. In this way, according to the geometric correspondence between images of different views, the SegCLIP features of the same object in images of different views can be aggregated as:

[0057]

[0058] in, represents the SegCLIP feature of the bth aggregation. b←j The value of m is determined by the deformation operation b←j The Heaviside function threshold τ is determined by the dot product ⊙ operation to control the degree of feature aggregation.

[0059] After the above operations, the SegCLIP features can be unified to reduce the impact of multi-view inconsistency. In addition, considering that directly aggregating SegCLIP features from different perspectives may smudge the object features, GGCCS averages the features of each object (by M s SegCLIP features of all pixels of the image are used to smooth the object features, thereby obtaining a SegCLIP-guided image representation. To supervise NeRF. p ,F w After the same operation, the multi-level SegCLIP-guided image representation can be obtained. Then, by adding them together, the final SegCLIP-guided signature can be generated. b=1,2,…,B, these features help 3D-OVS generate accurate 3D language radiation features.

[0060] 4. Adaptive Guidance Consistency Correction Strategy

[0061] So far, SegCLIP-guided features with unified representation have been collected to supervise 3D-OVS. NeRF essentially enforces multi-view consistency by predicting volume density σ based only on coordinate x. In order to suppress the influence of inconsistent knowledge from CLIP, the present invention designs SGCCS. SGCCS focuses on supervising 3D-OVS using reliable hot labels output by 3D-OVS.

[0062] In SGCCS, N images are selected from the multi-view images as optimized view images. In the 3D-OVS learning phase, the language radiation features associated with these optimized view images are further guided by reliable thermal labels. In order to obtain reliable thermal labels to supervise the language radiation features of these optimized view images, SGCCS first uses the text features of all views except the current multi-view image and the SegCLIP guided feature G b Generate a SegCLIP-guided correlation map using the cosine similarity metric to obtain the hot labels for all view images except the current multi-view image. Then, using formula (5), these hot labels are transferred to the current optimized view image, and the most credible hot labels are voted for the current optimized view image. (i.e., the reliable hot labels of the current optimized view image), and N is the number of optimized view images.

[0063]

[0064] For the current view image, a CLIP-NeRF guided correlation map is generated by text features and primary language radiation features. Then, SGCCS uses the cross entropy loss CE(·) to optimize the CLIP-NeRF guided correlation map S of the current optimized view image under the supervision of the corresponding reliable hot label R. The process can be expressed as:

[0065]

[0066] Among them, Z represents the text feature , h and w represent the height and width of the CLIP-NeRF guided correlation map. Here, the values ​​of h and w are equal to and

[0067] 5. Overall training loss

[0068] The training loss function of the present invention is based on the traditional 3D-OVS. However, in order to align it with MCGSeg, two modifications are made. First, since the method of the present invention uses SAM for object extraction, the loss function related to the object boundary is is removed. Secondly, the scale selection of CLIP features is eliminated to ensure the consistency of object features. Finally, the overall loss function of the present invention is:

[0069]

[0070] in, Focus on utilizing the final SegCLIP-guided feature G i To guide NeRF to understand the semantic information of the 3D scene, the specific formula is as follows:

[0071]

[0072] Among them, f b,i G represents the point features sampled at M points along the ray r in the bth image; b,i Indicates the SegCLIP-guided features at the corresponding positions.

[0073] First, the SegCLIP-guided feature G b The cosine similarity measurement with the text features is used to obtain the final SegCLIP guided correlation map To help NeRF communicate with open vocabulary text descriptions, the specific formula is as follows:

[0074]

[0075] in, represents the normalized SegCLIP-guided similarity distribution, P(b,i,z) represents the segmentation probability distribution, represents the average of the above two distributions.

[0076] Introduced by SGCCS, as shown in formula (6), to suppress the influence of inconsistent knowledge of CLIP.

[0077] Based on the above content, the present invention provides a three-dimensional open vocabulary segmentation method based on consistency regularization, which specifically includes:

[0078] (1) NeRF is used to reconstruct the 3D scene from multi-view images, and an initial language radiation feature is generated for each view image based on the reconstructed 3D scene.

[0079] (2) Use SAM and CLIP to extract SegCLIP features of objects in each view image from multi-view images.

[0080] SAM is used to divide the image into different objects, and CLIP is used to map these objects into distinguishable SegCLIP features.

[0081] SAM is used to segment each image in the multi-view image into different objects. Then, the CLIP visual encoder is used to map each object into a distinguishable SegCLIP feature to obtain the SegCLIP feature of each image.

[0082] In order to improve the SegCLIP feature extraction effect, the image can be segmented into multi-level objects. Specifically, SAM is used to segment each image in the multi-view image into multi-level objects M = {M s ,M p ,M w}, where M s ,M p ,M w They represent different numbers of sub-parts, parts, and whole-level objects respectively. Then, the CLIP visual encoder is used to map the multi-level objects into distinguishable SegCLIP features. By simply combining, the SegCLIP features of each image are generated. Where D is the dimension (fixed at 512 in the embodiment of the present invention), and H and W are the height and width of the image.

[0083] (3) Unification of SegCLIP features

[0084] The geometric correspondence between each view image and other view images is calculated. According to the geometric correspondence, the SegCLIP features of the same object in different view images are aggregated to obtain the SegCLIP guided features.

[0085] As a preferred method, a DKM network is used to calculate the geometric correspondence between each perspective image and other perspective images.

[0086] In order to avoid the problem that directly aggregating SegCLIP features from different perspectives may smear the object features, the SegCLIP features of all pixels of each object in the image can be averaged first, and then the SegCLIP features of the same object in images from different perspectives can be aggregated to obtain the SegCLIP guided features.

[0087] (4) Optimization of language radiation characteristics

[0088] After obtaining the SegCLIP-guided features, The loss function optimizes the language radiation characteristics of the optimized perspective image; at the same time, with the help of L RDA Implementing the final SegCLIP-guided correlation map R I (b,i,z) helps NeRF communicate with open vocabulary text descriptions.

[0089] In addition, according to the text features and SegCLIP guided features of each view image, the cosine similarity measurement method is used to generate the SegCLIP guided correlation map of each view image.

[0090] N images are selected from the multi-view images as optimized view images, and each optimized view image is optimized separately: according to the primary language radiation features and text features of the current optimized view image, the CLIP-NeRF guided correlation map of the current optimized view image is generated by the cosine similarity measurement method, and the SegCLIP guided correlation map of all other view images except the current optimized view image is used to optimize the CLIP-NeRF correlation map of the current optimized view image to obtain the optimal language radiation features of the current optimized view image.

[0091] Specifically, each optimized perspective image is optimized separately: according to the primary language radiation features and text features of the current optimized perspective image, the CLIP-NeRF guided correlation map of the current optimized perspective image is generated by the cosine similarity measurement method; according to the SegCLIP guided correlation map of all other perspective images except the current optimized perspective image, the hot label of the corresponding perspective image is converted, and with the help of the geometric correspondence between other perspective images and the current optimized perspective image obtained by the DKM network, all the obtained hot labels are transferred to the current optimized perspective image, and all the hot labels are voted to select the most credible hot label for the current optimized perspective image, so that the current optimized perspective image obtains a reliable hot label; cross entropy loss is used to Under the supervision of the reliable thermal labels, the initial language radiation features in the CLIP-NeRF guided correlation graph of the current optimized view image are optimized to obtain the optimal language radiation features of the current optimized view image;

[0092] After the above optimization operation, each optimized perspective image obtains an optimal language radiation feature.

[0093] (5)Split

[0094] For each optimized view image, similarity measurement is performed on its optimal language radiation features and text features, and the result with the highest score is the predicted segmentation label, which is the segmentation result of the corresponding optimized view image.

[0095] Simulation Example

[0096] The present invention has carried out simulation experiments on multiple images, and the obtained results are shown in Table 1 and Figure 2-8 As shown, it can be seen that the method of the present invention can achieve accurate image segmentation.

[0097] Table 1 Image segmentation results of the simulation example of the present invention

[0098]

[0099] The following are device embodiments of the present invention, which can be used to perform method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0100] In yet another embodiment of the present invention, a three-dimensional open vocabulary segmentation system based on consistency regularization is provided, which can be used to implement the above-mentioned three-dimensional open vocabulary segmentation method based on consistency regularization. Specifically, the three-dimensional open vocabulary segmentation system based on consistency regularization includes:

[0101] An initialization module is used to reconstruct a 3D scene based on multi-view images using NeRF, and generate an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0102] A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features;

[0103] The optimization module is used to generate SegCLIP guided correlation graphs of all view images by using the cosine similarity measurement method according to the text features and SegCLIP guided features of each view image; select multiple images from the multi-view images as optimized view images; optimize each optimized view image separately: use the primary language radiation features and text features of the current optimized view image to generate the CLIP-NeRF guided correlation graph of the current optimized view image by using the cosine similarity measurement, and use the SegCLIP guided correlation graphs of all other view images except the current optimized view image to optimize the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image to obtain the optimal language radiation features of the current optimized view image;

[0104] The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

[0105] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the three-dimensional open vocabulary segmentation method based on consistency regularization.

[0106] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the three-dimensional open vocabulary segmentation method based on consistency regularization in the above embodiment.

[0107] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional open vocabulary segmentation method based on consistency regularization, characterized in that: include: S1, using NeRF to reconstruct the 3D scene from multi-view images, and generating an initial language radiation feature for each view image based on the reconstructed 3D scene; S2, using SAM and CLIP to extract the SegCLIP features of the objects in each view image, calculating the geometric correspondence between each view image and other view images, and aggregating the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain the SegCLIP guided features; S3, according to the text features and SegCLIP guided features of each view image, the SegCLIP guided correlation map of each view image is generated by the cosine similarity measurement method; multiple images are selected from the multi-view images as optimized view images; each optimized view image is optimized separately: according to the primary language radiation features and text features of the current optimized view image, the CLIP-NeRF guided correlation map of the current optimized view image is generated by the cosine similarity measurement method, and the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image are optimized by using the SegCLIP guided correlation maps of all other view images except the current optimized view image, so as to obtain the optimal language radiation features of the current optimized view image; S4, for each optimized view image, the similarity between the optimal language radiation feature and the text feature is measured, and the result with the highest score is taken as the segmentation result of the corresponding optimized view image.

2. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that: In S2, SAM and CLIP are used to extract the SegCLIP features of the objects in each view image. Specifically, each view image is segmented into different objects using SAM; then, the CLIP visual encoder is used to map the SegCLIP features of each object to obtain the SegCLIP features of the objects in each view image.

3. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 2, characterized in that: SegCLIP features of objects in each view image are extracted using SAM and CLIP. Specifically, each view image is segmented into different multi-level objects using SAM. Then, each level of objects is mapped into SegCLIP features using the CLIP visual encoder. The SegCLIP features of objects at each level are combined to obtain the SegCLIP features of objects in each view image.

4. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that: In S2, the geometric correspondence between each viewing angle image and other viewing angle images is calculated, specifically, by using a DKM network to calculate the geometric correspondence between each viewing angle image and other viewing angle images.

5. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that: In S2, the SegCLIP features of the same object in the images of different viewpoints are aggregated according to the geometric correspondence to obtain the SegCLIP guiding features. Specifically, the SegCLIP features of the same object in the images of different viewpoints are aggregated by obtaining the geometric correspondence of the images of each viewpoint to obtain the aggregated SegCLIP features; then the SegCLIP features of all pixels of each object are averaged to obtain the SegCLIP guiding features.

6. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that: In S3, each optimized perspective image is optimized separately, specifically: according to the primary language radiation characteristics and text characteristics of the current optimized perspective image, a CLIP-NeRF guided correlation map of the current optimized perspective image is generated by a cosine similarity measurement method; according to the SegCLIP guided correlation map of all other perspective images except the current optimized perspective image, a hot label of the corresponding perspective image is obtained, and according to the geometric correspondence between the current optimized perspective image and other perspective images, all the obtained hot labels are transferred to the current optimized perspective image, and all the hot labels are voted to select a most credible hot label for the current optimized perspective image, so as to obtain a reliable hot label of the current optimized perspective image; Under the supervision of the reliable thermal labels, the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized viewing angle image are optimized to obtain the optimal language radiation features of the current optimized viewing angle image.

7. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 6, characterized in that: The initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image are optimized under the supervision of the reliable thermal labels. Specifically, the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image are optimized using cross entropy loss under the supervision of the reliable thermal labels.

8. A three-dimensional open vocabulary segmentation system based on consistency regularization, characterized in that: include: An initialization module is used to reconstruct a 3D scene based on multi-view images using NeRF, and generate an initial language radiation feature for each view image based on the reconstructed 3D scene; A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features; The optimization module is used to generate SegCLIP guided correlation graphs of all view images by using the cosine similarity measurement method according to the text features and SegCLIP guided features of each view image; select multiple images from the multi-view images as optimized view images; optimize each optimized view image separately: generate the CLIP-NeRF guided correlation graph of the current optimized view image by using the cosine similarity measurement method according to the primary language radiation features and text features of the current optimized view image; optimize the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image by using the SegCLIP guided correlation graphs of all other view images except the current optimized view image, and obtain the optimal language radiation features of the current optimized view image; The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented as described in any one of claims 1 to 7.

Images