A three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization

Abstract

The present invention provides a three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization. The present invention introduces a geometrically guided consistent correction strategy and an adaptively guided consistent correction strategy. First, the geometrically guided consistent correction strategy, with the help of a pre-trained geometric estimation network and SAM, can accurately explore object proposals and unify the object features of images with different perspectives, ensuring the accuracy of semantic reflection of features in images with different perspectives. Secondly, the adaptively guided consistent correction strategy fully considers the inherent multi-view consistency of NeRF itself, and effectively suppresses the adverse effects of inconsistent knowledge of CLIP. In summary, the present invention significantly improves the accuracy and reliability of 3D segmentation results through innovative strategies and ingenious designs, and has obvious advantages over the existing technology.

Description

Technical Field

[0001] The present invention belongs to the field of 3D open vocabulary segmentation and relates to a three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization. Background Art

[0002] With the rapid development of computer vision, understanding 3D scenes has become an important research topic. Among many applications, 3D open vocabulary segmentation is one of the key tasks. Its goal is to segment the 3D scene into semantic regions based on a given open vocabulary text description. This helps to fully understand the composition and structure of the 3D scene, and can effectively connect 3D scenes with natural language. It has broad application prospects in fields such as autonomous driving and robot navigation. In recent years, many methods for 3D open vocabulary segmentation have emerged. Among them, Neural Radiance Field (NeRF) can be used to reconstruct 3D scenes from multi-view images; the pre-trained base model is used to guide NeRF to perform 3D scene segmentation based on text descriptions.

[0003] A common base model is the Comparative Language-Image Pretraining (CLIP) model. CLIP is combined with NeRF, which is used to reconstruct 3D scenes from multi-view images. CLIP's visual and text encoders are used to map images to CLIP image features. This allows for the establishment of a semantic relationship between the 3D scene and the text description at the image level during NeRF's 3D reconstruction process, thereby achieving segmentation. To further improve segmentation results, DINO and the Segment Anything Model (SAM) are also incorporated, leveraging the object-level cues they generate to optimize segmentation boundaries and improve accuracy. These methods have achieved impressive segmentation results thanks to NeRF's powerful 3D reconstruction capabilities and the rich prior knowledge hidden in the pre-trained base models.

[0004] However, existing solutions combining CLIP with NeRF suffer from multi-view inconsistency caused by CLIP. Specifically, within a given set of multi-view images, the same "gray sofa" can be classified very differently in different views, potentially being mistakenly identified as a "Gundam" or an "Xbox Wireless Controller." This clearly reveals a serious flaw in CLIP: when processing multi-view images, CLIP is unable to extract consistent feature representations for the same object from different perspectives. This inconsistency negatively impacts the radiating neural field, significantly reducing the quality of the resulting segmentation results. Summary of the Invention

[0005] In order to solve the multi-view inconsistency problem caused by CLIP in the above-mentioned prior art, the present invention provides a three-dimensional open vocabulary segmentation method, system, device and storage medium based on consistency regularization.

[0006] The present invention is achieved through the following technical solutions:

[0007] A three-dimensional open vocabulary segmentation method based on consistency regularization, comprising:

[0008] S1, using NeRF to reconstruct the 3D scene from multi-view images, and generating an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0009] S2, using SAM and CLIP to extract the SegCLIP features of the objects in each view image, calculating the geometric correspondence between each view image and other view images, and aggregating the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain the SegCLIP guided features;

[0010] S3, based on the text features and SegCLIP guided features of each view image, a cosine similarity measurement method is used to generate a SegCLIP guided correlation map of each view image; multiple images are selected from the multi-view images as optimized view images; each optimized view image is optimized separately: the initial language radiation feature in the CLIP-NeRF correlation map of the current optimized view image is optimized using the SegCLIP guided correlation maps of all other view images except the current optimized view image, to obtain the optimal language radiation feature of the current optimized view image;

[0011] S4, for each optimized view image, the similarity between the optimal language radiation feature and the text feature is measured, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

[0012] Preferably, in S2, SAM and CLIP are used to extract the SegCLIP features of the objects in each perspective image, specifically: each perspective image is divided into different objects using SAM; then, the CLIP visual encoder is used to map the SegCLIP features of each object separately to obtain the SegCLIP features of the objects in each perspective image.

[0013] Furthermore, SegCLIP features of objects in each view image are extracted using SAM and CLIP. Specifically, each view image is segmented into different multi-level objects using SAM. Then, each level of objects is mapped into SegCLIP features using the CLIP visual encoder. The SegCLIP features of objects at each level are combined to obtain the SegCLIP features of objects in each view image.

[0014] Preferably, in S2, the calculating of the geometric correspondence between each perspective image and other perspective images is specifically: calculating the geometric correspondence between each perspective image and other perspective images using a DKM network.

[0015] Preferably, in S2, the SegCLIP features of the same object in images of different view angles are aggregated according to the geometric correspondence to obtain SegCLIP guiding features, specifically: the SegCLIP features of the same object in images of different view angles are aggregated by obtaining the geometric correspondence of the images of each view angle to obtain aggregated SegCLIP features; then the SegCLIP features of all pixels of each object are averaged to obtain the SegCLIP guiding features.

[0016] Preferably, in S3, each optimized perspective image is optimized separately, specifically: according to the SegCLIP guided correlation graph of all other perspective images except the current optimized perspective image, the thermal label of the corresponding perspective image is obtained, all the obtained thermal labels are transferred to the current optimized perspective image according to the geometric correspondence between the current optimized perspective image and other perspective images, and all the thermal labels are voted to select a most credible thermal label for the current optimized perspective image to obtain a reliable thermal label of the current optimized perspective image; under the supervision of the reliable thermal label, the initial language radiation feature in the CLIP-NeRF correlation graph of the current optimized perspective image is optimized to obtain the optimal language radiation feature of the current optimized perspective image.

[0017] Furthermore, the CLIP-NeRF correlation map of the current optimized view map is optimized under the supervision of the reliable hot labels, specifically: the CLIP-NeRF correlation map of the current optimized view map is optimized under the supervision of the reliable hot labels using cross entropy loss.

[0018] The present invention also provides a three-dimensional open vocabulary segmentation system based on consistency regularization, comprising:

[0019] An initialization module is used to reconstruct a 3D scene from multi-view images using NeRF, and to generate an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0020] A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features;

[0021] The optimization module is used to generate SegCLIP-guided correlation maps of all view images using the cosine similarity measurement method based on the text features and SegCLIP-guided features of each view image; select multiple images from the multi-view images as optimized view images; and optimize each optimized view image separately: using the SegCLIP-guided correlation maps of all other view images except the current optimized view image, the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image are optimized to obtain the optimal language radiation features of the current optimized view image;

[0022] The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

[0023] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization when executing the computer program.

[0024] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention proposes a multi-view Figure 1 The Multi-view ConsistencyGuided 3D Open-vocabulary Segmentation (MCGSeg) method solves the multi-view inconsistency problem caused by CLIP in the above-mentioned existing technology by introducing the Geometry Guided Consistent Correction Strategy (GGCCS) and the Self-adaption Guided Consistent Correction Strategy (SGCCS). First, GGCCS uses the pre-trained geometry estimation network and SAM to accurately explore object proposals and unify the object features of images from different perspectives, ensuring the accuracy of semantic reflection of features under different perspectives. Secondly, SGCCS fully considers the multi-view inherent in NeRF itself. Figure 1Consistency effectively suppresses the adverse effects of inconsistent CLIP knowledge. In summary, the present invention significantly improves the accuracy and reliability of 3D segmentation results through innovative strategies and ingenious designs, offering significant advantages over existing technologies. The present invention, MCGSeg, has broad application prospects. In the field of autonomous driving, accurate 3D scene segmentation enables vehicles to better perceive their surroundings and identify various objects such as roads, vehicles, and pedestrians, thereby improving the safety and reliability of autonomous driving. In robotic navigation, it helps robots more accurately understand their spatial environment, avoid obstacles, and plan optimal paths, thereby completing tasks more efficiently. In virtual reality and augmented reality, it can create a more realistic and immersive experience for users, making the integration of virtual objects and real scenes more natural and accurate. In industrial design and architecture, it can accurately segment and analyze complex 3D models, assisting designers and engineers in optimization and innovation. In the medical field, for example, 3D segmentation of medical images can help doctors more clearly understand the structural relationship between diseased and normal tissues, improving the accuracy of diagnosis and treatment. In summary, the present invention has enormous application potential in numerous fields requiring precise 3D scene understanding and analysis, and is expected to promote the development and advancement of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 This is the MCGSeg framework diagram of the present invention.

[0029] Figure 2 This is the bed segmentation result diagram in the example of the present invention.

[0030] Figure 3 This is the bench segmentation result diagram in the example of the present invention.

[0031] Figure 4 This is the blue sofa segmentation result diagram in the example of the present invention.

[0032] Figure 5 This is the lawn segmentation result diagram in the example of the present invention.

[0033] Figure 6 This is the sofa segmentation result diagram in the example of the present invention.

[0034] Figure 7 This is the blue sofa segmentation result diagram in the example of the present invention.

[0035] Figure 8 This is the snacks segmentation result diagram in the example of the present invention. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention.

[0037] It should be noted that the process equipment or devices not specifically specified in the following embodiments are all conventional equipment or devices in the art.

[0038] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Furthermore, unless otherwise specified, the numbering of each method step is merely a convenient tool for identifying each method step, and is not intended to limit the order of arrangement of each method step or to define the scope of the invention. Changes or adjustments to their relative relationships, without substantially changing the technical content, should also be considered within the scope of the invention.

[0039] In order to alleviate the multi-view inconsistency problem of CLIP and improve the segmentation results, this paper proposes MCGSeg based on NeRF framework, such as Figure 1 . First, considering the geometric correspondence of images in multi-view scenes (i.e., the coordinates of pixels in different view images should be the same), this paper proposes GGCCS to alleviate the multi-view inconsistency problem of CLIP. GGCCS uses pre-trained geometry estimation networks and SAM to explore object proposals and unify feature representations from images of different view angles. After applying GGCCS to CLIP, the obtained features can correctly reflect the semantics of images of different views. Secondly, considering the inherent multi-view Figure 1 The present invention directly utilizes the consistency of GGCCS to develop SGCCS to suppress the influence of inconsistent knowledge in CLIP. By embedding GGCCS and SGCCS into a NeRF-based framework, accurate 3D segmentation results can be produced.

[0040] First, several models adopted by the present invention are introduced.

[0041] 1. NeRF

[0042] NeRF is a basic framework for rendering new perspective images of scenes. Given a 3D coordinate x = (x, y, z) and a 2D viewing direction It accumulates the emitted color for a given coordinate and bulk density To calculate the color of each pixel, NeRF starts from the camera origin Emit light and along the direction Propagation, expressed as r(t) = o + td, where t∈[t n ,t f ] are predefined as the nearest and farthest boundaries. NeRF then samples M points for each ray and applies volume rendering to determine the final color representation of each pixel. This process can be expressed as:

[0043]

[0044] Among them, T i represents the transmission probability, C(r) represents the final color of the pixel rendered by NeRF, and c i and σ i Represent the color and volume density of the i-th sample along the ray, δ i =t i+1 -t i Indicates the distance between adjacent sampling points.

[0045] 2. NeRF-based 3D open vocabulary segmentation basic model

[0046] Our method is developed based on a basic model for 3D open vocabulary segmentation. Therefore, we refer to it as 3D-OVS and briefly introduce it here. 3D-OVS consists of two key components: NeRF and the basic model. NeRF first reconstructs the 3D scene from multi-view images. Then, based on the reconstructed 3D scene, NeRF is combined with an additional multi-layer perceptron to generate speech radiometric features for subsequent segmentation. These can be obtained using the following formula:

[0047]

[0048] in, Indicates language radiation characteristics, Represents the point features sampled at M points along the ray r.

[0049] The base model is used to guide NeRF to complete 3D scene segmentation. There are two base models in traditional 3D-OVS: CLIP and DINO. CLIP focuses on supervising NeRF to understand the semantic relationship between 3D scenes and text descriptions. To this end, it first converts text descriptions and multi-view images into text features and multi-scale visual features. Then, the cosine similarity measurement method is used to generate the CLIP correlation graph based on the text features and multi-scale visual features, and the language radiation features are optimized based on the multi-scale visual features. Here, the CLIP correlation graph represents the relationship between CLIP text features and multi-scale visual features. Next, two loss functions are developed and applied. and To complete the segmentation, we introduce DINO into the traditional 3D-OVS to refine the results from the perspective of segmentation boundaries. In this part, we develop another loss function

[0050] 3. Geometry-guided consistency correction strategy

[0051] The GGCCS pipeline can be divided into two steps. The first step is to extract SegCLIP features from multi-view images. The second step is to unify the obtained SegCLIP features by analyzing the geometric correspondences of multi-view images.

[0052] For the extraction of SegCLIP features: GGCCS uses SAM and CLIP to obtain effective feature representations from multi-view images. Here, SAM is used to divide each view image into different multi-level objects, and CLIP is used to map these objects into distinguishable SegCLIP features and convert text descriptions into text features. Preferably, GGCCS first uses SAM to segment the image into multi-level objects M = {M s ,M p ,M w}, where M s ,M p ,M w Representing different numbers of sub-parts, parts and whole-level objects respectively. Then, GGCCS uses CLIP visual encoder to map multi-level objects into distinguishable SegCLIP features. By simple combination, SegCLIP features of each view image can be generated. Where D is the dimension (fixed to 512 in the embodiment of the present invention), H and W are the height and width of the image. s is the SegCLIP feature of small-scale targets, F p is the SegCLIP feature of the mesoscale object, F wIt is the SegCLIP feature of large-scale objects. The integration of SAM and CLIP ensures that the content within the image can be fully and hierarchically explored.

[0053] Unification of SegCLIP features: Although SegCLIP features can appropriately present image content, they still face the problem of multi-view inconsistency. To solve this problem, GGCCS introduces image geometry constraints. In other words, GGCCS tends to locate the same object in multi-view images and then enhance the consistency of its SegCLIP features across different viewpoints. To this end, GGCCS applies a dense kernelized feature matching network (DKM network) to the obtained SegCLIP features. The DKM network was chosen because it can identify pixels representing the same object in different viewpoints. In addition, the DKM network also utilizes dense feature matching instead of the common sparse paradigm, which is more suitable for 3D multi-view images. The present invention uses F s Let's take this step as an example. For B multi-view images of a 3D scene The present invention uses the DKM network to calculate the geometric correspondence information between the b-th image and other images, as shown below:

[0054]

[0055] where DKM(·) represents the function of the DKM network, and Represent the geometric correspondence set and the geometric correspondence deterministic set for the b-th image. Here, p b←j (·) represents the deformation function that transforms the j-th image to the b-th image, is the certainty of the deformation operation for each b-th image pixel.

[0056] By warping other images to the bth image, GGCCS can transfer the same object from other images to the bth image, thereby locating the same object. In this way, according to the geometric correspondence between images of different views, the SegCLIP features of the same object in images of different views can be aggregated as:

[0057]

[0058] in, represents the SegCLIP feature of the b-th aggregation. b←j The value of m is determined by the deformation operation b←j The Heaviside function threshold τ is determined, and the degree of feature aggregation is controlled by the dot product ⊙ operation.

[0059] After the above operations, the SegCLIP features can be unified to reduce the impact of multi-view inconsistency. In addition, considering that directly aggregating SegCLIP features from different perspectives may smudge the object features, GGCCS averages the SegCLIP features of each object (by M s SegCLIP features of all pixels of the image are used to smooth the object features, thereby obtaining a SegCLIP-guided image representation. To supervise NeRF. p ,F w After performing the same operation, we can obtain the multi-level SegCLIP guided image representation Then, by adding them together, the final SegCLIP-guided signature can be generated. b=1,2,…,B, these features help 3D-OVS generate accurate 3D language radiation features.

[0060] 4. Adaptive Guidance Consistency Correction Strategy

[0061] So far, SegCLIP-guided features with unified representations have been collected to supervise 3D-OVS. NeRF essentially enforces multi-view consistency by predicting the volume density σ based only on coordinate x. To suppress the influence of inconsistent CLIP knowledge, we designed SGCCS. SGCCS focuses on leveraging reliable thermal labels output by 3D-OVS to supervise 3D-OVS.

[0062] In SGCCS, N images are selected from the multi-view images as optimized view images. In the 3D-OVS learning phase, the language radiometric features associated with these optimized view images are further guided by reliable thermal labels. In order to obtain reliable thermal labels to supervise the language radiometric features of these optimized view images, SGCCS first uses the text features of all views except the current multi-view image and the SegCLIP guided feature G b Generate SegCLIP guided correlation graph through cosine similarity metric to obtain hot labels of all view images except the current multi-view image Then, using formula (5), these hot labels are transferred to the current optimized view image, and the most credible hot labels are voted for the current optimized view image. (i.e., the reliable hot labels of the current optimized view image), and N is the number of optimized view images.

[0063]

[0064] For the current view image, a CLIP-NeRF guided correlation map is generated by text features and primary language radiation features. Then, SGCCS uses the cross entropy loss CE(·) to optimize the CLIP-NeRF guided correlation map S of the current optimized view image under the supervision of the corresponding reliable hot label R. The process can be expressed as:

[0065]

[0066] Among them, Z represents the text feature , h and w represent the height and width of the CLIP-NeRF guided correlation map. Here, the values ​​of h and w are equal to and

[0067] 5. Overall training loss

[0068] The training loss function of the present invention is based on the traditional 3D-OVS. However, in order to align it with MCGSeg, two modifications are made. First, since the method of the present invention uses SAM for object extraction, the loss function related to the object boundary is is removed. Secondly, the scale selection of CLIP features is eliminated to ensure the consistency of object features. Finally, the overall loss function of the present invention is:

[0069]

[0070] in, Focus on utilizing the final SegCLIP-guided feature G i To guide NeRF to understand the semantic information of the 3D scene, the specific formula is as follows:

[0071]

[0072] Among them, f b,i G represents the point features sampled at M points along the ray r in the b-th image; b,i Indicates the SegCLIP-guided features at the corresponding positions.

[0073] First, the SegCLIP guided feature G b Perform cosine similarity measurement with text features to obtain the final SegCLIP guided correlation graph To help NeRF communicate with open vocabulary text descriptions, the specific formula is as follows:

[0074]

[0075] in, represents the normalized SegCLIP-guided similarity distribution, P(b,i,z) represents the segmentation probability distribution, represents the mean of the above two distributions.

[0076] It is introduced by SGCCS as shown in formula (6) to suppress the influence of inconsistent knowledge of CLIP.

[0077] Based on the above content, the present invention provides a three-dimensional open vocabulary segmentation method based on consistency regularization, which specifically includes:

[0078] (1) NeRF is used to reconstruct the 3D scene from multi-view images, and an initial language radiation feature is generated for each view image based on the reconstructed 3D scene.

[0079] (2) Use SAM and CLIP to extract SegCLIP features of objects in each view image from multi-view images.

[0080] SAM is used to divide the image into different objects, and CLIP is used to map these objects into distinguishable SegCLIP features.

[0081] SAM is used to segment each image in the multi-view image into different objects; then, the CLIP visual encoder is used to map each object into a distinguishable SegCLIP feature to obtain the SegCLIP feature of each image.

[0082] In order to improve the SegCLIP feature extraction effect, the image can be segmented into multi-level objects. Specifically, each image in the multi-view image is segmented into multi-level objects M = {M s ,M p ,M w}, where M s ,M p ,M w Represent different numbers of sub-parts, parts and whole-level objects respectively. Then, CLIP visual encoder is used to map multi-level objects into distinguishable SegCLIP features. By simple combination, SegCLIP features of each image are generated. Where D is the dimension (fixed at 512 in the embodiment of the present invention), and H and W are the height and width of the image.

[0083] (3) Unification of SegCLIP features

[0084] The geometric correspondence between each view image and other view images is calculated. According to the geometric correspondence, the SegCLIP features of the same object in different view images are aggregated to obtain the SegCLIP guided features.

[0085] As a preferred approach, a DKM network is used to calculate the geometric correspondence between each perspective image and other perspective images.

[0086] In order to avoid the problem that directly aggregating SegCLIP features from different perspectives may smear the object features, the SegCLIP features of all pixels of each object in the image can be averaged first, and then the SegCLIP features of the same object in images from different perspectives can be aggregated to obtain the SegCLIP guided features.

[0087] (4) Optimization of language radiation characteristics

[0088] After obtaining the SegCLIP guidance features, The loss function optimizes the language radiation characteristics of the optimized perspective image; at the same time, with the help of L RDA Implementing the final SegCLIP-guided correlation map R I (b,i,z) helps NeRF communicate with open vocabulary text descriptions.

[0089] In addition, based on the text features and SegCLIP guided features of each view image, the cosine similarity measurement method is used to generate the SegCLIP guided correlation map of each view image.

[0090] N images are selected from the multi-view images as optimized view images, and each optimized view image is optimized separately: according to the primary language radiation characteristics and text characteristics of the current optimized view image, the CLIP-NeRF guided correlation map of the current optimized view image is generated by the cosine similarity measurement method, and the SegCLIP guided correlation map of all other view images except the current optimized view image is used to optimize the CLIP-NeRF correlation map of the current optimized view image to obtain the optimal language radiation characteristics of the current optimized view image.

[0091] Specifically, each optimized view image is optimized separately: according to the primary language radiation features and text features of the current optimized view image, the CLIP-NeRF guided correlation graph of the current optimized view image is generated by the cosine similarity measurement method; according to the SegCLIP guided correlation graphs of all other view images except the current optimized view image, the hot labels of the corresponding view images are converted, and with the help of the geometric correspondence between other view images and the current optimized view image obtained by the DKM network, all the obtained hot labels are transferred to the current optimized view image, and all the hot labels are voted to select the most credible hot label for the current optimized view image, so that the current optimized view image obtains a reliable hot label; cross entropy loss is used to Optimizing the initial language radiation features in the CLIP-NeRF guided correlation graph of the current optimized view image under the supervision of the reliable thermal labels to obtain the optimal language radiation features of the current optimized view image;

[0092] After the above optimization operation, each optimized perspective image obtains an optimal language radiation feature.

[0093] (5)Split

[0094] For each optimized view image, similarity measurement is performed on its optimal language radiation features and text features, and the result with the highest score is the predicted segmentation label, which is the segmentation result of the corresponding optimized view image.

[0095] Simulation Example

[0096] The present invention has carried out simulation experiments on multiple images, and the results obtained are shown in Table 1 and Figure 2-8 As shown, it can be seen that the method of the present invention can achieve accurate image segmentation.

[0097] Table 1 Image segmentation results of the simulation example of the present invention

[0098]

[0099] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0100] In another embodiment of the present invention, a three-dimensional open vocabulary segmentation system based on consistency regularization is provided, which can be used to implement the above-mentioned three-dimensional open vocabulary segmentation method based on consistency regularization. Specifically, the three-dimensional open vocabulary segmentation system based on consistency regularization includes:

[0101] An initialization module is used to reconstruct a 3D scene from multi-view images using NeRF, and to generate an initial language radiation feature for each view image based on the reconstructed 3D scene;

[0102] A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features;

[0103] An optimization module is used to generate SegCLIP-guided correlation maps of all view images using a cosine similarity measurement method based on the text features and SegCLIP-guided features of each view image; select multiple images from the multi-view images as optimized view images; and optimize each optimized view image separately: using the primary language radiation features and text features of the current optimized view image, a CLIP-NeRF-guided correlation map of the current optimized view image is generated using a cosine similarity measurement; and using the SegCLIP-guided correlation maps of all other view images except the current optimized view image to optimize the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image to obtain the optimal language radiation features of the current optimized view image;

[0104] The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

[0105] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the three-dimensional open vocabulary segmentation method based on consistency regularization.

[0106] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the three-dimensional open vocabulary segmentation method based on consistency regularization in the above embodiment.

[0107] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional open vocabulary segmentation method based on consistency regularization, characterized in that include: S1, using NeRF to reconstruct the 3D scene from multi-view images, and generating an initial language radiation feature for each view image based on the reconstructed 3D scene; S2, using SAM and CLIP to extract the SegCLIP features of the objects in each view image, calculating the geometric correspondence between each view image and other view images, and aggregating the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain the SegCLIP guided features; S3, based on the text features and SegCLIP guided features of each view image, a cosine similarity measurement method is used to generate a SegCLIP guided correlation map of each view image; multiple images are selected from the multi-view images as optimized view images; each optimized view image is optimized separately: based on the primary language radiation features and text features of the current optimized view image, a CLIP-NeRF guided correlation map of the current optimized view image is generated by the cosine similarity measurement method, and the SegCLIP guided correlation maps of all other view images except the current optimized view image are used to optimize the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image to obtain the optimal language radiation features of the current optimized view image; S4, for each optimized view image, the similarity between the optimal language radiation feature and the text feature is measured, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

2. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that In S2, SAM and CLIP are used to extract the SegCLIP features of objects in each view image. Specifically, SAM is used to segment each view image into different objects; then, the CLIP visual encoder is used to map the SegCLIP features of each object separately to obtain the SegCLIP features of the objects in each view image.

3. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 2, characterized in that SegCLIP features of objects in each view image are extracted using SAM and CLIP. Specifically, each view image is segmented into different multi-level objects using SAM. Then, each level of objects is mapped into SegCLIP features using the CLIP visual encoder. The SegCLIP features of objects at each level are combined to obtain the SegCLIP features of objects in each view image.

4. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that In S2, the geometric correspondence between each perspective image and other perspective images is calculated by using a DKM network to calculate the geometric correspondence between each perspective image and other perspective images.

5. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that In S2, the SegCLIP features of the same object in images of different viewpoints are aggregated according to the geometric correspondence to obtain SegCLIP guidance features. Specifically, the SegCLIP features of the same object in images of different viewpoints are aggregated according to the obtained geometric correspondence of the images of each viewpoint to obtain aggregated SegCLIP features; then the SegCLIP features of all pixels of each object are averaged to obtain SegCLIP guidance features.

6. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 1, characterized in that In S3, each optimized view image is optimized separately, specifically: according to the primary language radiation characteristics and text characteristics of the current optimized view image, a CLIP-NeRF guided correlation graph of the current optimized view image is generated by a cosine similarity measurement method; according to the SegCLIP guided correlation graphs of all other view images except the current optimized view image, a hot label of the corresponding view image is obtained, and according to the geometric correspondence between the current optimized view image and the other view images, all the obtained hot labels are transferred to the current optimized view image, and all the hot labels are voted to select a most credible hot label for the current optimized view image, thereby obtaining a reliable hot label for the current optimized view image; Under the supervision of the reliable thermal labels, the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image are optimized to obtain the optimal language radiation features of the current optimized view image.

7. The three-dimensional open vocabulary segmentation method based on consistency regularization according to claim 6, characterized in that The initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image are optimized under the supervision of the reliable thermal labels, specifically: the initial language radiation features in the CLIP-NeRF correlation graph of the current optimized view image are optimized under the supervision of the reliable thermal labels using cross entropy loss.

8. A three-dimensional open vocabulary segmentation system based on consistency regularization, characterized by: include: An initialization module is used to reconstruct a 3D scene from multi-view images using NeRF, and to generate an initial language radiation feature for each view based on the reconstructed 3D scene; A feature extraction module is used to extract SegCLIP features of objects in each view image using SAM and CLIP, calculate the geometric correspondence between each view image and other view images, and aggregate the SegCLIP features of the same object in different view images according to the geometric correspondence to obtain SegCLIP guided features; An optimization module is used to generate SegCLIP-guided correlation maps of all view images using a cosine similarity measurement method based on the text features and SegCLIP-guided features of each view image; select multiple images from the multi-view images as optimized view images; optimize each optimized view image separately: generate a CLIP-NeRF-guided correlation map of the current optimized view image using a cosine similarity measurement method based on the primary language radiation features and text features of the current optimized view image; use the SegCLIP-guided correlation maps of all other view images except the current optimized view image to optimize the initial language radiation features in the CLIP-NeRF correlation map of the current optimized view image to obtain the optimal language radiation features of the current optimized view image; The segmentation module is used to measure the similarity between the optimal language radiation features and the text features for each optimized view image, and the result with the highest score is used as the segmentation result of the corresponding optimized view image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional open vocabulary segmentation method based on consistency regularization are implemented.

Images