Three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction
Through cross-modal mask interaction technology, the three-dimensional and two-dimensional features are integrated, and the problem of insufficient fine-grained geometric shape segmentation and robustness in the existing technology is solved, achieving a more accurate and consistent three-dimensional open vocabulary semantic segmentation effect.
Patent Information
- Application Number
- CN202411146370.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
The existing three-dimensional perception methods of open vocabulary have shortcomings in fine-grained geometric shape segmentation and robustness, especially the global feature alignment ignores details and is rough, point-level comparison is susceptible to noise, and patch-level alignment is poor.
Using a cross-modal mask interaction method, features are extracted through implicit 3D descriptor and denoised Unet structure, fusion of 3D and 2D features, and fine-grained features are achieved using two-dimensional to three-dimensional mask regularization terms and CLIP encoder. Alignment to enhance the performance of three-dimensional open vocabulary semantic segmentation.
It improves the accuracy and robustness of semantic segmentation of three-dimensional open vocabulary, enhances the correlation between three-dimensional features and text features, realizes the effective complementarity of information between modals, and improves the ability to extrapolate in new categories.
Smart Images

Figure CN119006822A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cross-modal learning technology, and in particular to a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction. Background Art
[0002] As the integration of vision and language in deep learning continues to expand, textual descriptions are increasingly used in visual generation and perception tasks. This integration enhances the adaptability of models for real-world applications and improves the user experience in custom AI systems. Open-vocabulary 3D semantic segmentation is an example of a perception task that is trained on a base class and requires strong extrapolation capabilities to distinguish fine-grained geometry in new classes that are unseen during training. The base and new classes are connected only by a shared open vocabulary in the language space. However, constructing a shared 3D-text space while maintaining accurate, modality-specific representations remains a significant challenge.
[0003] Existing open-vocabulary 3D-aware methods mainly bridge the gap between 3D and textual representations by using the 2D modality as an intermediary. Some methods align 3D embeddings to the visual-language embedding space via global, patch-level, or point-level contrastive losses, while others explore extracting open-vocabulary knowledge from the underlying visual-language model into the 3D model. However, global feature alignment and model distillation techniques often ignore fine-grained 3D geometric details, resulting in coarse results. In contrast, point-level contrastive learning is susceptible to noise and outliers. While patch-level feature alignment provides a compromise, a single patch in 2D may correspond to multiple unrelated and discontinuous regions in 3D, which can be misleading and produce poor results. Summary of the invention
[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.
[0005] To this end, the first purpose of the present application is to propose a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction, so as to improve the performance of three-dimensional open vocabulary semantic segmentation based on cross-modal mask interaction, and solve the problems of geometric details being ignored and weak robustness in the three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction in the related art.
[0006] The second objective of this application is to propose a three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction.
[0007] The third objective of the present application is to provide an electronic device.
[0008] A fourth objective of the present application is to provide a computer-readable storage medium.
[0009] A fifth object of the present application is to provide a computer program product.
[0010] To achieve the above objectives, the first embodiment of the present application proposes a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction, including:
[0011] Acquire point cloud data to be segmented, and perform feature extraction on the point cloud data using a point cloud segmentation model to obtain three-dimensional features of the point cloud;
[0012] Encoding features output by a point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding;
[0013] The implicit conditional embedding is used as an implicit description and an input image is input into a denoising Unet structure of a diffusion model, and features are extracted from an established text-two-dimensional embedding space through the denoising Unet structure to obtain image two-dimensional features; and the image two-dimensional features are converted into two-dimensional segmentation mask features through a mask generator;
[0014] Fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature;
[0015] Based on the fused features, a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is output.
[0016] In some implementations, the method further includes:
[0017] Through gradient descent, the point cloud encoder extracts visual-linguistic knowledge from the denoising Unet structure through the implicit 3D descriptor to achieve gradient optimization and parameter update of the point cloud segmentation model.
[0018] In some implementations, a loss function for optimizing the point cloud segmentation model includes a two-dimensional to three-dimensional mask regularization term, and an implementation process of the two-dimensional to three-dimensional mask regularization term includes:
[0019] extracting a three-dimensional mask feature from the three-dimensional features of the point cloud using a back-projected three-dimensional binary mask;
[0020] Based on the two-dimensional segmentation mask feature and the input image, a real two-dimensional CLIP feature is obtained through a pre-trained CLIP image encoder;
[0021] The two-dimensional CLIP feature is used as a true value, and the three-dimensional mask feature is used as a prediction. A two-dimensional to three-dimensional mask regularization term is obtained through comparative learning of the two-dimensional CLIP feature and the three-dimensional mask feature.
[0022] In some implementations, outputting a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data based on the fused features includes:
[0023] Get the text description of the input;
[0024] Encoding the text description by using a pre-trained CLIP text encoder to obtain a CLIP text encoding;
[0025] Based on the fused features and the CLIP text encoding, a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is obtained.
[0026] In some implementations, the fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature includes:
[0027] At the points where there is a corresponding relationship, the point cloud three-dimensional features and the two-dimensional segmentation mask features are fused to obtain fused features.
[0028] In some implementations, the text description is a category label.
[0029] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction, including:
[0030] A three-dimensional branch module is used to obtain point cloud data to be segmented, and to extract features from the point cloud data through a point cloud segmentation model to obtain three-dimensional features of the point cloud;
[0031] The three-dimensional branch module is further used to encode features output by the point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding;
[0032] A two-dimensional branch module, used for taking the implicit conditional embedding as an implicit description and inputting the image into a denoising Unet structure of a diffusion model, extracting features from the established text-two-dimensional embedding space through the denoising Unet structure to obtain image two-dimensional features; and converting the image two-dimensional features into two-dimensional segmentation mask features through a mask generator;
[0033] A feature fusion module, used for fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain fused features;
[0034] The result acquisition module is used to output a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data based on the fusion feature.
[0035] In some implementations, the three-dimensional branching module is further used to:
[0036] Through gradient descent, the point cloud encoder extracts visual-linguistic knowledge from the denoising Unet structure through the implicit 3D descriptor to achieve gradient optimization and parameter update of the point cloud segmentation model.
[0037] In some implementations, a loss function for optimizing the point cloud segmentation model includes a two-dimensional to three-dimensional mask regularization term, and an implementation process of the two-dimensional to three-dimensional mask regularization term includes:
[0038] extracting a three-dimensional mask feature from the three-dimensional features of the point cloud using a back-projected three-dimensional binary mask;
[0039] Based on the two-dimensional segmentation mask feature and the input image, a real two-dimensional CLIP feature is obtained through a pre-trained CLIP image encoder;
[0040] The two-dimensional CLIP feature is used as a true value, and the three-dimensional mask feature is used as a prediction. A two-dimensional to three-dimensional mask regularization term is obtained through comparative learning of the two-dimensional CLIP feature and the three-dimensional mask feature.
[0041] In some implementations, the result acquisition module is specifically used to:
[0042] Get the text description of the input;
[0043] Encoding the text description by using a pre-trained CLIP text encoder to obtain a CLIP text encoding;
[0044] Based on the fused features and the CLIP text encoding, a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is obtained.
[0045] In some implementations, the feature fusion module is specifically used to:
[0046] At the points where there is a corresponding relationship, the point cloud three-dimensional features and the two-dimensional segmentation mask features are fused to obtain fused features.
[0047] In some implementations, the text description is a category label.
[0048] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.
[0049] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.
[0050] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.
[0051] The present application provides a three-dimensional open vocabulary semantic segmentation method, device, electronic device and storage medium based on cross-modal mask interaction, which controls the generation of two-dimensional masks through three-dimensional global features, utilizes two-dimensional masks to enhance the correlation between three-dimensional features and text features, and realizes information complementarity between modalities through cross-modal feature fusion, thereby improving the performance of three-dimensional open vocabulary semantic segmentation and solving the problems of geometric details being ignored and weak robustness in three-dimensional open vocabulary semantic segmentation methods in related technologies.
[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0054] Figure 1 A flowchart of a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction provided in an embodiment of the present application;
[0055] Figure 2 A design framework diagram of a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction provided in an embodiment of the present application;
[0056] Figure 3 A model architecture diagram adopted by a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction provided in an embodiment of the present application;
[0057] Figure 4 A block diagram of a three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction provided by an embodiment of the present application;
[0058] Figure 5 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0060] The following describes a three-dimensional open vocabulary semantic segmentation method, apparatus, and device based on cross-modal mask interaction according to an embodiment of the present application with reference to the accompanying drawings.
[0061] Figure 1 A flowchart of a three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction provided in an embodiment of the present application.
[0062] It should be noted that the executor of the three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction in the embodiment of the present application is the three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction in the embodiment of the present application. The three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction can be configured in an electronic device so that the electronic device can perform the three-dimensional open vocabulary semantic segmentation function based on cross-modal mask interaction.
[0063] like Figure 1 As shown, the three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction includes the following steps:
[0064] Step 101, obtaining point cloud data to be segmented, performing feature extraction on the point cloud data through a point cloud segmentation model, and obtaining three-dimensional features of the point cloud.
[0065] In this embodiment, the three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction of the embodiment of the present application is implemented by the XMask three-dimensional model, such as Figure 2 , 3 As shown, the XMask three-dimensional model includes a three-dimensional branch, a two-dimensional branch and a fusion block.
[0066] The 3D branch is used to obtain the point cloud data to be segmented, and extracts features from the point cloud data through the point cloud segmentation model to obtain the 3D features of the point cloud. The 3D branch can adapt to any point cloud segmentation model and is good at capturing geometric features, but performs poorly in extrapolating new categories.
[0067] Step 102, encoding the features output by the point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding.
[0068] The 3D branch is also used to encode the features output by the point cloud encoder of the point cloud segmentation model through an implicit 3D captioner to obtain an implicit conditional embedding.
[0069] In some embodiments, through gradient descent, the point cloud encoder extracts visual-language knowledge from the denoising Unet structure through an implicit 3D descriptor to achieve gradient optimization and parameter update of the point cloud segmentation model.
[0070] It can be understood that the embodiment of the present application proposes an implicit three-dimensional descriptor, such as Figure 3 As shown in Figure 1, the implicit 3D descriptor takes the global 3D features from the point cloud encoder as input and predicts implicit conditional embeddings to be injected into the diffusion model. Since the weights of the denoising Unet structure are frozen, the training objective of the 2D branch implicitly pushes the global 3D features into the text-2D image feature space. Through gradient descent, the point cloud encoder gradually extracts some visual-linguistic knowledge from the pre-trained and frozen denoising Unet structure. At the same time, the global 3D features contain rich 3D geometric information, which the 2D branch lacks due to the inherent occlusion and dimensionality compression problems of the image. Using global 3D features as conditions for the 2D branch can prompt the model to generate geometry-aware mask contours and embeddings, thereby facilitating the 3D space back-projection process of the 2D mask.
[0071] Step 103, using the implicit conditional embedding as the implicit description and the input image into the denoising Unet structure of the diffusion model, extracting features from the established text-two-dimensional embedding space through the denoising Unet structure to obtain the image two-dimensional features; and converting the image two-dimensional features into two-dimensional segmentation mask features through a mask generator.
[0072] It can be understood that the two-dimensional branch of the embodiment of the present application adopts the denoising Unet structure of the text-to-image diffusion model with strong open vocabulary capabilities, extracts features from the established text-two-dimensional embedding space, and then converts these features into segmentation masks through a mask generator.
[0073] This example uses a denoising UNet of a pre-trained text-to-image diffusion model as a 2D mask generator, which has advanced visual-language modeling capabilities. The diffusion model performs well in text-driven image generation and demonstrates strong potential in creating fine-grained segmentation masks for new categories.
[0074] It should be noted that the text-2D embedding space is established during the pre-training of the diffusion model Unet structure and the clip model.
[0075] In some embodiments, the loss function for optimizing the point cloud segmentation model includes a 2D to 3D mask regularization term, and the implementation process of the 2D to 3D mask regularization term includes: extracting 3D mask features from the 3D features of the point cloud using a back-projected 3D binary mask; obtaining true 2D CLIP features based on the 2D segmentation mask features and the input image (Image) through a pre-trained CLIP image encoder (CLIP Image Encoder); taking the 2D CLIP features as true values and the 3D mask features as predictions, and obtaining the 2D to 3D mask regularization term through comparative learning of the 2D CLIP features and the 3D mask features.
[0076] It can be understood that the embodiment of the present application proposes an explicit 2D to 3D mask regularization term for achieving fine-grained and consistent feature space alignment between 3D and 2D-text modalities. Specifically, a 3D mask embedding is extracted from a 3D feature using a back-projected 3D binary mask. Given a 2D binary mask and an input image, a true mask CLIP feature can also be derived using a pre-trained CLIP model. Subsequently, the 2D to 3D regularization term can be calculated using the classic cosine contrast loss.
[0077] Since each mask region ideally corresponds to a unique category, the pooled mask embedding achieves semantic consistency and representativeness. Therefore, mask-level contrastive learning is more fine-grained than global contrastive, more robust than point-to-point contrastive, and sharper than local contrastive. Through the proposed 2D-to-3D mask regularization, 3D features are explicitly aligned to the 2D-text feature space, enhancing the performance of the 3D branch in open vocabulary segmentation.
[0078] It should be noted that the 2D branch acts as a mask generator, and the predicted mask has an embedding that is aligned with the visual-linguistic feature space, but lacks 3D spatial perception capabilities.
[0079] Step 104: fuse the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature.
[0080] As an implementation method, the fusion block is implemented at points where there is a corresponding relationship, fusing the three-dimensional features of the point cloud and the two-dimensional segmentation mask features to obtain the fused features.
[0081] It can be understood that this step is used to fuse the three-dimensional features from the three-dimensional branch with the pseudo-mask features derived from the two-dimensional branch.
[0082] It is noteworthy that each element in the point cloud feature has a unique and distinctive embedding, while elements belonging to the same mask in the image feature share the same mask embedding. Therefore, the model of this embodiment provides detailed geometric structure information and provides semantic features with strong open vocabulary capabilities. The present invention combines the features of two-dimensional and three-dimensional modalities, and uses the complementary insights of the two modalities to form an excellent performance in precise geometric depiction and extensive semantic inference. Selectively fuse two-dimensional and three-dimensional features at points where there is a corresponding relationship.
[0083] It can be understood that the 3D branch performs poorly in extrapolation to new categories, and the 2D branch lacks 3D spatial perception, and the fusion block enables the two branches to complement each other.
[0084] Therefore, the embodiment of the present application realizes 2D-3D cross-modal mask knowledge interaction, controls 2D mask generation through 3D global features, utilizes 2D masks to enhance the correlation between 3D features and text features, and realizes inter-modal information complementarity through cross-modal feature fusion.
[0085] Step 105 , based on the fused features, output a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data.
[0086] As an implementation manner, a method for outputting a segmentation mask corresponding to an input text description in a point cloud corresponding to point cloud data based on fused features includes: obtaining an input text description; encoding the text description through a pre-trained CLIP text encoder (CLIP Text Encoder) to obtain a CLIP text encoding; and obtaining a segmentation mask corresponding to the input text description in a point cloud corresponding to the point cloud data based on the fused features and the CLIP text encoding.
[0087] As a way to achieve this, Figure 3 , a method for obtaining a segmentation mask corresponding to a text description includes: performing vector multiplication on the CLIP text encoding and the fusion feature to obtain a segmentation mask corresponding to the text description.
[0088] In some embodiments, the text description is a category label.
[0089] Exemplarily, it can be understood that now we get a result vector and multiple category vectors. If we want to know which category the result vector belongs to, we multiply them in sequence to get the similarity (scalar), and the one with the highest similarity is the final classification result.
[0090] Therefore, an embodiment of the present application proposes a 3D to 2D mask generation method, which conditions a denoising mask generator on global point cloud features to generate geometry-aware masks that are more suitable for transfer to a 3D modality. Mask-level regularization is applied to 3D features to align with the visual-language embedding space. This enhances the open vocabulary capability of 3D features on new categories while retaining fine-grained geometric information. A 3D-2D cross-modal mask feature fusion method is proposed. The mask features from the two modalities are merged in the fusion block to enhance the synergy between 2D and 3D features.
[0091] The three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction of this embodiment controls the generation of two-dimensional masks through three-dimensional global features, uses two-dimensional masks to enhance the correlation between three-dimensional features and text features, and realizes inter-modal information complementarity through cross-modal feature fusion, that is, a more accurate and consistent mask-level alignment method between three-dimensional features and two-dimensional-text embedding space is realized through a cross-modal mask inference method, thereby improving the performance of three-dimensional open vocabulary semantic segmentation and solving the problems of geometric details being ignored and weak robustness in three-dimensional open vocabulary semantic segmentation methods in related technologies.
[0092] In order to implement the above embodiments, the present application also proposes a three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction. Figure 4 A schematic diagram of the structure of a three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction provided in an embodiment of the present application. Figure 4 As shown, the three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction may include: a three-dimensional branch module 201, a two-dimensional branch module 202, a feature fusion module 203 and a result acquisition module 204.
[0093] The three-dimensional branch module 201 is used to obtain the point cloud data to be segmented, and extract the features of the point cloud data through the point cloud segmentation model to obtain the three-dimensional features of the point cloud;
[0094] The three-dimensional branch module 201 is further used to encode the features output by the point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding;
[0095] A two-dimensional branch module 202 is used to use the implicit conditional embedding as an implicit description and the input image into the denoising Unet structure of the diffusion model, extract features from the established text-two-dimensional embedding space through the denoising Unet structure to obtain image two-dimensional features; and convert the image two-dimensional features into two-dimensional segmentation mask features through a mask generator;
[0096] A feature fusion module 203 is used to fuse the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature;
[0097] The result acquisition module 204 is used to output a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data based on the fused features.
[0098] Furthermore, in a possible implementation of the embodiment of the present application, the three-dimensional branch module 201 is further used to:
[0099] Through gradient descent, the point cloud encoder extracts visual-language knowledge from the denoising Unet structure through an implicit 3D descriptor to achieve gradient optimization and parameter update of the point cloud segmentation model.
[0100] Furthermore, in a possible implementation of the embodiment of the present application, the loss function for optimizing the point cloud segmentation model includes a two-dimensional to three-dimensional mask regularization term, and the implementation process of the two-dimensional to three-dimensional mask regularization term includes:
[0101] The 3D mask features are extracted from the 3D features of the point cloud using the back-projected 3D binary mask;
[0102] Based on the 2D segmentation mask features and the input image, the real 2D CLIP features are obtained through the pre-trained CLIP image encoder;
[0103] The 2D CLIP feature is taken as the true value and the 3D mask feature is taken as the prediction. Through comparative learning of the 2D CLIP feature and the 3D mask feature, a 2D to 3D mask regularization term is obtained.
[0104] Furthermore, in a possible implementation of the embodiment of the present application, the result acquisition module 204 is specifically configured to:
[0105] Get the text description of the input;
[0106] Encode the text description through the pre-trained CLIP text encoder to obtain the CLIP text encoding;
[0107] Based on the fusion features and CLIP text encoding, the segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is obtained.
[0108] Furthermore, in a possible implementation of the embodiment of the present application, the feature fusion module 203 is specifically configured to:
[0109] At the points where there is a corresponding relationship, the three-dimensional features of the point cloud and the two-dimensional segmentation mask features are fused to obtain the fused features.
[0110] Furthermore, in a possible implementation of the embodiment of the present application, the text description is a category label.
[0111] It should be noted that the above explanation of the embodiment of the three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction is also applicable to the three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction in this embodiment, and will not be repeated here.
[0112] The three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction of this embodiment controls the generation of two-dimensional masks through three-dimensional global features, uses two-dimensional masks to enhance the correlation between three-dimensional features and text features, and realizes inter-modal information complementarity through cross-modal feature fusion, that is, a more accurate and consistent mask-level alignment method between three-dimensional features and two-dimensional-text embedding space is realized through a cross-modal mask inference method, thereby improving the performance of three-dimensional open vocabulary semantic segmentation and solving the problems of geometric details being ignored and weak robustness in three-dimensional open vocabulary semantic segmentation methods in related technologies.
[0113] In order to implement the above embodiment, the present application also proposes an electronic device. Figure 5 , Figure 5 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a processor 501, and a memory 502 communicatively connected to the processor 501; the memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory to implement the method provided in the aforementioned embodiment.
[0114] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.
[0115] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.
[0116] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.
[0117] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.
[0118] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.
[0119] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0123] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0124] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0125] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0126] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A three-dimensional open vocabulary semantic segmentation method based on cross-modal mask interaction, characterized in that: The following steps are involved: Acquire point cloud data to be segmented, and perform feature extraction on the point cloud data using a point cloud segmentation model to obtain three-dimensional features of the point cloud; Encoding features output by a point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding; The implicit conditional embedding is used as an implicit description and an input image is input into a denoising Unet structure of a diffusion model, and features are extracted from an established text-two-dimensional embedding space through the denoising Unet structure to obtain image two-dimensional features; and converting the two-dimensional image features into two-dimensional segmentation mask features through a mask generator; Fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature; Based on the fused features, a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is output.
2. The method according to claim 1, characterized in that The method further comprises: Through gradient descent, the point cloud encoder extracts visual-linguistic knowledge from the denoising Unet structure through the implicit 3D descriptor to achieve gradient optimization and parameter update of the point cloud segmentation model.
3. The method according to claim 1, characterized in that The loss function for optimizing the point cloud segmentation model includes a two-dimensional to three-dimensional mask regularization term, and the implementation process of the two-dimensional to three-dimensional mask regularization term includes: extracting a three-dimensional mask feature from the three-dimensional features of the point cloud using a back-projected three-dimensional binary mask; Based on the two-dimensional segmentation mask feature and the input image, a real two-dimensional CLIP feature is obtained through a pre-trained CLIP image encoder; The two-dimensional CLIP feature is used as a true value, and the three-dimensional mask feature is used as a prediction. A two-dimensional to three-dimensional mask regularization term is obtained through comparative learning of the two-dimensional CLIP feature and the three-dimensional mask feature.
4. The method according to claim 1, characterized in that The step of outputting a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data based on the fusion feature includes: Get the text description of the input; Encoding the text description by using a pre-trained CLIP text encoder to obtain a CLIP text encoding; Based on the fused features and the CLIP text encoding, a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data is obtained.
5. The method according to claim 1, characterized in that The step of fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain a fused feature comprises: At the points where there is a corresponding relationship, the point cloud three-dimensional features and the two-dimensional segmentation mask features are fused to obtain fused features.
6. The method according to claim 1, characterized in that The text description is a category label.
7. A three-dimensional open vocabulary semantic segmentation device based on cross-modal mask interaction, characterized in that: include: A three-dimensional branch module is used to obtain point cloud data to be segmented, and to extract features from the point cloud data through a point cloud segmentation model to obtain three-dimensional features of the point cloud; The three-dimensional branch module is further used to encode features output by the point cloud encoder of the point cloud segmentation model through an implicit three-dimensional descriptor to obtain an implicit conditional embedding; A two-dimensional branch module, used for inputting the implicit conditional embedding as an implicit description and an input image into a denoising Unet structure of a diffusion model, extracting features from the established text-two-dimensional embedding space through the denoising Unet structure to obtain image two-dimensional features; and converting the two-dimensional image features into two-dimensional segmentation mask features through a mask generator; A feature fusion module, used for fusing the point cloud three-dimensional features and the two-dimensional segmentation mask features to obtain fused features; The result acquisition module is used to output a segmentation mask corresponding to the input text description in the point cloud corresponding to the point cloud data based on the fusion feature.
8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.
Citation Information
Cited By
Indoor point cloud scene open type semantic segmentation method based on text prompt
CN120612489A
Point cloud panorama segmentation method and device based on look-around asymmetric encoder, and medium
CN120808346A
