A three-dimensional directional target segmentation method under weak supervision setting

Abstract

The application discloses a three-dimensional directional target segmentation method under weak supervision, and relates to the technical field of three-dimensional directional target segmentation.The method comprises the following steps: an instance extraction process: acquiring a point cloud scene and a text description, extracting 3D instances in the scene, and representing the 3D instances as instance masks and category score predictions; a multi-expert mining process: extracting information from semantic clues in three dimensions of complete context, attributes and categories through multiple expert models, and respectively calculating semantic distributions of targets; and a multi-expert aggregation process: automatically giving greater weights to input distributions from more accurate expert models according to characteristics of a current sample, highlighting target instances while excluding interference information.The three-dimensional directional target segmentation method under weak supervision provided by the application only provides three-dimensional point cloud data and corresponding directional descriptions, and does not provide mask annotations, so that time and energy for mask annotation are not consumed.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional directional target segmentation technology, and in particular to a three-dimensional directional target segmentation method under weak supervision. Background Technology

[0002] 3D Referring Expression Segmentation (3D-RES) has attracted much attention in the multimodal field, aiming to segment 3D instances of targets by aligning directional descriptions with point clouds, thereby generating accurate target segmentation masks. Benchmark datasets for the 3D-RES task mainly include Scanrefer, ReferIt3D, and Multi3DRefer.

[0003] Existing 3D-RES methods rely on fully supervised setups and require expensive point-level mask annotation. For example, annotating point cloud scenes on the ScanNet-v2 dataset takes an average of 22.3 minutes. The labor-intensive nature of these annotations hinders the collection of large-scale 3D datasets and limits the scalability of this task in data-scarce domains. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a three-dimensional directional target segmentation method under weak supervision, which only requires three-dimensional point cloud data and corresponding directional description, without the need to provide mask annotations, thus eliminating the need to spend time and effort on mask annotation.

[0005] This invention provides a three-dimensional directional target segmentation method under weak supervision, comprising:

[0006] Instance extraction process: Obtain point cloud scene and text description, extract 3D instances in scene, and represent them as instance masks and class score predictions;

[0007] Multi-expert mining process: Information is extracted from semantic cues in three dimensions—complete context, attributes, and categories—through multiple expert models, and the semantic distribution of the target is calculated for each. Specifically, the complete context expert processes the complete visual and textual context, evaluating the degree of matching between the instance and the description from the perspective of spatial relationships and the entire scene; the attribute expert focuses on details such as the shape, color, and texture of the instance to distinguish instances at the attribute level; and the category expert focuses on the semantic category matching between the instance and the text description.

[0008] Multi-expert aggregation process: Based on the characteristics of the current sample, the input distribution from more accurate expert models is automatically assigned greater weight, highlighting the target instance while eliminating interfering information.

[0009] Furthermore, the instance extraction process specifically includes:

[0010] Given a set containing A point cloud scene with 3D coordinates and RGB information for each point. Extract 3D instances from the scene and represent them as instance masks. and category score prediction ,in This indicates the number of 3D instances extracted. Indicates the number of categories;

[0011] Given a sentence containing The text description of each word is represented as , used to describe the target instance.

[0012] Furthermore, in the multi-expert mining process, the complete contextual expert model specifically refers to:

[0013] Given input description A pre-trained Long-CLIP text encoder is used to extract complete contextual text features. :

[0014]

[0015] in, This represents a Long-CLIP text encoder, where the feature dimension is [missing information].

[0016] Given a point cloud scene with Original RGB-D video frames ,in , Let represent the width and height of the image, respectively. First, all 3D instances are projected onto all frame sequences to calculate the visibility of each 3D instance in each frame. Then, based on the visibility, the K frames with the highest visibility for each 3D instance are selected as the visual representation of that instance, denoted as . ,in, This indicates the number of 3D instances extracted. , These 2D frames represent the width and height of the image; features are extracted from these frames by a pre-trained Long-CLIP image encoder, and then average pooling is used to obtain the complete 2D contextual features. :

[0017]

[0018] in, This indicates a Long-CLIP image encoder. This indicates an average pooling operation;

[0019] Finally, the semantic similarity at the full context level is calculated. and having a probability mass function Normalized complete context semantic distribution :

[0020]

[0021]

[0022] in Represents matrix multiplication. This indicates the softmax operation.

[0023] Furthermore, in the multi-expert mining process, the attribute expert model specifically refers to:

[0024] Given input description The natural language parser is used to decompose it into five semantic components: main object, auxiliary object, attribute, pronoun, and relation; then the main object and its attributes are extracted and represented as target words. and modifiers ; target words and modifiers Connecting them together yields a short attribute phrase. The phrase contains only the name of the target instance and its modifiers; subsequently, a pre-trained CLIP text encoder is used to extract attribute text features. :

[0025]

[0026] in, Indicates CLIP text encoder;

[0027] Simultaneously, an object renderer is used to render a multi-view 2D image for each 3D instance as its visual representation, represented as... ,in This indicates the number of 3D instances extracted. Indicates the number of rendering angles. , These represent the width and height of the image, respectively; the 2D rendered image is obtained by feature extraction from a pre-trained CLIP image encoder, followed by average pooling to obtain 2D attribute features. :

[0028]

[0029] in This indicates the CLIP image encoder. This indicates an average pooling operation;

[0030] Finally, the semantic similarity of the attribute dimensions between the text and the 3D instance is calculated. and having a probability mass function Normalized attribute semantic distribution :

[0031]

[0032]

[0033] in Represents matrix multiplication. This indicates the softmax operation.

[0034] Furthermore, in the multi-expert mining process, the category expert model specifically refers to:

[0035] Given input description Complete contextual text features are extracted by a pre-trained Long-CLIP text encoder. Then, the text classifier outputs the predicted text category score. :

[0036]

[0037] in This represents a text classifier that uses a classification loss based on cross-entropy (CE). Training from scratch:

[0038]

[0039] in The category label representing the target instance;

[0040] Finally, the semantic similarity between the text and the 3D instance in the category dimension is calculated. and having a probability mass function Normalized category semantic distribution :

[0041]

[0042]

[0043] in Represents matrix multiplication. This indicates the softmax operation.

[0044] Furthermore, the multi-expert aggregation process assumes a finite number of probability distributions. Fusion is the process of combining these distributions into a single probability distribution. and with symbols Specify; assume They are discrete, each possessing a probability mass function. and common atoms ,in ;So There exist and the following expressions are equivalent:

[0045] (i) ;

[0046] (ii) ;

[0047] in It refers to the point located in the set of real Borel probability measures. The Dirac delta measure, i.e., if ,but ;if but Replace the distribution with a normalized categorical semantic distribution. Normalized attribute semantic distribution and normalized complete context semantic distribution ,get:

[0048]

[0049] Among them, common atoms It means 3D instance ;

[0050] Using the properties of Borel sets, it can be rewritten as a probability mass function. :

[0051]

[0052] in Represents a 3D instance. Merging of multidimensional semantic distributions The probability mass function;

[0053] Finally from Mid-sampling to predict target instances :

[0054]

[0055] in, The sampling operation is defined as starting from the highest probability mass. Choose a target from the suggestions, among which .

[0056] The multi-expert mining proposed in this invention can effectively mine and match visual and textual contexts. Compared with direct summation or set methods, multi-expert aggregation can effectively highlight target instances and eliminate interfering information, thereby using multiple clues to complete text-based directional reasoning without the need for mask annotation, thus saving time and effort in mask annotation. Attached Figure Description

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] Figure 1 This is a general flowchart of the method according to an embodiment of the present invention;

[0059] Figure 2 This is a flowchart illustrating an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram illustrating the working principle of the module in an embodiment of the present invention;

[0061] Figure 4 This is a visual diagram illustrating the results of an embodiment of the present invention. Detailed Implementation

[0062] This invention provides a three-dimensional directional target segmentation method under weak supervision, which only requires three-dimensional point cloud data and corresponding directional descriptions, without the need for mask annotations, thus saving time and effort in mask annotation.

[0063] The overall concept of the technical solutions in the embodiments of the present invention is as follows:

[0064] This invention proposes a weakly supervised 3D-RES setting for multi-expert network-based 3D directional target segmentation. In this setting, only 3D point cloud data and corresponding directional descriptions are provided, without mask annotations, thus eliminating the time and effort required for mask annotation. A straightforward approach is to first segment the point cloud scene using a pre-trained 3D model, extract 3D instances, render them as 2D images, and then directly match them with the directional descriptions using mature 2D image-text matching techniques (such as CLIP). However, this method isolates the 3D instances from the scene, failing to utilize the key contextual clues provided in the description for accurate target localization. Another approach is to select the most visible 2D image for each 3D instance from the video used to construct the 3D point cloud and then use CLIP for matching. While this method more effectively utilizes the contextual information contained in the text, it can lead to ambiguous matching if the image contains multiple instances. Therefore, the main challenge of weakly supervised 3D-RES lies in effectively mining and matching visual and textual context while filtering out potential interference factors.

[0065] To address this, this invention proposes a Multi-Expert Network (MEN), a method for text-based directional reasoning utilizing multiple cues. This method decomposes the problem into two sub-tasks: how to effectively mine and match visual and textual contexts; and how to merge the results of the first step, highlighting the target instance while filtering out distracting information. For the first sub-task, this invention introduces a Multi-Expert Mining (MEM) module. This module is designed with experts capable of extracting information from multiple semantic cues, focusing primarily on three dimensions: complete context, attributes, and category, calculating the semantic distribution of the target instance. Specifically, the complete context expert processes the complete visual and textual context, evaluating the matching degree between the instance and description from the perspective of spatial relationships and the entire scene; the attribute expert focuses on details such as the instance's shape, color, and texture, distinguishing instances at the attribute level; and the category expert focuses on semantic category matching between the instance and the textual description. For the second sub-task, this invention proposes a Multi-Expert Aggregation (MEA) module. This module automatically assigns greater weight to the input distribution from more accurate experts. Compared to direct summation or set methods, MEA effectively highlights the target instance and eliminates distracting information.

[0066] This embodiment provides a three-dimensional directional target segmentation method under weak supervision, such as... Figure 1 and Figure 2 As shown, it includes:

[0067] S1. Instance extraction process: Obtain point cloud scene and text description, extract 3D instances in the scene, and represent them as instance masks and category score predictions.

[0068] Given a set containing A point cloud scene with 3D coordinates and RGB information for each point. 3D instances in the scene are extracted using pre-trained SoftGroup and represented as instance masks. and category score prediction ,in This indicates the number of 3D instances extracted. Indicates the number of categories.

[0069] Given a sentence containing The text description of each word is represented as , used to describe the target instance.

[0070] S2. Multi-Expert Mining Process: This process extracts information from semantic cues across three dimensions—complete context, attributes, and category—using multiple expert models, and calculates the semantic distribution of the target for each dimension. Specifically, the complete context expert processes the full visual and textual context, evaluating the match between the instance and description from the perspectives of spatial relationships and the entire scene. The attribute expert focuses on details such as the instance's shape, color, and texture, differentiating instances at the attribute level. The category expert focuses on the semantic category matching between the instance and the textual description. This process aims to extract information from semantic cues across three different dimensions and calculate the semantic distribution of the target.

[0071] (1) Complete Context Expert Model

[0072] Given input description A pre-trained Long-CLIP text encoder is used to extract complete contextual text features. :

[0073] (1)

[0074] in, This represents a Long-CLIP text encoder, where the feature dimension is [not specified].

[0075] Given a point cloud scene with Original RGB-D video frames ,in , Let represent the width and height of the image, respectively. First, all 3D instances are projected onto all frame sequences to calculate the visibility of each 3D instance in each frame. Then, based on the visibility, the K frames with the highest visibility for each 3D instance are selected as the visual representation of that instance, denoted as . ,in, This indicates the number of 3D instances extracted. , These 2D frames represent the width and height of the image; features are extracted from these frames by a pre-trained Long-CLIP image encoder, and then average pooling is used to obtain the complete 2D contextual features. :

[0076] (2)

[0077] in, This indicates a Long-CLIP image encoder. This indicates the average pooling operation.

[0078] Finally, the semantic similarity at the full context level is calculated. and having a probability mass function Normalized complete context semantic distribution :

[0079] (3)

[0080] (4)

[0081] in Represents matrix multiplication. This indicates the softmax operation.

[0082] (2) Attribute expert model

[0083] Given input description The SceneGraphParser natural language parser is used to decompose the language into five distinct semantic components: the main object, the auxiliary object, attributes, pronouns, and relationships. The main object and its corresponding attributes are then extracted and represented as target words. and modifiers These words will be connected to form a short attribute phrase. The phrase contains only the name of the target instance and its modifiers; subsequently, a pre-trained CLIP text encoder is used to extract attribute text features. :

[0084] (5)

[0085] in, This indicates the CLIP text encoder.

[0086] Simultaneously, an object renderer is used to render a multi-view 2D image for each 3D instance as its visual representation, represented as... ,in This indicates the number of 3D instances extracted. Indicates the number of rendering angles. , These represent the width and height of the image, respectively; the 2D rendered image is obtained by feature extraction from a pre-trained CLIP image encoder, followed by average pooling to obtain 2D attribute features. :

[0087] (6)

[0088] in This indicates the CLIP image encoder. This indicates an average pooling operation;

[0089] Finally, the semantic similarity of the attribute dimensions between the text and the 3D instance is calculated. and having a probability mass function Normalized attribute semantic distribution :

[0090] (7)

[0091] (8)

[0092] in Represents matrix multiplication. This indicates the softmax operation.

[0093] (3) Category expert model

[0094] Given input description Complete contextual text features are extracted by a pre-trained Long-CLIP text encoder. Then, the text classifier outputs the predicted text category score. :

[0095] (9)

[0096] in This represents a text classifier that uses a classification loss based on cross-entropy (CE). Training from scratch:

[0097] (10)

[0098] in The category label representing the target instance;

[0099] Finally, the semantic similarity between the text and the 3D instance in the category dimension is calculated. and having a probability mass function Normalized category semantic distribution :

[0100] (11)

[0101] (12)

[0102] in Represents matrix multiplication. This indicates the softmax operation.

[0103] S3. Multi-expert aggregation process: Based on the characteristics of the current sample, the input distribution from a more accurate expert model is automatically assigned greater weight, highlighting the target instance while eliminating interfering information.

[0104] The multi-expert aggregation module assigns greater weights to input distributions from more accurate experts from a mathematical perspective, especially those distributions with smaller standard deviations. The final distribution obtained through the multi-expert aggregation module is both the unique minimum likelihood ratio merge and the unique proportional likelihood ratio merge for a given input distribution, ensuring robust integration of expert predictions.

[0105] Therefore, we first analyze the advantages and disadvantages of the three types of experts mentioned above: Complete context experts excel at capturing complete spatial context, but may introduce interference from complex 2D video frames, making them suitable for scenarios where spatial relationships are crucial for target identification; Attribute experts focus on the visual attributes of instances and corresponding textual descriptions, but may be affected by rendering accuracy and parsing tools, making them ideal for situations where the target's attributes are unique and distinct; Category experts utilize the classification capabilities of SoftGroups, effectively eliminating irrelevant objects that do not match the target category, demonstrating strong anti-interference capabilities, but often lacking differentiation between similar objects, making them suitable for situations where the target's category is unique. The multi-expert aggregation module can leverage the strengths of more accurate experts based on the characteristics of the current sample.

[0106] Assuming a finite number of probability distributions Fusion is the process of combining these distributions into a single probability distribution. and with symbols Specify; assume They are discrete, each possessing a probability mass function. and common atoms ,in ;So There exist and the following expressions are equivalent:

[0107] (i) ;

[0108] (ii) ;

[0109] in It refers to the point located in the set of real Borel probability measures. The Dirac delta measure, i.e., if ,but ;if but Replace the distribution with a normalized categorical semantic distribution. Normalized attribute semantic distribution and normalized complete context semantic distribution ,get:

[0110] (13)

[0111] Among them, common atoms It means 3D instance .

[0112] Using the properties of Borel sets, this can be rewritten as a probability mass function. :

[0113] (14)

[0114] in Represents a 3D instance. Merging of multidimensional semantic distributions The probability mass function;

[0115] Finally from Mid-sampling to predict target instances :

[0116] (15)

[0117] in, The sampling operation is defined as starting from the highest probability mass. Choose a target from the suggestions, among which .

[0118] The following experiments illustrate the beneficial effects of this embodiment. The specific experimental results are as follows:

[0119] This embodiment was trained and tested on the 3D directional object segmentation dataset ScanRefer, which contains 51,583 natural language expressions from 11,046 objects in 800 ScanNet scenes. All samples can be divided into two categories: 1) unique, where no other object of the same category exists in the scene; 2) multiple, where other objects of the same category exist in the scene.

[0120] All experiments were implemented using PyTorch and trained on a single NVIDIA GeForce RTX 3090 GPU.

[0121] This embodiment uses the average crossover-union ratio (mIoU) and the first The results were more than the expected. Recall rate (R@ ,IoU@ This is used as the evaluation metric in this embodiment. The crossover ratio (CLOUD) of the first n results exceeds [a certain threshold]. The recall rate represents the number of times a candidate selects a candidate. In one instance, the IoU of the overlapping portion between the predicted mask and the ground truth annotation is greater than [value missing]. The proportion of the descriptive text of the instances, where , .

[0122] The table below shows some results of the 3D-REC and RES models on the 3D-GRES task. The weakly supervised 3D-RES Baseline in this embodiment is built upon the first weakly supervised 3D-REC model with some modifications, including changing the object detection model to the instance segmentation model SoftGroup and excluding components related to word reconstruction and distillation. The Baseline achieved the expected performance with an mIoU of 19.9%. This embodiment also randomly selects a target object from the same category based on the text category, termed "Baseline-random". Compared to the baseline, this embodiment improves mIoU and R@1, IoU@0.5 by 13.5% and 15.2%, respectively, especially in the more challenging "multiple" class by 13.9%. This demonstrates that this embodiment can not only accurately identify simple "unique" targets but also effectively handle difficult situations where multiple similar interfering objects exist in the scene. This highlights how the MEM module of this embodiment effectively captures the complex semantics of the scene and descriptive text, while the MEA module effectively consolidates these advantages, providing strong generalization and robust discriminative capabilities. Therefore, in terms of mIoU, the weakly supervised method in this embodiment is significantly higher than the fully supervised 3D-RES method TGNN by 5.6%.

[0123] To extend the weakly supervised setup directly to fully supervised 3D-RES models, the MEN method in this embodiment can also serve as a teacher model to generate pseudo-labels, such as target instance masks, to supervise existing fully supervised 3D-RES models. Results are represented as "Ours_Model Name". As shown in Table 1, using pseudo-labels generated by MEN for supervision can help ordinary 3D-RES models achieve competitive performance, and even surpass the performance of MEN.

[0124] Table 1 Results of the 3D-RES task on ScanRefer

[0125]

[0126] The embodiments of the present invention also provide a visual analysis of the principles of MEM and MEA modules. Figure 3 Attention heatmaps and semantic distributions for category experts, attribute experts, and full-text experts are presented to explore their respective roles. Figure 4As shown, these experts each have their own roles, assisting the model in locating the target at different levels. The full context expert focuses on scene corners related to the description but cannot distinguish objects in these corners, such as “TV,” “TV cabinet,” and “blue object” as shown in (a). The attribute expert follows the instruction of “rectangular gray TV cabinet,” indicating interest in all “rectangular,” “gray,” or potentially classifiable as “TV cabinet.” Similarly, the category expert also focuses on all objects in the scene that are similar in category to the target object. Therefore, while relying solely on information about the target object is suitable for scenarios with unique categories and attributes, it proves ineffective in complex “multiple” scenarios. It can be observed that only objects exhibiting high similarity among all experts are likely to be predicted as targets. For example, the “blue object” near the target shows high attention in both “category” and “full context” but low attention in “attribute,” which reduces its probability of being selected as a target. In summary, each expert has its own strengths and weaknesses. As shown in (b), the MEA module fully leverages the complementary strengths of the three experts while filtering out interfering factors, thereby increasing the importance of the target object.

[0127] Furthermore, embodiments of the present invention also demonstrate visualization results with a weakly supervised 3D-RES baseline. For example... Figure 4 As shown, this invention successfully identifies objects matching the target category from geometrically similar objects, as illustrated in Case (a). Furthermore, this invention effectively utilizes attribute information to distinguish similar objects, such as the "three seats" mentioned in the input description of Case (b). Moreover, embodiments of this invention can infer the target object from multiple similar objects based on the complete context, as shown in Case (c). In contrast, weakly supervised baselines show almost the same level of interest in all similar objects, leading to inaccurate predictions. More notably, as in Case (d), even if the target belongs to the "other" category, embodiments of this invention effectively eliminate interference and achieve precise localization with the help of the effective integration of MEM and MEA modules.

[0128] The multi-expert mining proposed in this invention can effectively mine and match visual and textual contexts. Compared with direct summation or set methods, multi-expert aggregation can effectively highlight target instances and eliminate interfering information, thereby using multiple clues to complete text-based directional reasoning without the need for mask annotation, thus saving time and effort in mask annotation.

[0129] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for three-dimensional directional target segmentation under weak supervision, characterized in that, Comprise: Instance extraction process: obtain a point cloud scene and a text description, extract 3D instances in the scene, and represent them as instance masks and class score predictions; Multi-expert mining process: information is extracted from semantic clues in three dimensions of complete context, attributes and categories by multiple expert models, and the semantic distribution of the target is calculated respectively; wherein, the complete context expert processes the complete visual and text context, evaluates the matching degree between the instance and the description from the perspective of spatial relationship and the whole scene; the attribute expert pays attention to the shape, color and texture of the instance, and distinguishes the instance at the attribute level; the category expert focuses on the semantic category matching between the instance and the text description; The complete context expert model is specifically: Given input description , extract full context text features with pre-trained Long-CLIP text encoder : wherein, denotes the text encoder for Long-CLIP, and d is the feature dimension. Given a raw RGB-D video with frames used to build a point cloud scene , denote the width and height of the image, respectively, all 3D instances are first projected onto all frame sequences to compute the visibility of each 3D instance on each frame; then, according to the visibility, the K frames with the highest visibility are selected for each 3D instance as its visual representation, denoted as , where , denote the width and height of the image; these 2D frames are fed into a pre-trained Long-CLIP image encoder to extract features, and then average pooling is applied to obtain the 2D complete context features : wherein, denotes a Long-CLIP image encoder, denotes an average pooling operation; Finally, the semantic similarity of the full context dimension is calculated and the normalized full context semantic distribution with the probability mass function :​ wherein denotes a matrix multiplication, denotes a softmax operation; The attribute expert model is specifically: Given the input description , a natural language parser is employed to decompose it into five semantic components: main object, auxiliary object, attribute, pronoun, and relationship; then the main object and its attribute are extracted and denoted as target word and modifier word , respectively; the target word and modifier word are concatenated to obtain a short attribute phrase , which only contains the name of the target instance and its modifier; subsequently, a pre-trained CLIP text encoder is employed to extract attribute text features : wherein, denotes a CLIP text encoder; Meanwhile, the object renderer is used to render multi-view 2D images for each 3D instance as its visual representation, denoted as where denotes the number of extracted 3D instances, denotes the number of rendering angles, , denote the width and height of the image, respectively; the 2D renderings are fed into a pre-trained CLIP image encoder to extract features, and then average pooling is used to obtain 2D attribute features : wherein denotes a CLIP image encoder, denotes an average pooling operation; Finally, semantic similarity between the attributes dimensions of the text and the 3D instance is calculated and normalized attribute semantic distribution with probability mass function :​ wherein denotes a matrix multiplication, denotes a softmax operation; The category expert model is specifically: Given input description Full context text features are extracted by a pre-trained Long-CLIP text encoder Text class score predictions are output by a text classifier : wherein denotes a text classifier employing a cross-entropy (CE) based classification loss Train from scratch: wherein represents a class label of the target instance; Finally, the semantic similarity between the text and the 3D instance in the category dimension is calculated and the normalized category semantic distribution with the probability mass function :​ wherein denotes a matrix multiplication, denotes a softmax operation; Multi-expert aggregation process: according to the characteristics of the current sample, the input distribution from the more accurate expert model is automatically given greater weight, highlighting the target instance while excluding interference information.

2. The method of claim 1, wherein, The instance extraction process specifically comprises: Given a point cloud scene containing points, each point has 3D coordinates and RGB information, extract 3D instances in the scene, represented as instance masks and class score prediction , where represents the number of extracted 3D instances, represents the number of classes; Given a textual description of a sentence containing words, denoted as , for describing target instances.

3. The method of claim 1, wherein: The multi-expert aggregation process is assuming a finite number of probability distributions The fusion is merging these distributions into a single probability distribution and denoted by ; assuming are discrete, respectively, with probability mass functions and a common atom where ; then exists and the following equations are equivalent: （i） ； (ii) ; where denotes the Dirac delta measure in the set of real Borel probability measures located at the point , i.e. if then ; if then ; replacing the distribution by the normalized class semantic distribution , the normalized attribute semantic distribution and the normalized complete context semantic distribution , we obtain: wherein the common atoms refers to 3D instances ; Using the properties of Borel sets, rewrite as a probability mass function : wherein represents a 3D instance, represents a merging of multi-dimensional semantic distributions probability mass function; Finally, the target instance is predicted from the sampled in the middle : wherein, represents a sampling operation defined as selecting the target from the top proposals with the highest probability mass, wherein .

Images