Three-dimensional directivity target segmentation method under weak supervision setting

Abstract

The invention discloses a three-dimensional directivity target segmentation method under weak supervision setting, and relates to the technical field of three-dimensional directivity target segmentation. The method comprises the following steps: an instance extraction process: obtaining a point cloud scene and text description, extracting 3D instances in the scene, and representing the 3D instances as instance masks and category score prediction; in the multi-expert mining process, information is extracted from semantic clues of three dimensions of complete context, attribute and category through a plurality of expert models, and semantic distribution of targets is calculated respectively; and a multi-expert aggregation process: according to characteristics of a current sample, automatically endowing input distribution from a more accurate expert model with a larger weight, and eliminating interference information while highlighting a target instance. According to the three-dimensional directivity target segmentation method under weak supervision setting provided by the invention, only the three-dimensional point cloud data and the corresponding directivity description are provided, and mask annotation is not provided, so that time and energy do not need to be consumed for mask annotation.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional directional target segmentation, and in particular to a three-dimensional directional target segmentation method under weak supervision setting. Background Art

[0002] 3D Referring Expression Segmentation (3D-RES) has attracted much attention in the multimodal field. It aims to segment the target 3D instance by aligning the directional description with the point cloud and generate accurate target segmentation masks. The benchmark datasets used for the 3D-RES task mainly include Scanrefer, ReferIt3D and Multi3DRefer.

[0003] Existing 3D-RES methods rely on a fully supervised setting that requires expensive point-level mask annotations. For example, it takes an average of 22.3 minutes to annotate a point cloud scene in the ScanNet-v2 dataset. The labor-intensive nature of these annotations hinders the collection of large-scale 3D datasets, limiting the scalability of this task in data-scarce domains. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a three-dimensional directional target segmentation method under a weak supervision setting, which only requires providing three-dimensional point cloud data and corresponding directional descriptions, without providing mask annotations, thereby eliminating the need to spend time and energy on mask annotations.

[0005] The present invention provides a three-dimensional directional target segmentation method under weak supervision setting, comprising: Instance extraction process: obtain point cloud scenes and text descriptions, extract 3D instances in the scene, and represent them as instance masks and category score predictions; Multi-expert mining process: Multiple expert models are used to extract information from semantic clues in three dimensions: full context, attributes, and categories, and the semantic distribution of the target is calculated respectively. Among them, the full context expert processes the complete visual and textual context, and evaluates the matching degree between the instance and the description from the perspective of spatial relationship and the whole scene. The attribute expert focuses on the details of the instance, such as shape, color, texture, etc., and distinguishes the instance at the attribute level. The category expert focuses on the semantic category matching between the instance and the text description. Multi-expert aggregation process: According to the characteristics of the current sample, a larger weight is automatically given to the input distribution from a more accurate expert model, highlighting the target instance while excluding interference information.

[0006] Furthermore, the example extraction process specifically includes: Given a containing Point cloud scene with 3D coordinates and RGB information for each point. 3D instances in the scene are extracted and represented as instance masks. And the category score prediction ,in Indicates the number of 3D instances extracted, Indicates the number of categories; Given a sentence containing The text description of words is represented as , used to describe the target instance.

[0007] Furthermore, in the multi-expert mining process, the complete context expert model is specifically: Given an input description , use the pre-trained Long-CLIP text encoder to extract complete contextual text features :

[0008] in, Represents the text encoder of Long-CLIP, which is the feature dimension; Given a point cloud scene with Raw RGB-D video of the frame ,in , Represent the width and height of the image respectively. First, all 3D instances are projected onto all frame sequences to calculate the visibility of each 3D instance on each frame. Then, according to the visibility, the K frames with the highest visibility are selected as the visual representation of the instance, which is expressed as ,in, Indicates the number of 3D instances extracted, , Represents the width and height of the image; these 2D frames are feature extracted by the pre-trained Long-CLIP image encoder, and then the 2D complete context features are obtained by average pooling operation :

[0009] in, represents the Long-CLIP image encoder, represents the average pooling operation; Finally, the semantic similarity of the complete context dimension is calculated and a probability mass function Normalized full context semantic distribution of :

[0010]

[0011] in represents matrix multiplication, Represents the softmax operation.

[0012] Furthermore, in the multi-expert mining process, the attribute expert model is specifically: Given an input description , using a natural language parser to decompose it into five semantic components: main object, auxiliary object, attribute, pronoun, and relationship; then extract the main object and its attributes, which are represented as target words and modifiers ; The target word and modifiers Concatenate to get a short attribute phrase , the phrase only contains the name of the target instance and its modifiers; then, the pre-trained CLIP text encoder is used to extract attribute text features :

[0013] in, Represents the CLIP text encoder; At the same time, an object renderer is used to render a multi-view 2D rendering of each 3D instance as its visual representation, which is expressed as ,in Indicates the number of 3D instances extracted, Indicates the number of rendering angles. , Represent the width and height of the image respectively; the 2D rendering image is feature extracted by the pre-trained CLIP image encoder, and then the 2D attribute features are obtained by average pooling operation. :

[0014] in represents the CLIP image encoder, represents the average pooling operation; Finally, the semantic similarity of the attribute dimension between the text and the 3D instance is calculated and a probability mass function Normalized attribute semantic distribution of :

[0015]

[0016] in represents matrix multiplication, Represents the softmax operation.

[0017] Furthermore, in the multi-expert mining process, the category expert model is specifically: Given an input description , extracting full contextual text features from the pre-trained Long-CLIP text encoder , and then the text category score prediction is obtained by the text classifier output :

[0018] in Represents a text classifier that uses a classification loss based on cross entropy (CE) Training from scratch:

[0019] in Represents the category label of the target instance; Finally, the semantic similarity of the category dimension between the text and the 3D instance is calculated and a probability mass function Normalized category semantic distribution of :

[0020]

[0021] in represents matrix multiplication, Represents the softmax operation.

[0022] Furthermore, the multi-expert aggregation process assumes a finite number of probability distributions The fusion is to combine these distributions into a single probability distribution , and use the symbol specify; assume are discrete, with probability mass functions and public atoms ,in ;So exists and the following are equivalent: (i) ; (ii) ; in It refers to the real Borel probability measure set located at point The Dirac delta measure of ,but ;if but ; Replace the distribution with the normalized category semantic distribution , Normalized attribute semantic distribution and normalized full context semantic distribution ,get:

[0023] The common atoms means 3D instances ; Using the properties of the Borel set, rewrite it as a probability mass function :

[0024] in represents a 3D instance, Represents the merging of multi-dimensional semantic distributions The probability mass function of Finally from Sampling to predict the target instance :

[0025] in, represents the sampling operation, which is defined as the sampling from the top Select the target from among the proposals .

[0026] The multi-expert mining proposed in the present invention can effectively mine and match the context of vision and text. At the same time, the multi-expert aggregation proposed in the present invention can effectively highlight the target instance and exclude interference information compared with the direct summation or collection method, thereby using multiple clues to complete text-based directional reasoning without providing mask annotations, thereby saving time and energy for mask annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0028] Figure 1 is an overall flow chart of the method of the embodiment of the present invention; Figure 2 It is a schematic diagram of a flow chart of an embodiment of the present invention; Figure 3 This is a schematic diagram of the working principle of the module of the embodiment of the present invention; Figure 4 It is a schematic diagram for visualizing the results of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiment of the present invention provides a three-dimensional directional target segmentation method under a weak supervision setting, which only requires providing three-dimensional point cloud data and corresponding directional descriptions without providing mask annotations, thereby eliminating the need to spend time and effort on mask annotations.

[0030] The technical solution in the embodiment of the present invention has the following general idea: The three-dimensional directional target segmentation method of the multi-expert network under the weak supervision setting of the present invention proposes a weakly supervised 3D-RES setting, in which only three-dimensional point cloud data and corresponding directional descriptions are provided, but mask annotations are not provided, so there is no need to spend time and energy on mask annotations. A direct solution is to first segment the point cloud scene using a pre-trained three-dimensional model, extract the three-dimensional instance and render it into a two-dimensional image, and then use mature 2D image-text matching technology (such as CLIP) to directly match it with the directional description. However, such a method isolates the three-dimensional instance from the scene and cannot accurately locate the target using the key context clues provided in the description. Another method is to select the two-dimensional image with the highest visibility for each three-dimensional instance from the video that constructs the three-dimensional point cloud, and then use CLIP for matching. Although this method can more effectively utilize the contextual information contained in the text, if the image contains multiple instances, it will lead to ambiguous matching. Therefore, the main challenge of weakly supervised 3D-RES is how to effectively mine and match visual and textual contexts while filtering potential interference factors.

[0031] To this end, the embodiment of the present invention proposes a Multi-Expert Network (MEN), a method for completing text-based directional reasoning using multiple clues. The method decomposes the problem into two subtasks: how to effectively mine and match the context of vision and text; how to merge the results of the first step to filter out interference information while highlighting the target instance. For the first subtask, the present invention introduces a Multi-Expert Mining (MEM) module, which designs experts who can extract information from multiple semantic clues, focusing on the three dimensions of complete context, attributes and categories, and respectively calculates the semantic distribution of the target. Among them, the full context expert processes the complete visual and textual context, and evaluates the degree of match between the instance and the description from the perspective of spatial relationship and the entire scene; the attribute expert focuses on the details of the instance such as shape, color, texture, etc., and distinguishes the instance at the attribute level; the category expert focuses on the semantic category matching between the instance and the text description. For the second subtask, the present invention proposes a Multi-Expert Aggregation (MEA) module, which automatically gives a greater weight to the input distribution from more accurate experts. Compared with the direct summation or collection method, MEA can effectively highlight the target instance and exclude interference information.

[0032] This embodiment provides a three-dimensional directional target segmentation method under a weak supervision setting, such as Figure 1 and Figure 2 As shown, including: S1. Instance extraction process: obtain the point cloud scene and text description, extract the 3D instances in the scene, and represent them as instance masks and category score predictions.

[0033] Given a containing Point cloud scene with points, each point has 3D coordinates and RGB information. The pre-trained SoftGroup is used to extract 3D instances in the scene, represented as instance masks And the category score prediction ,in Indicates the number of 3D instances extracted, Indicates the number of categories.

[0034] Given a sentence containing The text description of words is represented as , used to describe the target instance.

[0035] S2, multi-expert mining process: multiple expert models are used to extract information from semantic clues in three dimensions: full context, attributes, and categories, and the semantic distribution of the target is calculated respectively; the full context expert processes the complete visual and textual context, and evaluates the degree of match between the instance and the description from the perspective of spatial relationship and the entire scene; the attribute expert focuses on the details of the instance such as shape, color, texture, etc., and distinguishes the instance at the attribute level; the category expert focuses on the semantic category match between the instance and the text description. This process aims to extract information from semantic clues in three different dimensions and calculate the semantic distribution of the target.

[0036] (1) Full context expert model Given an input description , use the pre-trained Long-CLIP text encoder to extract complete contextual text features : (1) in, Represents the text encoder of Long-CLIP, which is the feature dimension.

[0037] Given a point cloud scene with Raw RGB-D video of the frame ,in , Represent the width and height of the image respectively. First, all 3D instances are projected onto all frame sequences to calculate the visibility of each 3D instance on each frame. Then, according to the visibility, the K frames with the highest visibility are selected as the visual representation of the instance, which is expressed as ,in, Indicates the number of 3D instances extracted, , Represents the width and height of the image; these 2D frames are feature extracted by the pre-trained Long-CLIP image encoder, and then the 2D complete context features are obtained by average pooling operation : (2) in, represents the Long-CLIP image encoder, Represents an average pooling operation.

[0038] Finally, the semantic similarity of the complete context dimension is calculated and a probability mass function Normalized full context semantic distribution of : (3) (4) in represents matrix multiplication, Represents the softmax operation.

[0039] (2) Attribute Expert Model Given an input description , using the natural language parser SceneGraphParser to decompose it into five different semantic components: the main object, the auxiliary object, attributes, pronouns, and relationships; then the main object and its corresponding attributes are extracted and represented as target words respectively. and modifiers ; These words will be concatenated to form a short attribute phrase , the phrase only contains the name of the target instance and its modifiers; then, the pre-trained CLIP text encoder is used to extract attribute text features : (5) in, Represents the CLIP text encoder.

[0040] At the same time, an object renderer is used to render a multi-view 2D rendering of each 3D instance as its visual representation, which is expressed as ,in Indicates the number of 3D instances extracted, Indicates the number of rendering angles. , Represent the width and height of the image respectively; the 2D rendering image is feature extracted by the pre-trained CLIP image encoder, and then the 2D attribute features are obtained by average pooling operation. : (6) in represents the CLIP image encoder, represents the average pooling operation; Finally, the semantic similarity of the attribute dimension between the text and the 3D instance is calculated and a probability mass function Normalized attribute semantic distribution of : (7) (8) in represents matrix multiplication, Represents the softmax operation.

[0041] (3) Category Expert Model Given an input description , extracting full contextual text features from the pre-trained Long-CLIP text encoder , and then the text category score prediction is obtained by the text classifier output : (9) in Represents a text classifier that uses a classification loss based on cross entropy (CE) Training from scratch: (10) in Represents the category label of the target instance; Finally, the semantic similarity of the category dimension between the text and the 3D instance is calculated and a probability mass function Normalized category semantic distribution of : (11) (12) in represents matrix multiplication, Represents the softmax operation.

[0042] S3, multi-expert aggregation process: according to the characteristics of the current sample, automatically give greater weight to the input distribution from the more accurate expert model, highlight the target instance while excluding interference information.

[0043] The multi-expert aggregation module mathematically assigns greater weights to input distributions from more accurate experts, especially those with smaller standard deviations. The final distribution obtained by the multi-expert aggregation module is both the only minimum likelihood ratio combination and the only proportional likelihood ratio combination for a given input distribution, ensuring a robust integration of expert predictions.

[0044] To this end, we first analyze the advantages and disadvantages of the above three types of experts: Full context experts are good at capturing complete spatial contexts, but may introduce interference from complex two-dimensional video frames, so they are suitable for scenarios where spatial relationships are critical to identifying targets; attribute experts focus on the visual attributes of instances and the corresponding text descriptions, but may be affected by rendering accuracy and parsing tools, so they are very suitable for situations where the attributes of the target are unique and distinct; category experts use the classification capabilities of SoftGroup to effectively exclude irrelevant objects that do not match the target category, showing strong anti-interference capabilities, but often lack distinction between objects of the same type, and are suitable for situations where the target category is unique. The multi-expert aggregation module can allow more accurate experts to play a greater role based on the characteristics of the current sample.

[0045] Assume a finite number of probability distributions The fusion is to combine these distributions into a single probability distribution , and use the symbol specify; assume are discrete, with probability mass functions and public atoms ,in ;So exists and the following are equivalent: (i) ; (ii) ; in It refers to the real Borel probability measure set located at point The Dirac delta measure of ,but ;if but ; Replace the distribution with the normalized category semantic distribution , Normalized attribute semantic distribution and normalized full context semantic distribution ,get: (13) The common atoms means 3D instances .

[0046] Using the properties of the Borel set, this can be rewritten as a probability mass function : (14) in represents a 3D instance, Represents the merging of multi-dimensional semantic distributions The probability mass function of Finally from Sampling to predict the target instance : (15) in, represents the sampling operation, which is defined as the sampling from the top Select the target from among the proposals .

[0047] The beneficial effects of this embodiment are described below through specific experiments, and the specific experimental results are as follows: This example is trained and tested on the 3D directional target segmentation dataset ScanRefer, which contains 51,583 natural language expressions of 11,046 objects from 800 ScanNet scenes. All samples can be divided into two categories: 1) unique, where there are no other objects of the same category as the target in the scene; 2) multiple, where there are other objects of the same category as the target in the scene.

[0048] All experiments are implemented in PyTorch and trained on a single NVIDIA GeForce RTX 3090 GPU.

[0049] This embodiment uses the average intersection over union (mIoU) and Results and results exceeded The recall rate (R@ , IoU@ ) is used as the evaluation index of this embodiment. The intersection and union ratio of the first n results exceeds The recall rate indicates that the There are instances where the IoU between the predicted mask and the true annotation is greater than The ratio of the descriptive text of the instance, where , .

[0050] The following table shows the results of some 3D-REC and RES models in the 3D-GRES task. The weakly supervised 3D-RES Baseline of this embodiment is built on the basis of the first weakly supervised 3D-REC model, and some modifications are made, including changing the object detection model to the instance segmentation model SoftGroup, and excluding the components related to word reconstruction and distillation. The Baseline achieves the expected performance with an mIoU of 19.9%. This embodiment also randomly selects a target object from the same category according to the text category, called "Baseline-random". Compared with the baseline, this embodiment improves mIoU and R@1, IoU@0.5 by 13.5% and 15.2%, respectively, especially in the more challenging "multiple" class. The improvement of 13.9% shows that this embodiment can not only accurately identify simple "unique" targets, but also effectively handle the difficult situation where there are multiple interference objects similar to the target in the scene. This highlights how the MEM module of this embodiment effectively captures the complex semantics of the scene and descriptive text, while the MEA module effectively consolidates these advantages, providing strong generalization and robust discrimination capabilities. Therefore, in terms of the mIoU index, the weakly supervised method of this embodiment is significantly higher than the fully supervised 3D-RES method TGNN by 5.6%.

[0051] In order to directly extend the weak supervision setting to the fully supervised 3D-RES model, the MEN method of this embodiment can also be used as a teacher model to generate pseudo labels, such as target instance masks, to supervise the existing fully supervised 3D-RES model. The results are represented by "Ours_Model Name". As shown in Table 1, using the pseudo labels generated by MEN for supervision can help ordinary 3D-RES models achieve competitive performance, or even exceed the performance of MEN.

[0052] Table 1 3D-RES task results on ScanRefer

[0053] The embodiment of the present invention also performs a visual analysis of the principles of the MEM and MEA modules. Figure 3 The attention heatmaps and semantic distributions of category experts, attribute experts, and full-text experts are displayed to explore their respective roles. Figure 4As shown in (a), these experts each perform their own duties and assist the model in locating the target at different levels. The full context expert focuses on the corners of the scene that are relevant to the description, but cannot distinguish between the objects in these corners, such as "TV", "TV cabinet" and "blue object" shown in (a). The attribute expert follows the instruction of "rectangular gray TV cabinet", indicating interest in all objects that are "rectangular", "gray" or may be classified as "TV cabinet". Similarly, the category expert will pay attention to all objects in the scene that are similar to the target object category. Therefore, while relying solely on the information of the target object is suitable for scenes with unique categories and unique attributes, it proves to be ineffective in complex "multiple" scenes. It can be observed that only objects that show high similarity among all experts are likely to be predicted as targets. For example, the "blue object" near the target shows high attention in both "category" and "full context", but it has low attention in "attribute", which reduces the probability of it being selected as the target. In short, each expert has its own strengths and weaknesses. As shown in (b), the MEA module makes full use of the complementary advantages of the three experts, while filtering out distracting factors, thereby increasing the importance of the target object.

[0054] In addition, the embodiment of the present invention also shows the visualization results of the weakly supervised 3D-RES Baseline. Figure 4 As shown, the present invention successfully identifies objects that match the target category from geometrically similar objects, as shown in case (a). At the same time, the present invention can effectively use attribute information to distinguish similar objects, such as the "three seats" mentioned in the input description of case (b). In addition, the embodiment of the present invention can also infer the target object from multiple similar objects based on the complete context, as shown in case (c). In contrast, the weak supervision baseline has almost the same level of interest in all similar objects, resulting in inaccurate predictions. More noteworthy is that, as shown in case (d), even if the target belongs to the "other" category, the embodiment of the present invention can effectively eliminate interference and achieve precise positioning with the help of the effective integration of MEM and MEA modules.

[0055] The multi-expert mining proposed in the present invention can effectively mine and match the context of vision and text. At the same time, the multi-expert aggregation proposed in the present invention can effectively highlight the target instance and exclude interference information compared with the direct summation or collection method, thereby using multiple clues to complete text-based directional reasoning without providing mask annotations, thereby saving time and energy for mask annotation.

[0056] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional directional target segmentation method under weak supervision setting, characterized in that: include: Instance extraction process: obtain the point cloud scene and text description, extract the 3D instances in the scene, and represent them as instance masks and category score predictions; Multi-expert mining process: Multiple expert models are used to extract information from semantic clues in three dimensions: full context, attributes, and categories, and the semantic distribution of the target is calculated respectively. Among them, the full context expert processes the complete visual and textual context, and evaluates the matching degree between the instance and the description from the perspective of spatial relationship and the whole scene. The attribute expert focuses on the details of the instance, such as shape, color, texture, etc., and distinguishes the instance at the attribute level. The category expert focuses on the semantic category matching between the instance and the text description. Multi-expert aggregation process: According to the characteristics of the current sample, a larger weight is automatically given to the input distribution from a more accurate expert model, highlighting the target instance while excluding interference information.

2. The method according to claim 1, characterized in that The instance extraction process specifically includes: Given a containing Point cloud scene with 3D coordinates and RGB information for each point. 3D instances in the scene are extracted and represented as instance masks. And the category score prediction ,in Indicates the number of 3D instances extracted, Indicates the number of categories; Given a sentence containing The text description of words is represented as , used to describe the target instance.

3. The method according to claim 1, characterized in that In the multi-expert mining process, the complete context expert model is specifically: Given an input description , use the pre-trained Long-CLIP text encoder to extract complete contextual text features : in, Represents the text encoder of Long-CLIP, which is the feature dimension; Given a point cloud scene with Raw RGB-D video of the frame ,in , Represent the width and height of the image respectively. First, all 3D instances are projected onto all frame sequences to calculate the visibility of each 3D instance on each frame. Then, according to the visibility, the K frames with the highest visibility are selected as the visual representation of the instance, which is expressed as ,in, Indicates the number of 3D instances extracted, , Represents the width and height of the image; these 2D frames are feature extracted by the pre-trained Long-CLIP image encoder, and then the 2D complete context features are obtained by average pooling operation : in, represents the Long-CLIP image encoder, represents the average pooling operation; Finally, the semantic similarity of the complete context dimension is calculated and a probability mass function Normalized full context semantic distribution of : in represents matrix multiplication, Represents the softmax operation.

4. The method according to claim 1, characterized in that: In the multi-expert mining process, the attribute expert model is specifically: Given an input description , using a natural language parser to decompose it into five semantic components: main object, auxiliary object, attribute, pronoun, and relationship; then extract the main object and its attributes, which are represented as target words and modifiers ; The target word and modifiers Concatenate to get a short attribute phrase , the phrase consists only of the name of the target instance and its modifiers; Then, the pre-trained CLIP text encoder is used to extract attribute text features. : in, Represents the CLIP text encoder; At the same time, an object renderer is used to render a multi-view 2D rendering of each 3D instance as its visual representation, which is expressed as ,in Indicates the number of 3D instances extracted, Indicates the number of rendering angles. , Represent the width and height of the image respectively; the 2D rendering image is feature extracted by the pre-trained CLIP image encoder, and then the 2D attribute features are obtained by average pooling operation. : in represents the CLIP image encoder, represents the average pooling operation; Finally, the semantic similarity of the attribute dimension between the text and the 3D instance is calculated and a probability mass function Normalized attribute semantic distribution of : in represents matrix multiplication, Represents the softmax operation.

5. The method according to claim 1, characterized in that In the multi-expert mining process, the category expert model is specifically: Given an input description , extracting full contextual text features from the pre-trained Long-CLIP text encoder , and then the text category score prediction is obtained by the text classifier output : in Represents a text classifier that uses a classification loss based on cross entropy (CE) Training from scratch: in Represents the category label of the target instance; Finally, the semantic similarity of the category dimension between the text and the 3D instance is calculated and a probability mass function Normalized category semantic distribution of : in represents matrix multiplication, Represents the softmax operation.

6. The method according to claim 1, characterized in that: The multi-expert aggregation process assumes a finite probability distribution The fusion is to combine these distributions into a single probability distribution , and use the symbol specify; assume are discrete, with probability mass functions and public atoms ,in ;So exists and the following are equivalent: （i） ； (ii) ; in It refers to the real Borel probability measure set located at point The Dirac delta measure of ,but ;if but ; Replace the distribution with the normalized category semantic distribution , Normalized attribute semantic distribution and normalized full context semantic distribution ,get: The common atoms means 3D instances ; Using the properties of the Borel set, rewrite it as a probability mass function : in represents a 3D instance, Represents the merging of multi-dimensional semantic distributions The probability mass function of Finally from Sampling to predict the target instance : in, represents the sampling operation, which is defined as the sampling from the top Select the target from among the proposals .

Images