A detailed three-dimensional directional target segmentation method

Abstract

The present invention discloses a detailed three-dimensional directional target segmentation method, comprising the following steps: S1, defining a task form and defining task evaluation indicators; S2, modifying and enhancing a ScanRefer dataset by combining manual and large models to generate a DetailRefer dataset; S3, constructing a DetailBase baseline model, and segmenting language at the sentence level or phrase level through the DetailBase baseline model; by defining the task form, generating the DetailRefer dataset, and constructing the DetailBase baseline model, the method can enhance the ability to understand and locate text context in 3D vision and language tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data processing, and in particular relates to a detailed three-dimensional directional target segmentation method. Background Art

[0002] Vision-language interaction stands at the forefront of computer vision research, enabling machines to understand and reason about the visual world using natural language. In recent years, with the rapid advancement of 3D sensing technologies and deep learning models, 3D vision-language tasks have become a key research focus in this field. These tasks play an important role in key applications such as robotics, autonomous navigation, mixed reality, and assistive technologies, where understanding the relationship between the 3D environment and natural language is essential. By combining recent advances, researchers can now develop smarter and more intuitive human-computer interaction systems that can not only "see" their surroundings but also interpret and communicate this information in a human-understandable way. This opens up new possibilities in a variety of areas, from improving the quality of life for people with disabilities to enhancing the realism of virtual reality experiences.

[0003] Among these tasks, 3D visual localization (3D-VG) tasks are particularly important because they require locating objects in 3D scenes based on text instructions - a fundamental capability for physical AI and autonomous systems. The field has been systematically developed, initially with 3D directed object localization (3D-REC), which uses a rough 3D bounding box to locate objects and formulates the problem as a coordinate regression (e.g. Figure 1 Subsequently, to meet the needs of finer-grained positioning, 3D directional target segmentation (3D-RES) was introduced, which requires point-level segmentation and transforms the task into an expression-to-point matching problem (e.g. Figure 1 (b) in

[15] ). However, both methods are limited to processing only one-to-one mapping between sentences and objects, which restricts their practical applicability in situations where instructions may involve multiple objects or no objects at all. To address this limitation, generalized 3D directed object segmentation (3D-GRES) (e.g. Figure 1 Part (c) of the paper expands the task description to accommodate situations where each text describes zero, one, or multiple objects. These developments have significantly advanced the field of 3D-VG, making it a cornerstone of 3D scene understanding. This approach enables researchers to create more flexible and powerful systems that can better understand and respond to complex natural language instructions, demonstrating great potential across multiple application areas.

[0004] Despite these significant advances, current 3D-VG tasks are still limited to sentence-level objectives, which hinders a comprehensive evaluation of the model's fine-grained language understanding capabilities. This sentence-centric approach fundamentally limits the modeling of sentence internal relations and semantic structures. Figure 1 In part (b) of the figure, traditional 3D-RES methods cannot provide a mechanism to determine whether the model has correctly understood the various elements of a single instruction. This limitation creates a critical gap in interpretability because effective referential expression understanding inherently relies on the ability to reason about the context in the text. Limited by the sentence-level supervision paradigm, current methods have difficulty developing these necessary contextual reasoning skills. Therefore, a more detailed, phrase-level understanding is needed to enable 3D vision-language models to develop strong contextual reasoning capabilities, which represents an important unexplored opportunity in this field. By introducing this more refined level of understanding, researchers can better evaluate and improve the performance of models when processing complex natural language instructions, thereby advancing the development of 3D-VG technology. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a detailed three-dimensional directional target segmentation method.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A detailed three-dimensional directional target segmentation method includes the following steps:

[0008] S1. Define the task form and the evaluation indicators of the task;

[0009] In step S1, the task form is defined to segment the mask corresponding to each target phrase given in the sentence from the point cloud scene. The specific process is:

[0010] S11. Given a point cloud scene ,in, is the number of points, is the characteristic length; the characteristic length includes coordinates XYZ, color RGB and normal vector;

[0011] S12. Given a text description ,in, is the number of words in the text;

[0012] S13. Given a set of indexes ,in, For the text The index position of the target phrase to be segmented; the index corresponds to the position of the target phrase to be segmented in the text. The model outputs a point cloud scene mask for each noun, which is used to understand the natural language description and mark the corresponding object or area in three-dimensional space.

[0013] S2, modify and enhance the ScanRefer dataset by combining manual and large models to generate the DetailRefer dataset;

[0014] S3. Build a DetailBase baseline model and use it to segment language at the sentence or phrase level.

[0015] In step S3, the specific processing process of the DetailBase baseline model is as follows:

[0016] S31. Input point cloud scene , text description and the index of the nouns that need to be split , by inputting the point cloud into the 3D U-Net network to obtain point-level features; where the point cloud only uses the coordinates XYZ and color RGB as the initial features of each point;

[0017] S32, use super point pooling to simplify the point-level features, and Perform unsupervised over-segmentation to generate Super points, among which ;

[0018] S33, average the features of all points belonging to the same superpoint, and then transform the pooled features into visual features for multimodal information fusion through two independent linear transformations and super-point features for predicting masks ,in, Represents feature dimension;

[0019] S34. For a given text description , after adding special words at the beginning and end, input it into the MPNet network to obtain word features, and use the word features to generate the initial query through the multi-layer perceptron ;

[0020] S35, the initial query The input is sent to the decoder, where cross-attention is used to integrate information from the visual modality, self-attention is used to focus on information within the sentence, and nonlinear transformation is performed through a feedforward neural network;

[0021] S36. Calculate the query and super-point features output by the last layer The affinity between them is calculated, and the affinity is binarized to obtain the super-point mask corresponding to the query, and then the super-point mask is broadcasted to obtain the point-level mask; for sentence-level segmentation, the mask corresponding to the [CLS] tag is used as the segmentation result; for phrase-level segmentation, the mask generated by the query corresponding to the position provided in the index is used as the segmentation result.

[0022] Preferably, in step S1, the specific process of defining the evaluation index of the task is:

[0023] S14. Focus on the average IoU, Acc@0.25, and Acc@0.5 at the phrase level. At the phrase level, the average IoU is the average of the IoUs calculated for all phrases to be segmented. Acc@0.25 and Acc@0.5 represent the proportion of all segmented phrases with an IoU greater than 0.25 and 0.5, respectively.

[0024] S15: Focus on the sentence-level average IoU. That is, after calculating and averaging the phrase IoU within each sentence, we average it across all sentences at the dataset level.

[0025] S16. Focus on the model's performance on long texts and complex scene descriptions. Define texts containing more than 50 words as long texts, and evaluate the model's indicators on long texts to reflect its ability to understand long texts. Define texts that need to be segmented into four or more phrases as complex scene descriptions, and test the model's indicators on complex scene descriptions to reflect its ability to understand complex scene descriptions.

[0026] Preferably, the specific process of step S2 is:

[0027] S21. In the first stage, a piece of code is written in Python to partition the descriptions in the ScanRefer dataset so that all descriptions of the same object are grouped together. Then, the Large Language Model (LLM) is used to combine the sentences into a more comprehensive new description. After obtaining a threshold number of new descriptions, all noun phrases in each text are manually annotated, and all noun phrases are associated with specific objects in the 3D scene. During the annotation process, inaccuracies in the new descriptions are corrected to achieve the modification of the ScanRefer dataset.

[0028] S22. In the second stage, the annotated text is converted into a format that the LLM large language model can understand. A bracket is added immediately after each noun phrase, and the object ID corresponding to the noun phrase is placed in the bracket. The sentence is then input into the LLM large language model, and the LLM large language model is instructed to generate several different expressions while retaining the original semantic meaning and keeping the object ID closely following the corresponding noun phrase. This is used to expand the dataset to five times its original size, thereby enhancing the ScanRefer dataset.

[0029] S23. Traverse each object mentioned in the enhanced ScanRefer dataset, extract all texts involving the object from the dataset of the first stage, and input the text into the LLM large language model for integration to generate text with a broader description and obtain the DetailRef dataset.

[0030] Preferably, the threshold number in step S21 is 10,000.

[0031] Preferably, the decoder in step S35 adopts a serial multi-layer architecture, and the decoder includes multiple cross-attention, self-attention and feedforward neural network structures.

[0032] After adopting the above technical solution, the present invention has the following beneficial effects:

[0033] 1. This paper proposes a new task, Detailed 3D Directed Object Segmentation (3D-DRES), which is a novel fine-grained visual localization task designed to enhance the ability to understand and localize text context in 3D vision and language tasks. This task requires the model to segment the mask corresponding to each noun phrase in the sentence. Specifically, Figure 1 As shown in part (d), the dataset provides the locations of all phrases to be segmented within a sentence (e.g., "trash can," "table," and "television"). The model needs to generate a corresponding mask for each phrase. The nature of the 3D-DRES task dictates that models trained on this task focus more on fine-grained semantics within a sentence. This characteristic complements the 3D-RES task, which emphasizes sentence-level semantics. Furthermore, 3D-RES and 3D-DRES can mutually enhance each other, as demonstrated in experiments.

[0034] 2. This paper constructs a new dataset, called DetailRefer, based on the ScanRefer dataset, combining detailed manual annotation with the assistance of a large language model. The DetailRefer dataset contains 55,432 descriptions covering 11,054 different objects, with an average text length of 24.9 words, significantly exceeding existing datasets (9.7 to 20.1 words). Unlike the sentence-segmentation annotation format commonly used in current datasets, where each sentence corresponds to one segmentation result, the DetailRefer dataset introduces a groundbreaking phrase-segmentation format, in which each noun phrase corresponds to a separate segmentation mask. This results in an unprecedented dataset density of 2.9 masks per paragraph of text. The dataset strategically incorporates 7.4% of "long" text (over 50 words) and a large number of "complex" examples requiring segmentation of four or more noun phrases, providing strong support for evaluating models' fine-grained language understanding capabilities in 3D environments.

[0035] 3. This paper addresses the fact that most existing models are designed for the "sentence-segmentation" sample format and are unable to output multiple masks or specify masks for specific tokens. This limitation prevents current methods from being directly applied to the 3D-DRES task. Therefore, as the initiator of this task, we propose the DetailBase baseline model, a deliberately simplified yet effective baseline model, to lay the foundation for future research. Our design philosophy prioritizes simplicity to ensure high scalability and adaptability, while demonstrating sufficient effectiveness to validate the potential of the task. The DetailBase baseline model has an elegant architecture that supports both sentence-level and phrase-level segmentation. Results show that training on this fine-grained task leads to significant improvements even on the traditional 3D-RES benchmark, demonstrating that phrase-level understanding can enhance overall spatial reasoning capabilities. By introducing the DetailBase baseline model, we not only provide a strong starting point for the 3D-DRES task but also demonstrate the potential of fine-grained language understanding to improve performance in 3D vision tasks. This paves the way for further exploration and development of more advanced 3D vision-language models, advancing the field. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A comparison chart of targets for different tasks;

[0037] Figure 2 is a flow chart of the present invention;

[0038] Figure 3 This is a framework diagram of the DetailBase baseline model of the present invention;

[0039] Figure 4This is a visualization comparison result diagram of the DetailBase baseline model and the 3D-STMN model of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] like Figures 1 to 4 As shown in FIG, a detailed three-dimensional directional target segmentation method (abbreviated as 3D-DRES) includes the following steps:

[0042] S1. Define the task form and the evaluation indicators of the task;

[0043] In step S1, the task form is defined to segment the mask corresponding to each target phrase given in the sentence from the point cloud scene. The specific process is:

[0044] S11. Given a point cloud scene ,in, is the number of points, is the characteristic length; the characteristic length includes coordinates XYZ, color RGB and normal vector;

[0045] S12. Given a text description ,in, is the number of words in the text;

[0046] S13. Given a set of indexes ,in, For the text The index position of the target phrase to be segmented; the index corresponds to the position of the target phrase to be segmented in the text. The model outputs a point cloud scene mask for each noun, which is used to understand the natural language description and mark the corresponding object or area in three-dimensional space.

[0047] In step S1, the specific process of defining the evaluation indicators of the task is as follows:

[0048] S14. Focus on the average IoU, Acc@0.25, and Acc@0.5 at the phrase level. At the phrase level, the average IoU is the average of the IoUs calculated for all phrases to be segmented. Acc@0.25 and Acc@0.5 represent the proportion of all segmented phrases with an IoU greater than 0.25 and 0.5, respectively.

[0049] S15: Focus on the sentence-level average IoU. That is, after calculating and averaging the phrase IoU within each sentence, we average it across all sentences at the dataset level.

[0050] S16. Focus on the model's performance on long texts and complex scene descriptions. Define texts containing more than 50 words as long texts, and evaluate the model's indicators on long texts to reflect its ability to understand long texts. Define texts that require segmentation into four or more phrases as complex scene descriptions, and test the model's indicators on complex scene descriptions to reflect its ability to understand complex scene descriptions.

[0051] S2, modify and enhance the ScanRefer dataset by combining manual and large models to generate the DetailRefer dataset;

[0052] The specific process of step S2 is:

[0053] S21. In the first stage, a piece of code is written in Python to partition the descriptions in the ScanRefer dataset so that all descriptions of the same object are grouped together. Then, the Large Language Model (LLM) is used to combine the sentences into a more comprehensive new description. After obtaining a threshold number of new descriptions, all noun phrases in each text are manually annotated, and all noun phrases are associated with specific objects in the 3D scene. During the annotation process, inaccuracies in the new descriptions are corrected to achieve the modification of the ScanRefer dataset.

[0054] The threshold number in step S21 is 10000;

[0055] S22. In the second stage, the annotated text is converted into a format that the LLM large language model can understand. A bracket is added immediately after each noun phrase, and the object ID corresponding to the noun phrase is placed in the bracket. The sentence is then input into the LLM large language model, and the LLM large language model is instructed to generate several different expressions while retaining the original semantic meaning and keeping the object ID closely following the corresponding noun phrase. This is used to expand the dataset to five times its original size, thereby enhancing the ScanRefer dataset.

[0056] S23, traverse each object mentioned in the enhanced ScanRefer dataset, extract all texts related to the object from the dataset of the first stage, and input the text into the LLM large language model for integration to generate text with a broader description, and obtain the DetailRef dataset;

[0057] S3. Build a DetailBase baseline model and use it to segment language at the sentence or phrase level.

[0058] In step S3, the specific processing process of the DetailBase baseline model is as follows:

[0059] S31. Input point cloud scene , text description and the index of the nouns that need to be split , by inputting the point cloud into the 3D U-Net network to obtain point-level features; where the point cloud only uses the coordinates XYZ and color RGB as the initial features of each point;

[0060] S32, use super point pooling to simplify the point-level features, and Perform unsupervised over-segmentation to generate Super points, among which ;

[0061] S33, average the features of all points belonging to the same superpoint, and then transform the pooled features into visual features for multimodal information fusion through two independent linear transformations and super-point features for predicting masks ,in, Represents feature dimension;

[0062] S34. For a given text description , after adding special words at the beginning and end, input it into the MPNet network to obtain word features, and use the word features to generate the initial query through the multi-layer perceptron ;

[0063] S35, the initial query The input is sent to the decoder, where cross-attention is used to integrate information from the visual modality, self-attention is used to focus on information within the sentence, and nonlinear transformation is performed through a feedforward neural network;

[0064] The decoder in step S35 adopts a serial multi-layer architecture, and the decoder includes multiple cross-attention, self-attention and feedforward neural network structures;

[0065] S36. Calculate the query and super-point features output by the last layer The affinity between them is calculated, and the affinity is binarized to obtain the super-point mask corresponding to the query, and then the super-point mask is broadcasted to obtain the point-level mask; for sentence-level segmentation, the mask corresponding to the [CLS] tag is used as the segmentation result; for phrase-level segmentation, the mask generated by the query corresponding to the position provided in the index is used as the segmentation result.

[0066] Performance testing:

[0067] Given that no existing models are directly adaptable to the task of our method for detailed three-dimensional directional object segmentation (3D-DRES), two existing models, the PNG model and the 3D-STMN model, were adapted to suit the task. The PNG model originates from the 2D field of panoramic narrative localization, a task very similar to ours. First, its input is modified by using point clouds instead of images. SPFormer is then used to extract instance candidates, and the point features within each instance are averaged as instance features. The result matching method is also modified, from selecting the candidate with the highest matching similarity to selecting candidates with matching similarity above a certain threshold. This adjustment is due to the fact that in this task, a single noun phrase may correspond to multiple instances. For the 3D-STMN model, its supervision and result generation methods were modified to align with the DetailBase dataset. Furthermore, in the original 3D-STMN model, the 3D-UNet is frozen. To ensure fairness, it is included in the training process here.

[0068] Table 1 shows the quantitative comparison results of the PNG model, 3D-STMN model and DetailBase baseline model on the DetailRefer dataset. It can be seen that although the modified PNG model achieved an average intersection-over-union (IoU) of 40.4 on the test set, it still lags significantly behind the other models. The reason for this is that the current 3D instance segmentation network is not ideal; such two-stage models usually perform worse than single-stage models. The modified 3D-STMN model demonstrated its capabilities in this task, but due to the low compatibility of its design modules with the task, its performance was not as good as the DetailBase baseline model proposed in this paper. The DetailBase baseline model achieved an mIoU of 55.7 on the test set, which laid a good foundation for the further development of the 3D-DRES method. In addition, the model framework of the present invention is relatively simple and highly scalable, making it very suitable as an initial method for this task.

[0069] Table 1: Quantitative comparison results of the PNG model, 3D-STMN model, and DetailBase baseline model on the DetailRefer dataset

[0070]

[0071] Phrase-level segmentation emphasizes fine-grained semantic understanding, while sentence-level segmentation focuses on holistic understanding. These two approaches are not mutually exclusive, but rather complementary. Here, a joint training experiment was conducted to validate this concept. By treating the word [CLS] (which serves as the root node in the 3D-STMN model) as the "noun phrase" to be segmented, the format of the two tasks was unified. The models were trained separately on the ScanRefer and DetailRefer datasets, and jointly on these two datasets. The average intersection-over-union (IoU) of the models on the validation set is reported in Table 2. Clearly, joint training yields better results for both the 3D-STMN model and the DetailBase baseline model compared to separate training. Notably, joint training significantly improves the performance of the 3D-RES task, with the DetailBase baseline model's score increasing by 2.8 points and the 3D-STMN model's score increasing by as much as 3.2 points. In summary, the tasks proposed in this paper not only have their own unique value but also can synergize and complement traditional tasks.

[0072] Table 2: Comparison results between the DetailBase baseline model and the 3D-STMN model under different training strategies

[0073]

[0074] exist Figure 4 The visual comparison results of the DetailBase baseline model and the 3D-STMN model are shown in Figure 2. Figure 4 In the example, we can more intuitively perceive the advantages of the 3D-DRES task. Figure 4 As shown in part (a), if it is a traditional 3D-RES task, the text will point to the object "toilet", and both models have successfully segmented the target. In traditional 3D-RES tasks, since the results are only evaluated based on a single entity in the text, it is difficult to evaluate the model's fine-grained understanding of the entire text. However, under the 3D-DRES task setting of the present invention, the model's overall understanding of the text can be observed in more detail. For example, the 3D-STMN model has a better understanding of the text. Figure 4 The understanding of part (b) is completely wrong, and the text Figure 4 The understanding of part (c) is partly correct and partly incorrect.

[0075] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A detailed three-dimensional directional target segmentation method, characterized in that: The following steps are involved: S1. Define the task form and the evaluation indicators of the task; In step S1, the task form is defined to segment the mask corresponding to each target phrase given in the sentence from the point cloud scene. The specific process is: S11. Given a point cloud scene ,in, is the number of points, is the characteristic length; the characteristic length includes coordinates XYZ, color RGB and normal vector; S12. Given a text description ,in, is the number of words in the text; S13. Given a set of indexes ,in, For the text The index position of the target phrase to be segmented; the index corresponds to the position of the target phrase to be segmented in the text. The model outputs a point cloud scene mask for each noun, which is used to understand the natural language description and mark the corresponding object or area in three-dimensional space. S2, modify and enhance the ScanRefer dataset by combining manual and large models to generate the DetailRefer dataset; S3. Build a DetailBase baseline model and use it to segment language at the sentence or phrase level. In step S3, the specific processing process of the DetailBase baseline model is as follows: S31. Input point cloud scene , text description and the index of the nouns that need to be split , by inputting the point cloud into the 3DU-Net network to obtain point-level features; where the point cloud only uses the coordinates XYZ and color RGB as the initial features of each point; S32, use super point pooling to simplify the point-level features, and Perform unsupervised over-segmentation to generate Super points, among which ; S33, average the features of all points belonging to the same superpoint, and then transform the pooled features into visual features for multimodal information fusion through two independent linear transformations and super-point features for predicting masks ,in, Represents feature dimension; S34. For a given text description , after adding special words at the beginning and end, input it into the MPNet network to obtain word features, and use the word features to generate the initial query through the multi-layer perceptron ; S35, the initial query The input is sent to the decoder, where cross-attention is used to integrate information from the visual modality, self-attention is used to focus on information within the sentence, and nonlinear transformation is performed through a feedforward neural network; S36. Calculate the query and super-point features output by the last layer The affinity between them is calculated, and the affinity is binarized to obtain the super-point mask corresponding to the query, and then the super-point mask is broadcasted to obtain the point-level mask; for sentence-level segmentation, the mask corresponding to the [CLS] tag is used as the segmentation result; for phrase-level segmentation, the mask generated by the query corresponding to the position provided in the index is used as the segmentation result.

2. A detailed three-dimensional directional target segmentation method according to claim 1, characterized in that: In step S1, the specific process of defining the evaluation indicators of the task is as follows: S14. Focus on the average IoU, Acc@0.25, and Acc@0.5 at the phrase level. At the phrase level, the average IoU is the average of the IoUs calculated for all phrases to be segmented. Acc@0.25 and Acc@0.5 represent the proportion of all segmented phrases with an IoU greater than 0.25 and 0.5, respectively. S15: Focus on the sentence-level average IoU. That is, after calculating and averaging the phrase IoU within each sentence, we average it across all sentences at the dataset level. S16. Focus on the model's performance on long texts and complex scene descriptions. Define texts containing more than 50 words as long texts, and evaluate the model's indicators on long texts to reflect its ability to understand long texts. Define texts that need to be segmented into four or more phrases as complex scene descriptions, and test the model's indicators on complex scene descriptions to reflect its ability to understand complex scene descriptions.

3. A detailed three-dimensional directional target segmentation method as claimed in claim 1, characterized in that: The specific process of step S2 is: S21. In the first stage, a piece of code is written in Python to partition the descriptions in the ScanRefer dataset so that all descriptions of the same object are grouped together. Then, the Large Language Model (LLM) is used to combine the sentences into a more comprehensive new description. After obtaining a threshold number of new descriptions, all noun phrases in each text are manually annotated, and all noun phrases are associated with specific objects in the 3D scene. During the annotation process, inaccuracies in the new descriptions are corrected to achieve the modification of the ScanRefer dataset. S22. In the second stage, the annotated text is converted into a format that the LLM large language model can understand. A bracket is added immediately after each noun phrase, and the object ID corresponding to the noun phrase is placed in the bracket. The sentence is then input into the LLM large language model, and the LLM large language model is instructed to generate several different expressions while retaining the original semantic meaning and keeping the object ID closely following the corresponding noun phrase. This is used to expand the dataset to five times its original size, thereby enhancing the ScanRefer dataset. S23. Traverse each object mentioned in the enhanced ScanRefer dataset, extract all texts involving the object from the dataset of the first stage, and input the text into the LLM large language model for integration to generate text with a broader description and obtain the DetailRef dataset.

4. A detailed three-dimensional directional target segmentation method as claimed in claim 3, characterized in that: The threshold number in step S21 is 10,000.

5. The detailed three-dimensional directional target segmentation method according to claim 1, wherein: The decoder in step S35 adopts a serial multi-layer architecture, and the decoder includes multiple cross-attention, self-attention and feedforward neural network structures.

Images