A point cloud segmentation method based on neighborhood homogeneity constraint and text guidance
By employing neighborhood homogeneity constraints and text-guided methods, key homogeneous points are selected for long-range self-attention enhancement and sparse mask generation. This solves the problem of imbalance between point cloud and text information density, improves the efficiency and accuracy of point cloud segmentation in large scenes, and is applicable to intelligent robots and autonomous driving scenarios.
Patent Information
- Application Number
- CN202610818986.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-08
- Publication Date
- 2026-08-25
AI Technical Summary
Existing point cloud segmentation frameworks based on large language models face problems such as low cross-modal alignment efficiency and insufficient mask segmentation accuracy when processing large-scale point clouds, due to the imbalance between point cloud information density and text information density. The accuracy drops significantly, especially when dealing with objects with complex geometric structures.
By employing neighborhood homogeneity constraints and text-guided methods, key homogeneous points are selected for long-range self-attention enhancement. Combined with sparse mask generation and dense mask propagation, a large language model is used to understand user intent. Furthermore, by combining point cloud local neighborhood distribution similarity calculation, efficient visual-text feature fusion and mask generation are achieved.
It significantly improves cross-modal alignment efficiency, reduces computational complexity, and enhances the boundary accuracy and geometric consistency of segmentation masks, achieving efficient and accurate open-text guided point cloud segmentation.
Smart Images
Figure CN122636638A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of interdisciplinary technology of 3D vision and natural language processing, and in particular to a point cloud segmentation method and system based on a large language model, which is especially suitable for engineering deployment scenarios that require natural language-guided segmentation of large-scale 3D point clouds, such as embodied intelligent navigation, autonomous driving environment perception, and robot operation. Background Technology
[0002] Point cloud data is a fundamental representation in 3D scene perception, directly acquired by sensors such as LiDAR and depth cameras. Each point contains 3D coordinates (x, y, z) and possible physical attributes such as color and reflection intensity. In autonomous driving, a single road test scan can generate tens of thousands to hundreds of thousands of point clouds, fully describing the spatial geometry of elements such as intersections, vehicles, pedestrians, and traffic signs. In embodied intelligence, service robots use point clouds to perceive the position and boundaries of indoor furniture, appliances, and obstacles in real time. Point cloud segmentation is the process of separating a specific target object's point set from these raw geometric measurement data, and it is a core component of 3D scene understanding.
[0003] In recent years, point cloud segmentation frameworks based on Large Language Models (LLMs) have become a research hotspot. The core idea is to leverage the powerful semantic reasoning capabilities of LLMs to enable the system to adaptively identify and finely segment target objects in point clouds based on natural language instructions (such as "segment out the red cushion on the sofa"). This capability has significant practical value for fields such as embodied intelligence and autonomous driving, because in real-world tasks, users often cannot predict all target categories, requiring the system to possess semantic understanding and geometric localization capabilities with open vocabularies.
[0004] However, existing LLM-based point cloud segmentation frameworks still face the following key technical challenges when processing large-scale point clouds:
[0005] First, the severe imbalance between point cloud information density and text information density leads to low efficiency in cross-modal alignment. Large-scale point clouds typically contain hundreds of thousands or even millions of points, with extremely high information density; while user-inputted text commands contain only a few to dozens of words, with extremely low information density. Existing methods require comprehensive interaction between high-dimensional point cloud features and low-dimensional text features when performing cross-modal attention calculations. The computational complexity increases quadratically with the size of the point cloud, making it difficult to meet the deployment requirements of practical engineering scenarios.
[0006] Second, the cross-level feature comparison method leads to insufficient mask segmentation accuracy. To alleviate computational pressure, existing methods typically perform significant downsampling of the point cloud and generate masks using a "compress first, then restore" strategy. This process inevitably loses geometric details, resulting in blurred segmentation boundaries and difficulty in aligning with the actual edges of the object. The accuracy drop is particularly noticeable when processing objects with complex geometric structures.
[0007] To address the aforementioned issues, developing an LLM-driven point cloud segmentation method that can alleviate the imbalance between point cloud and text information density and improve cross-modal alignment efficiency and mask segmentation accuracy has become an urgent technical challenge in this field. Summary of the Invention
[0008] This invention addresses the aforementioned problems in existing technologies by providing a point cloud segmentation method based on neighborhood homogeneity constraints and text guidance. This method can receive natural language text instructions and raw point clouds as input, utilizes the reasoning capabilities of a large language model to understand user intent, and achieves adaptive target segmentation based on user-specified requirements for specific scenarios. It identifies the targets to be identified within the scene and improves segmentation efficiency and accuracy by incorporating the homogeneity constraint of the local neighborhood distribution of the point cloud.
[0009] Step S1: Data preprocessing, reading raw point cloud data to complete downsampling, noise reduction, DEM construction, neighborhood construction and other data processing operations;
[0010] Step S2: Extract features from the pre-trained multi-layer architecture point cloud encoder on the pre-processed point cloud data and assign corresponding features to each point;
[0011] Step S3: Calculate the neighborhood homogeneity parameter. Calculate the channel-level distribution similarity of each point based on the similarity distance of the time trend of the multi-layer feature output. Then, complete the calculation of point-level and neighborhood-level distribution similarity by aggregating channel by channel and point by point.
[0012] Step S4: Using the calculated neighborhood-level distribution weights, select a small number of representative key homogeneous points, and perform long-range self-attention based on them to expand their receptive field and complete feature enhancement to obtain key visual features.
[0013] Step S5: Point cloud-text cross-modal encoding. Natural language instructions are input into a pre-trained large language model to utilize the reasoning ability of the large model to understand implicit needs, and the generated semantic features are aligned with key visual features to obtain enhanced visual-text fusion features.
[0014] Step S6: Mask generation. The enhanced visual-text features and key visual features are cross-attentioned to obtain an initial sparse mask. The similarity between the remaining points and the key representative points is recalculated using the distribution similarity calculation method in step S3, and a dense mask is generated to complete fine-grained mask segmentation.
[0015] Step S7: Generate text loss, cross-entropy loss, DICE loss, and consistency loss based on step S3, and perform joint loss optimization. By calculating these losses, the parameter adjustment direction of the neural network, including the point cloud encoder and the large language model, during the training process is corrected. That is, the correct text understanding, point cloud semantic label classification, and neighborhood distribution consistency discrimination characteristics are learned, ensuring that the parameter adjustment of the neural network conforms to the expected direction.
[0016] The specific descriptions of steps S1-S7 are as follows:
[0017] For step S1: data preprocessing, the specific steps are as follows:
[0018] Step S1.1: Based on the coordinate projection transformation, downsampling, and filtering and denoising preprocessing operation flow, the raw point cloud data collected by the sensor is transformed into preprocessed point cloud data;
[0019] Step S1.1: Based on the preprocessed point cloud data, construct a local neighborhood index for each point using the K-nearest neighbor algorithm or radius search method.
[0020] For step S2: point cloud feature extraction, the specific steps are as follows:
[0021] Step S2.1: Initialize the point cloud encoder and load the pre-trained point cloud encoder weights. The selected point cloud encoder should have a multi-layer architecture, including PointNext, PointMeta, DeLA networks and their variants.
[0022] Step S2.2: Input the preprocessed point cloud and local index into the point cloud encoder to perform layer-by-layer point cloud feature encoding, and record the feature output value of each layer.
[0023] For step S3: calculating the neighborhood homogeneity parameter, the specific steps are as follows:
[0024] Step S3.1: Arrange the recorded feature values in step S2.2 in order and merge adjacent feature channels to reduce the amount of computation. After processing, each point has a trend vector of length T, where T is the number of module layers of the encoder.
[0025] Step S3.2: Aggregate the trend vector from step S3.1 according to the pre-collected neighborhood index, and refer to the center point in each neighborhood index. Perform neighborhood point Trend vector similarity comparison can be measured using spherical rotation distance, cosine similarity, or KL divergence methods to form channel-level similarity weights. , where c represents the sequence number of the feature channel;
[0026] Step S3.3: Aggregate the channel similarity weights from Step S3.2, and calculate the similarity of each point using average pooling. , where C is the number of feature channel dimensions;
[0027] Step S3.4: Statistically analyze the similarity distribution of each point within each neighborhood and calculate the neighborhood homogeneity distribution weight. The variance ratio can be used. Perform calculations, where Represents the similarity between points in this neighborhood. The actual variance, and The theoretical maximum variance representing the similarity between points in this neighborhood. , where max is the maximum value in the neighborhood and min is the minimum value in the neighborhood.
[0028] For step S4: key homogeneous point selection and feature enhancement, the specific steps are as follows:
[0029] Step S4.1: Screening of key homogeneous representative points, based on the neighborhood homogeneity distribution weights proposed in Step S3.4. Filter the top K key homogeneous representative points in the global scope by sorting the values from largest to smallest;
[0030] Step S4.2: Introduce the selected key homogeneous representative points into the long-range self-attention mechanism to expand the receptive field and enhance features, thereby obtaining enhanced visual features. .
[0031] For step S5: point cloud-text cross-modal coding, the specific steps are as follows:
[0032] Step S5.1: Input the natural language text instruction into the word segmenter of the large model to obtain the text vector and the corresponding position encoding;
[0033] Step S5.2: Input the text vector obtained after processing the text instructions in step S5.1 and the corresponding positional encoding into the large model to obtain the corresponding text features. ;
[0034] Step S5.3: Apply the enhanced visual features obtained in step S4.2 Text features obtained in step S5.2 Perform cross-attention queries, where As a query As keys and values, visual-text fusion features are then obtained. .
[0035] For step S6: mask generation, the specific steps are as follows:
[0036] Step S6.1: Obtain the visual-text fusion features from step S5.3 As query criteria, the key representative visual features obtained in step S4.2 are used as query conditions. Perform cross-attention to obtain the initial sparse mask. ;
[0037] Step S6.2: Based on the calculation formulas in Steps S3.2 and S3.3, recalculate the channel-level similarity and the similarity of each point between the remaining points and the key representative points, and obtain the product of the two. And by a sparse mask of key representative points. With weight Calculate the probability of generating the corresponding dense mask. ),in, This represents the probability value of the dense mask for non-key point i, while K represents the set of key representative points in the neighborhood of point i, where K represents the number of neighborhood points. The mask probability value represents the key homogeneous point j. This represents the point-level similarity between point i and point j;
[0038] Step S6.3: Based on the threshold segmentation method, the dense mask probability values obtained in step S6.2 are truncated to generate a binary mask.
[0039] For step S7: joint loss optimization, the specific steps are as follows:
[0040] Step S7.1: Introduce text generation loss during training. ,in Represents the length of the output sequence. Represents the real token at position t. This represents the predicted tokens for the first t-1 positions, and X represents the input conditions, namely point cloud features and user text, ultimately forming a statistical representation of the correct probability of text generation by the model; a binary cross-entropy loss is introduced during training. N represents the total number of points in the point cloud, and i represents the index number of the point cloud. The actual label representing point i. The predicted probability of point i represents the final accuracy statistic for point classification; DICE loss is introduced during training. This generates statistics on the mask segmentation accuracy; consistency loss is introduced during training. ,in This represents the similarity of point cloud i to its neighboring point j within its own neighborhood. The temperature parameter is used to adjust the scaling. For indicator functions, when Its value is 1 when the point cloud labels are inconsistent, and 0 otherwise. That is, the relevant value is only counted when the point cloud labels are inconsistent within the neighborhood, and finally a probability statistic of neighborhood consistency and label correctness is formed.
[0041] Step S7.2: Perform a weighted summation of each loss function. By calculating these losses, the parameter adjustment direction of the neural network, including the point cloud encoder and the large language model, during the training process is corrected. This ensures that the neural network learns the correct characteristics of text understanding, point cloud semantic label classification, and neighborhood distribution consistency discrimination, and that the parameter adjustment of the neural network conforms to the expected direction.
[0042] The beneficial effects of this invention are:
[0043] Compared with existing technologies, this invention effectively alleviates the imbalance between high-density point cloud information and low-density text information by selecting key homogeneous points through neighborhood homogeneity calculation for long-range self-attention enhancement and cross-modal alignment, and significantly reduces computational complexity. At the same time, it adopts the strategy of "sparse mask generation + homogeneity weighted dense propagation" and introduces a geometric consistency loss function, which significantly improves the boundary accuracy and geometric consistency of the segmentation mask, and realizes efficient and accurate open text-guided point cloud segmentation. Attached Figure Description
[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Appendix Figure 1 A simplified flowchart of a point cloud segmentation method based on neighborhood homogeneity constraints and text guidance.
[0047] Appendix Figure 2 An illustrated example of an input point cloud for a point cloud segmentation method based on neighborhood homogeneity constraints and text guidance.
[0048] Appendix Figure 3A schematic diagram illustrating the output point cloud of an embodiment of a point cloud segmentation method based on neighborhood homogeneity constraints and text guidance. Detailed Implementation
[0049] To make the technical solution and objectives of this invention clearer and more understandable, the implementation steps of this invention will be described in detail below.
[0050] This invention will be illustrated using the S3DIS large-scale indoor point cloud segmentation dataset developed by Stanford University as an example. This example is a specific illustration of the invention to fully demonstrate its implementation approach; however, its implementation is not limited to this example.
[0051] The overall process of the point cloud segmentation method based on neighborhood homogeneity constraints and text guidance proposed in this invention is as follows: Figure 1 As shown, the specific implementation steps based on the above example are as follows:
[0052] Step S1: Data preprocessing, reading raw point cloud data and performing downsampling, noise reduction, DEM construction, and neighborhood construction data processing operations.
[0053] Step S1.1: Based on coordinate projection transformation, downsampling, filtering and noise reduction, the digital elevation model constructs a preprocessing operation flow to convert the raw point cloud data collected by the sensor into preprocessed point cloud data. The input point cloud data is as follows: Figure 2 As shown;
[0054] Step S1.2: Based on the preprocessed point cloud data, construct a local neighborhood index for each point using the K-nearest neighbor algorithm or radius search method.
[0055] Step S2: Extract features from the pre-trained multi-layer point cloud encoder on the pre-processed point cloud data, assigning corresponding features to each point.
[0056] Step S2.1: Initialize the point cloud encoder and load the pre-trained point cloud encoder weights. The selected point cloud encoder should have a multi-layer architecture. In this invention, the DeLAv1 version of the backbone network is selected as the baseline of the point cloud encoder. The network depth is four layers, and the corresponding feature dimensions are 64, 128, 256 and 512, respectively. At the same time, each layer is equipped with 4, 4, 8 and 4 neighborhood aggregation modules, respectively.
[0057] Step S2.2: Input the preprocessed point cloud and local index into the point cloud encoder to perform layer-by-layer point cloud feature encoding, and record the feature output value of each layer. The length of the corresponding feature sequence should correspond to the number of neighborhood aggregation modules in each layer.
[0058] Step S3: Calculate the neighborhood homogeneity parameter. Based on the time trend of the multi-layer feature output, calculate the channel-level distribution similarity of each point, thereby completing the calculation of point-level and neighborhood-level distribution similarity.
[0059] Step S3.1: Arrange the recorded feature values in step S2.2 in order and merge adjacent feature channels to reduce the amount of computation. After processing, each point has a trend vector of length T, where T is the number of module layers of the encoder. T corresponds to the number of neighborhood aggregation modules of 4, 4, 8, and 4.
[0060] Step S3.2: Aggregate the trend vector from step S3.1 according to the pre-collected neighborhood index, and refer to the center point in each neighborhood index. Perform neighborhood point The trend vector similarity comparison uses spherical rotation distance as a metric to form channel-level similarity weights. Where c represents the feature channel number and t represents the trend vector dimension number. This represents the change in point i at step t in a trend vector of length T. The magnitude representing its change;
[0061] Step S3.3: Aggregate the channel similarity weights from Step S3.2, and calculate the similarity of each point using average pooling. , where C is the number of feature channel dimensions;
[0062] Step S3.4: Statistically analyze the similarity distribution of each point within each neighborhood and calculate the neighborhood homogeneity distribution weight. The variance ratio can be used. Perform calculations, where Represents the similarity between points in this neighborhood. The actual variance, and The theoretical maximum variance representing the similarity between points in this neighborhood. , where max is the maximum value in the neighborhood and min is the minimum value in the neighborhood.
[0063] Step S4: Using the calculated neighborhood-level distribution weights, select a small number of representative key homogeneous points, and perform long-range self-attention on them to expand their receptive field and complete feature enhancement, thereby obtaining key visual features.
[0064] Step S4.1: Screening of key homogeneous representative points, based on the neighborhood homogeneity distribution weights proposed in Step S3.4. In this example, the selection ratio is set to the top K key homogeneous representative points in descending order of value across the entire scope.
[0065] Step S4.2: Introduce the selected key homogeneous representative points into the long-range self-attention mechanism to expand the receptive field and enhance features, thereby obtaining enhanced visual features. .
[0066] Step S5: Point cloud-text cross-modal encoding. Natural language instructions are input into a pre-trained large language model to leverage the model's reasoning capabilities to understand implicit needs. The generated semantic features are then aligned with key visual features to obtain enhanced visual-text fusion features.
[0067] Step S5.1: Input the natural language text instruction into the word segmenter of the large model to obtain the text vector and the corresponding position encoding. In this example, the instruction is "find the chair in the scene".
[0068] Step S5.2: Input the processed text instructions from step S5.1 into the large model to obtain the corresponding text features. ;
[0069] Step S5.3: Apply the enhanced visual features obtained in step S4.2 Text features obtained in step S5.2 Perform cross-attention queries, where As a query As keys and values, visual-text fusion features are then obtained. .
[0070] Step S6: Mask generation. The visual-text fusion features and visual features are queried step-by-step to generate sparse masks and refine dense masks.
[0071] Step S6.1: Obtain the visual-text fusion features from step S5.3 As query criteria, the key representative visual features obtained in step S4.2 are used as query conditions. Perform cross-attention to obtain the initial sparse mask. ;
[0072] Step S6.2: Based on the calculation formulas in Steps S3.2 and S3.3, recalculate the channel-level similarity and the similarity of each point between the remaining points and the key representative points, and obtain the product of the two. And by a sparse mask of key representative points. With weight Calculate the probability of generating the corresponding dense mask. ),in, This represents the probability value of the dense mask for non-key point i, while K represents the set of key representative points in the neighborhood of point i, where K represents the number of neighborhood points. The mask probability value represents the key homogeneous point j. This represents the point-level similarity between point i and point j;
[0073] Step S6.3: Based on the threshold segmentation method, the dense mask probability values obtained in step S6.2 are truncated to generate a binary mask. In this example, the truncation threshold is 0.8, i.e. , The value of the segmentation mask representing point i is shown below. The final mask segmentation effect is as follows: Figure 3 As shown.
[0074] Step S7: Generate text loss, cross-entropy loss, DICE loss, and consistency loss obtained in step S3, perform joint loss optimization, and conduct end-to-end training.
[0075] Step S7.1: Introduce text generation loss during training. ,in Represents the length of the output sequence. Represents the real token at position t. This represents the predicted tokens for the first t-1 positions, and X represents the input conditions, namely point cloud features and user text, ultimately forming a statistical representation of the correct probability of text generation by the model; a binary cross-entropy loss is introduced during training. N represents the total number of points in the point cloud, and i represents the index number of the point cloud. The actual label representing point i. The predicted probability of point i represents the final accuracy statistic for point classification; DICE loss is introduced during training. This generates statistics on the mask segmentation accuracy; consistency loss is introduced during training. ,in This represents the similarity of point cloud i to each point j in its own neighborhood, ultimately forming a probability statistic of neighborhood consistency and label correctness;
[0076] Step S7.2: Perform a weighted summation of each loss function, which in this embodiment are respectively... We assign weights of 0.5, 0.5, 1.0, and 1.0, and calculate these losses to correct the direction of parameter adjustment in the neural network, which includes the point cloud encoder and the large language model, during training. This ensures that the neural network learns the correct characteristics of text understanding, point cloud semantic label classification, and neighborhood distribution consistency discrimination, and that the parameter adjustment of the neural network conforms to the expected direction.
[0077] As described above, this invention achieves target object segmentation guided by natural language instructions by preprocessing the original point cloud, calculating neighborhood homogeneity, filtering key homogeneous points and enhancing features, cross-modal alignment, and generating and propagating masks. This effectively solves the problem of low segmentation efficiency caused by the imbalance between point cloud and text information density in large-scene point clouds, as well as the problem of blurred mask boundaries caused by downsampling, thus realizing high-precision and high-efficiency open point cloud segmentation.
[0078] This invention proposes a point cloud segmentation method based on neighborhood homogeneity constraints and text guidance. It uses computer processing equipment to process 3D point cloud data, introduces a large language model to achieve implicit instruction understanding, and combines a geometric consistency loss function to force the segmentation boundary to align with the geometric edge of the object. This enables the invention to be applied to various task scenarios such as open vocabulary segmentation and referential segmentation, providing technical support for fields such as intelligent robot interaction and autonomous driving scene understanding.
[0079] Any obvious modifications, equivalent substitutions, or further optimizations made to the above details without departing from the basic technical concept and spirit of the present invention shall be included within the scope of the claims of the present invention.
Claims
1. A point cloud segmentation method based on neighborhood homogeneity constraints and text guidance, characterized in that, The method can receive user-specified natural language text commands and raw point cloud data collected by 3D sensors such as LiDAR and depth cameras as input. It utilizes the reasoning capabilities of a large language model to understand user intent and point cloud data, and performs adaptive target segmentation based on user-specified requirements for specific scenarios, marking the target within the scene. It also incorporates geometric homogeneity constraints of the local neighborhood of the point cloud to improve segmentation efficiency and accuracy. The method includes the following steps: Step S1: Data preprocessing, read the raw point cloud data and complete data processing operations such as downsampling, noise reduction, DEM construction, and neighborhood construction; Step S2: Extract features from the pre-trained multi-layer architecture point cloud encoder on the pre-processed point cloud data and assign corresponding features to each point. Step S3: Calculate the neighborhood homogeneity parameter. Calculate the channel-level distribution similarity of each point based on the similarity distance of the time trend of the multi-layer feature output. Then, complete the calculation of point-level and neighborhood-level distribution similarity by aggregating channel by channel and point by point. Step S4: Using the calculated neighborhood-level distribution weights, select a small number of representative key homogeneous points, and perform long-range self-attention based on them to expand their receptive field and complete feature enhancement to obtain key visual features. Step S5: Point cloud-text cross-modal encoding. Natural language instructions are input into a pre-trained large language model to utilize the reasoning ability of the large model to understand implicit needs, and the generated semantic features are aligned with key visual features to obtain enhanced visual-text fusion features. Step S6: Mask generation. The enhanced visual-text features and key visual features are cross-attentioned to obtain an initial sparse mask. The similarity between the remaining points and the key representative points is recalculated using the distribution similarity calculation method in step S3, and a dense mask is generated to complete fine-grained mask segmentation. Step S7: Generate text loss, cross-entropy loss, DICE loss, and consistency loss based on step S3, and perform joint loss optimization. By calculating these losses, the parameter adjustment direction of the neural network, including the point cloud encoder and the large language model, during the training process is corrected. That is, the correct text understanding, point cloud semantic label classification, and neighborhood distribution consistency discrimination characteristics are learned, ensuring that the parameter adjustment of the neural network conforms to the expected direction.
2. The method according to claim 1, characterized in that, In step S1, the specific steps are as follows: Step S1.1: Based on the preprocessing operation process of coordinate projection transformation, downsampling, filtering and noise reduction, and digital elevation construction, the raw point cloud data collected by the sensor is transformed into preprocessed point cloud data; Step S1.2: Based on the preprocessed point cloud data, construct a local neighborhood index for each point using the K-nearest neighbor algorithm or radius search method.
3. The method according to claim 1, characterized in that, In step S2, the specific steps are as follows: Step S2.1: Initialize the point cloud encoder and load the pre-trained point cloud encoder weights. The selected point cloud encoder has a multi-layer architecture, including PointNext, PointMeta, DeLA networks and their variants. Step S2.2: Input the preprocessed point cloud and local index into the point cloud encoder to perform layer-by-layer point cloud feature encoding, and record the feature output value of each layer.
4. The method according to claim 1, characterized in that, In step S3, the specific steps are as follows: Step S3.1: Arrange the recorded feature values in step S2.2 in order and merge adjacent feature channels to reduce the amount of computation. After processing, each point has a trend vector of length T, where T is the number of module layers of the encoder. Step S3.2: Aggregate the trend vector from step S3.1 according to the pre-collected neighborhood index, and refer to the center point in each neighborhood index. Perform neighborhood point Trend vector similarity comparison can be measured using spherical rotation distance, cosine similarity, or KL divergence methods to form channel-level similarity weights. , where c represents the sequence number of the feature channel; Step S3.3: Aggregate the channel similarity weights from Step S3.2, and calculate the similarity of each point using average pooling. , where C is the number of feature channel dimensions; Step S3.4: Statistically analyze the similarity distribution of each point within each neighborhood and calculate the neighborhood homogeneity distribution weight. The variance ratio can be used. Perform calculations, where Represents the similarity between points in this neighborhood. The actual variance, and The theoretical maximum variance representing the similarity between points in this neighborhood. , where max is the maximum value in the neighborhood and min is the minimum value in the neighborhood.
5. The method according to claim 1, characterized in that, In step S4, the specific steps are as follows: Step S4.1: Screening of key homogeneous representative points, based on the neighborhood homogeneity distribution weights proposed in Step S3.
4. Filter the top K key homogeneous representative points globally, sorted by value from largest to smallest; Step S4.2: Introduce the selected key homogeneous representative points into the long-range self-attention mechanism to expand the receptive field and enhance features, thereby obtaining enhanced visual features. .
6. The method according to claim 1, characterized in that, In step S5, the specific steps are as follows: Step S5.1: Input the natural language text instruction into the word segmenter of the large model to obtain the text vector and the corresponding position encoding; Step S5.2: Input the text vector and positional encoding obtained after processing the text instructions in step S5.1 into the large model to obtain the corresponding text features. ; Step S5.3: Apply the enhanced visual features obtained in step S4.2 Text features obtained in step S5.2 Perform cross-attention queries, where As a query As keys and values, visual-text fusion features are then obtained. .
7. The method according to claim 1, characterized in that, In step S6, the specific steps are as follows: Step S6.1: Obtain the visual-text fusion features from step S5.3 As query criteria, the key representative visual features obtained in step S4.2 are used as query conditions. Perform cross-attention to obtain the initial sparse mask. ; Step S6.2: Based on the calculation formulas in Steps S3.2 and S3.3, recalculate the channel-level similarity and the similarity of each point between the remaining points and the key representative points, and obtain the product of the two. And by a sparse mask of key representative points. With weight Calculate the probability of generating the corresponding dense mask. ),in, This represents the probability value of the dense mask for non-key point i, while The set of key representative points in the neighborhood of point i. The mask probability value represents the key homogeneous point j. This represents the point-level similarity between point i and point j; Step S6.3: Based on the threshold segmentation method, the dense mask probability values obtained in step S6.2 are truncated to generate a binary mask.
8. The method according to claim 1, characterized in that, In step S7, the specific steps are as follows: Step S7.1: Introduce text generation loss during training. Binary cross-entropy loss DICE losses With consistency loss Where N represents the number of point clouds, Category labels representing points, This represents the similarity between point i and its neighboring point j. The temperature parameter is used to adjust the scaling. For indicator functions, when Its value is 1 when the point cloud labels are inconsistent within the neighborhood, and 0 otherwise. That is, the relevant value is only counted when the point cloud labels are inconsistent within the neighborhood. This results in statistics on the correct probability of model text generation, the accuracy of point classification, the accuracy of mask segmentation, and the probability statistics of neighborhood consistency and label correctness. Step S7.2: Perform a weighted summation of each loss function. By calculating these losses, the parameter adjustment direction of the neural network, including the point cloud encoder and the large language model, during the training process is corrected. This ensures that the neural network learns the correct characteristics of text understanding, point cloud semantic label classification, and neighborhood distribution consistency discrimination, and that the parameter adjustment of the neural network conforms to the expected direction.