A point cloud salient object detection method based on a SAM unsupervised decoder

Abstract

The application discloses a point cloud saliency target detection method based on a SAM prompt-free decoder, and comprises the following steps: firstly, original point cloud data is acquired, the original point cloud data is serialized by using a space-filling curve, and then initial point cloud features are obtained through embedding layer processing; then, the initial point cloud features are input into a pre-trained encoder, and multi-scale point cloud features are output; adapters are respectively introduced for each attention block in the encoder, feature adaptation is performed on the multi-scale point cloud features output by the encoder; the adapted multi-scale point cloud features are input into a hierarchical de-grid compensation module for layer-by-layer upsampling fusion, geometric position deviation information of a current layer and a previous layer is used to assist feature recovery, spatial discontinuity in the upsampling process is eliminated, and decoder input features are obtained; finally, the decoder input features are input into a SAM decoder, feature interaction is performed through a bidirectional attention mechanism, and point-level saliency prediction results are output.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and is particularly aimed at salient object detection. Specifically, it relates to a point cloud salient object detection method based on a SAM non-cue decoder. Background Technology

[0002] Salient Object Detection (SOD) aims to mimic the human eye's visual attention mechanism, automatically identifying and segmenting the regions or objects in an image or scene that are most visually appealing. This technology has significant application value in many downstream tasks such as image compression, object tracking, autonomous driving, and robot navigation.

[0003] Currently, salient target detection technology based on traditional two-dimensional color images has matured. However, with the rapid development and widespread adoption of three-dimensional sensor technologies such as LiDAR and RGB-D cameras, extending the salient target detection task from the two-dimensional plane to the three-dimensional point cloud space is becoming increasingly urgent for achieving more accurate three-dimensional scene understanding and autonomous decision-making in complex environments.

[0004] However, 3D point cloud data inherently possesses sparsity, disorder, and unstructured characteristics, making it extremely difficult to directly apply mature algorithms from 2D images. Currently, research on salient object detection in point clouds mainly faces the following technical challenges:

[0005] 1. Compared to the massive amounts of 2D image annotation data, 3D point cloud data annotation requires specialized tools and a significant amount of manual labor, resulting in extremely high costs. Therefore, publicly available point cloud salient object detection datasets are generally small in scale. Existing technologies (such as general point cloud segmentation networks like PointNet++, PointMLP, and PTv3, and dedicated salient object detection models like PointSal and PSOD-Net) typically employ strategies of "training from scratch" or "full parameter fine-tuning." When faced with scarce training samples, these methods are highly prone to overfitting, causing a sharp decline in the model's generalization ability in complex or unseen real-world scenarios, severely limiting their practical deployment and application.

[0006] 2. In recent years, general segmentation models, represented by the Segment Anything Model (SAM), have demonstrated excellent zero-shot generalization capabilities in the field of 2D image segmentation. However, because the core architecture of SAM is designed for 2D grid-like data, its input layer and feature extraction backbone cannot directly process unstructured 3D point cloud data. More importantly, the working mechanism of SAM heavily relies on external cues (such as points, boxes, or text) provided by the user to guide target segmentation, while the essence of salient object detection is to "automatically" discover and segment salient objects, resulting in a fundamental conflict in application logic. Therefore, how to overcome the modal limitations and cue dependence of SAM and leverage its powerful segmentation prior knowledge for 3D point cloud scenarios has become a pressing technical challenge in this field.

[0007] 3. In the encoder-decoder process of point cloud features, in order to restore the resolution of the original point cloud and output point-by-point prediction results, upsampling operations such as inverse pooling or interpolation are usually required. However, in existing solutions, direct upsampling operations often ignore the continuous changes in the spatial location of features, resulting in obvious gridding artifacts or feature breaks in the recovered feature map. This discontinuous feature representation directly impairs the localization accuracy of the final salient target boundary and the spatial smoothness of the prediction map, making the detection results visually inconsistent and difficult to meet the requirements of high-precision applications.

[0008] In summary, how to overcome modal differences, effectively utilize the prior knowledge of large models, and solve the discontinuity problem in the feature upsampling process under limited labeled samples, in order to achieve high-precision and high-generalization salient target detection in 3D point clouds, is a technical problem that urgently needs to be solved in the field of computer vision. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a point cloud salient target detection method based on SAM non-cue decoder. This invention designs a hierarchical demesh compensation module (HDCM). To address the problem of discontinuous group boundaries caused by hard grouping in dynamic parameter prediction, an overlap-aware serial neighborhood grouping strategy is proposed. This strategy expands the neighborhood coverage through a controllable overlap rate and performs feature fusion in the overlapping area, thereby improving prediction continuity.

[0010] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:

[0011] A point cloud salient object detection method based on SAM silent decoder includes the following steps:

[0012] Step 1: Obtain the raw point cloud data, serialize the raw point cloud data using a space filling curve, and then process it through an embedding layer to obtain the initial point cloud features;

[0013] Step 2: Input the initial point cloud features into a pre-trained encoder. The encoder includes multiple feature extraction stages. Each feature extraction stage uses several attention blocks to extract features and performs downsampling through grid pooling to output multi-scale point cloud features.

[0014] Step 3: For each attention block in the encoder, an adapter is introduced to adapt the multi-scale point cloud features output by the encoder. The adapter is an adapter based on an overlap-aware dynamic parameter projector. The adapter based on the overlap-aware dynamic parameter projector includes a normalization module, an overlap-aware dynamic projection layer and a projection output layer (ODP layer). The overlap-aware dynamic projection layer dynamically generates projection weights according to the input features to achieve parameter fine-tuning.

[0015] Step 4: Input the adapted multi-scale point cloud features into the hierarchical demeshing compensation module for layer-by-layer upsampling and fusion. Use the geometric positional deviation information between the current layer and the previous layer to assist feature recovery, eliminate spatial discontinuities in the upsampling process, and obtain the fused multi-level features. The fused multi-level features are then used for intermediate saliency prediction by an auxiliary prediction head to obtain auxiliary prediction results for intermediate supervision during the training phase. The auxiliary prediction head consists of three linear layers, with Gaussian error linear unit activation functions between adjacent linear layers. The last linear layer outputs a single-channel point-level prediction value. On the other hand, after passing through one linear layer, the decoder input features are obtained.

[0016] Step 5: Input the input features of the decoder into the SAM decoder. The SAM decoder removes the prompt input mechanism and outputs the point-level saliency prediction result after feature interaction through the bidirectional attention mechanism.

[0017] Preferably, in step S2, the encoder is a Sonata encoder, which includes five attention blocks, and adjacent attention blocks are downsampled through grid pooling.

[0018] Preferably, in step 2, the attention block adopts the Flash Attention mechanism. In the Flash Attention mechanism, the query matrix is ​​multiplied by the transpose of the key matrix and divided by the square root of the attention head dimension. After processing by the Softmax function, it is multiplied by the value matrix to obtain the attention output. The Flash Attention mechanism adopts a block strategy to integrate the calculation of the attention matrix and the soft maximum normalization processing online.

[0019] Preferably, the attention block includes conditional location encoding and a feedforward network; the data processing flow of the attention block is as follows:

[0020] First, add the input feature to the output of the conditional positional encoding to obtain the first intermediate feature;

[0021] The first intermediate feature is normalized by the layer and then input into the Flash Attention mechanism to obtain the first attention output. The first attention output is added to the first intermediate feature to obtain the second intermediate feature.

[0022] The second intermediate feature is normalized by the layer and then input into the feedforward network to obtain the feedforward output. The feedforward output, the output of the second intermediate feature processed by the adapter based on the overlapping sensing dynamic parameter projector, and the second intermediate feature are added together to obtain the output feature of the attention block.

[0023] Preferably, in step 2, the grid pooling process uses a feature summation method to fuse two layers of features during downsampling, and the depooling layer is followed by a normalization layer and a Gaussian error linear unit activation function.

[0024] Preferably, in step 3, a ReLU activation function is provided between the overlapping sensing dynamic projection layer and the projection output layer, and the output of the projection output layer is activated by the Sigmoid activation function to obtain the output features of the adapter.

[0025] Preferably, the data processing flow of the adapter is as follows: the normalization module performs layer normalization on the input features to obtain normalized features; the normalized features are input into the overlapping perception dynamic projection layer to obtain dynamic projection features; the dynamic projection features are activated by the ReLU activation function and then input into the projection output layer to obtain the projection output; the projection output is activated by the Sigmoid function and then used as the output features of the adapter.

[0026] Preferably, the overlapping sensing dynamic projection layer includes a parameter basis set and a dynamic weight router; the parameter basis set consists of multiple learnable parameter bases, and the dimension of each parameter base is the input feature dimension multiplied by the hidden layer dimension.

[0027] The dynamic weighted router performs average pooling on the grouping terms of each local point cloud group to obtain the average pooled group descriptors of each group; after processing the group descriptors of each group through two layers of linear networks, it is processed by the Softmax function to obtain the routing coefficient matrix of each group to each parameter basis; the routing coefficients of each group are weighted and merged with the corresponding parameter basis to generate the dynamic projection weights of each group; the dynamic projection weights are used as the projection matrix to project each grouping term to obtain the projected features; the projected features are inversely grouped and scaled and then superimposed on the input features of the adapter.

[0028] Preferably, the dynamic weight router adopts an overlapping sensing sequence neighborhood grouping strategy for local point cloud groups, sets a predetermined overlap rate between each group, and performs feature fusion in the overlapping area of ​​adjacent groups.

[0029] Preferably, in step 3, the adapter based on the overlapping sensing dynamic parameter projector is only set after each attention block; a static adapter is set after the remaining attention blocks. The static adapter projects the input features through the projection layer, activates them through the modified linear unit, and then outputs them through another projection layer.

[0030] Preferably, in step 4, the hierarchical demeshing compensation module receives three inputs: the coordinate offset between the current layer and the previous layer, the output features of the hierarchical demeshing compensation module of the previous layer, and the input features of the current layer, and outputs the compensated output features of the current layer.

[0031] The hierarchical demeshing compensation module sequentially processes the coordinate offset between the current layer and the previous layer through linear transformation, Gaussian error linear unit activation, and layer normalization to obtain the first branch feature; the output feature of the hierarchical demeshing compensation module of the previous layer is sequentially processed through linear transformation, Gaussian error linear unit activation, and layer normalization to obtain the second branch feature; the input feature of the current layer is sequentially processed through linear transformation, Gaussian error linear unit activation, and layer normalization to obtain the third branch feature.

[0032] The first branch feature, the second branch feature, and the upsampled third branch feature are concatenated to obtain the concatenated feature. The concatenated feature is then subjected to linear transformation, Gaussian error linear unit activation, and linear transformation in sequence to obtain the compensation feature of the current layer. The compensation feature of the current layer is added to the upsampled input feature of the current layer, and then concatenated with the output feature of the hierarchical demeshing compensation module of the previous layer to obtain the compensated output feature of the current layer.

[0033] The hierarchical demeshing compensation module performs weighted fusion of overlapping regions during each feature merging process.

[0034] Preferably, in step 5, the SAM decoder includes learnable intersection-union tokens and mask tokens; the learnable intersection-union tokens and mask tokens interact with the decoder input features via a bidirectional Transformer, which employs a bidirectional attention mechanism.

[0035] The interacted mask token is processed by a multilayer perceptron to obtain the mask token processing result; the interacted decoder input features are processed by an output upsampling layer, which consists of two linear layers, to obtain the decoder input feature processing result; the mask token processing result and the decoder input feature processing result are multiplied by a dot to obtain a point-level saliency prediction result.

[0036] Preferably, in step 1, the embedding layer uses a three-dimensional submanifold convolution with a kernel size of 5; the input point cloud data of the embedding layer is convolved by the three-dimensional submanifold and then sequentially processed by batch normalization and Gaussian error linear unit activation, resulting in a point cloud feature with 32 output channels.

[0037] Preferably, in step 2, the Sonata encoder loads pre-trained weights, and the encoder's parameters, excluding the adapter, are frozen during training.

[0038] Preferably, the overall loss function for model training consists of both cross-entropy loss and cross-union ratio loss.

[0039] This invention has the following characteristics and beneficial effects:

[0040] This invention employs a strategy of frozen pre-trained encoder and lightweight adapter fine-tuning. By loading a Sonata model pre-trained on a large amount of unsupervised point cloud data as the backbone for feature extraction, and using only the invention's unique Overlap-Aware Dynamic Parameter Projector (ODPP) adapter for minimal parameter updates, the model can efficiently learn task-specific knowledge with very few labeled samples. As shown in Table 1's quantitative evaluation, the generalization performance of this invention on the PCSOD dataset far surpasses existing state-of-the-art methods: compared to the powerful baseline PTv3 with full fine-tuning, this invention achieves a 1.8% improvement in Intersection over Union (IoU) while reducing the number of parameters by 86.7%; compared to the recent state-of-the-art DSFormer method, the mean absolute error (MAE) is significantly reduced by 41.7%. This fully demonstrates that this invention possesses extremely strong robustness and generalization ability with limited data resources.

[0041] This invention completely removes the cue input port of the SAM decoder and innovatively uses it as the decoding backbone network for salient point cloud object detection. This invention breaks the traditional technical bias that SAM must rely on cue input to function. By redesigning the decoder's training objectives and fine-tuning strategies, the original externally guided SAM decoder is transformed into a dedicated decoder capable of autonomously locating and segmenting salient objects in a 3D scene. Ablation experiments strongly demonstrate the effectiveness of this modification: when the cue-free SAM decoder is removed, the model performance shows the largest drop (IoU plummets from 0.847 to 0.830), confirming the successful transfer of SAM's powerful general segmentation prior knowledge to the 3D point cloud salient detection task, which is the key technical support for the performance breakthrough of this invention.

[0042] This invention designs a hierarchical demesh compensation module (HDCM). This module abandons the traditional simple unpooling or interpolation approach, creatively utilizing geometric positional deviation information (coordinate offset) as a guide. Through multi-source information stitching and residual compensation strategies, it restores the spatial continuity of features recovered from each level of upsampling. Ablation experiments verify the effectiveness of this module: after removing the HDCM module, the model shows a significant decline in both the F-measure and IoU metrics (IoU decreased from 0.847 to 0.841). This indicates that the HDCM module effectively smooths high-frequency noise in the feature recovery process, ensuring the clarity of the final salient target boundaries and the spatial smoothness and coherence of the predicted map.

[0043] This invention abandons the traditional hard grouping method in the ODPP adapter and introduces a neighborhood grouping strategy with controllable overlap rate. By setting a 75% overlap rate between different grouping windows and performing feature fusion in the overlapping area, sub-mesh artifacts caused by window boundary fragmentation are effectively avoided. The overlap rate comparison experiment in Table 3 systematically verifies the optimality of this strategy: the 75% overlap rate achieves the best balance across all evaluation metrics, ensuring sufficient modeling of the local context while guaranteeing continuous transition of features across windows.

[0044] In summary, this invention achieves comprehensive performance that surpasses the current best methods in point cloud salient target detection, while maintaining an extremely low number of trainable parameters. This perfect balance between high performance and high efficiency makes this invention highly practical and promising for widespread application in real-world industrial deployments (such as autonomous driving and robot navigation). Attached Figure Description

[0045] Figure 1 This is a model overall framework diagram of an embodiment of a point cloud salient target detection method based on a SAM non-cue decoder according to the present invention;

[0046] Figure 2 This is a diagram of the attention block structure in an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of the adapter and the Overlapping Perception Dynamic Parameter Layer (ODP layer) in an embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of the HDCM module in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the salient target detection effect in a point cloud scene in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0051] A point cloud salient target detection method based on SAM silent decoder, such as Figure 1 As shown, the system mainly consists of three parts: an encoder, an adapter, and a decoder. The encoder loads pre-trained Sonata weights and uses grid pooling for downsampling between different stages to extract local geometric features, long-range dependency features, and global semantic features, respectively. The adapter is based on the dynamic parameter projection structure of the Overlapping Aware Dynamic Parametric Projector (ODPP) and employs an overlapping window grouping strategy to efficiently adapt the features output from each stage of the encoder. The decoder uses the SAM decoder and removes the cueing mechanism to automatically complete the detection and segmentation of salient targets. Furthermore, this invention designs an HDCM module to address the feature discontinuity problem during feature upsampling.

[0052] Specifically, it includes the following steps:

[0053] Step 1: Obtain the original point cloud data, serialize the original point cloud data using a space filling curve, and then process it through an embedding layer to obtain the initial point cloud features.

[0054] In this embodiment, the original irregular point cloud is preprocessed into a serialized representation that the model can process, and high-dimensional features are generated. First, the point cloud is serialized using a space-filling curve, and then an embedding layer is used to obtain point cloud features with 32 channels. The embedding layer formula is:

[0055] (20)

[0056] in, and These are the input and output of the embedding layer, respectively. SubMConv5 represents a 3D sub-stream convolution with a kernel size of 5, BN represents batch normalization, and GELU represents the Gaussian error linear unit activation function.

[0057] Step 2: Input the initial point cloud features into a pre-trained encoder. The encoder includes multiple feature extraction stages. Each stage uses several attention blocks to extract features and performs downsampling through grid pooling to output multi-scale point cloud features.

[0058] In this embodiment, the encoder uses a pre-trained Sonata encoder, which comprises five stages. Each stage uses several attention blocks to extract features and performs downsampling through grid pooling. Figure 2 As shown, the attention block adopts the FlashAttention mechanism. In the FlashAttention mechanism, the query matrix is ​​multiplied by the transpose of the key matrix and divided by the square root of the attention head dimension. After being processed by the Softmax function, it is multiplied by the value matrix to obtain the attention output. The FlashAttention mechanism adopts a block strategy to integrate the calculation of the attention matrix and the soft maximum normalization processing online.

[0059] The attention mechanism uses the Flash Attention mechanism, which works according to the following formula:

[0060] (1)

[0061] Where Q, K, and V represent the query, key, and value of the attention layer, respectively. This refers to the dimension of each attention head. Flash Attention integrates the calculation of the attention matrix and softmax normalization online through a block-based strategy, avoiding the storage of the complete attention matrix in GPU memory, thereby significantly reducing GPU memory usage and improving computation speed.

[0062] Furthermore, the attention block includes conditional location encoding and a feedforward network; the data processing flow of the attention block is as follows:

[0063] First, the input features are concatenated with the output of the conditional positional encoding to obtain the first intermediate feature;

[0064] The first intermediate feature is normalized by the layer and then input into the Flash Attention mechanism to obtain the first attention output. The first attention output is then concatenated with the first intermediate feature to obtain the second intermediate feature.

[0065] The second intermediate feature is normalized by the layer and then input into the feedforward network to obtain the feedforward output. The feedforward output, the output of the second intermediate feature processed by the adapter based on the overlapping sensing dynamic parameter projector, and the second intermediate feature are concatenated to obtain the output feature of the attention block.

[0066] Specifically, the Conditional Location Encoding (xCPE) and the Feedforward Network (FFN) are integrated into the attention block according to the following formula:

[0067] (2)

[0068] (3)

[0069] (4)

[0070] Here, LN represents layer normalization, FlashAttention represents Flash Attention, xCPE represents the conditional location encoding module, used to enhance the representation of location information, and ODPP represents the overlapping perceptual dynamic projection layer, used to achieve a balance between local context and cross-window continuity. The encoder uses grid pooling for downsampling and uses feature summation instead of channel concatenation to fuse features. After the depooling layer, batch normalization and the GELU activation function are applied.

[0071] Step 3: For each attention block in the encoder, an adapter is introduced to adapt the multi-scale point cloud features output by the encoder. The adapter is an adapter based on an overlap-aware dynamic parameter projector. The adapter based on the overlap-aware dynamic parameter projector includes a normalization module, an overlap-aware dynamic projection layer and a projection output layer. It adopts an overlap-aware sequence neighborhood grouping strategy, expands the neighborhood coverage through a controllable overlap rate and performs feature fusion in the overlap area to alleviate grid-like artifacts and improve prediction continuity.

[0072] Specifically, such as Figure 3 As shown, an adapter based on the Overlap-Aware Dynamic Parameter Projector (ODPP) is introduced at each stage of the encoder for efficient parameter fine-tuning based on the pre-trained encoder. The adapter consists of three parts: a normalization module (Norm), an Overlap-Aware Dynamic Projection layer (ODP layer), and a projection output layer. ReLU activation is used between the ODP layer and the projection layer, and the final output is a sigmoid function. The adapter operates according to the following formula:

[0073] (5)

[0074] (6)

[0075] in, For adapter input, For ODP layer output, This is the final adapter output.

[0076] It should be noted that the adapter based on the overlap-aware dynamic parameter projector is only set after each attention block; a static adapter is set after the remaining attention blocks. The static adapter projects the input features through a projection layer, activates them through a modified linear unit, and then outputs them through another projection layer to ensure stable feature processing. The static adapter uses the following formula:

[0077] (7)

[0078] (8)

[0079] Furthermore, the ODP layer is the core submodule of the ODPP adapter, responsible for dynamically generating projection weights for each group of local point clouds. The core mechanism of the ODP layer consists of two parts: a parameter basis set and a dynamic weight router. It generates dedicated projection weights for different point cloud groups through input adaptation, enabling the adapter to flexibly cope with diverse geometric changes in 3D scenes.

[0080] The parameter basis set consists of multiple learnable parameter bases, and the dimension of each parameter base is the input feature dimension multiplied by the hidden layer dimension.

[0081] In this embodiment, the parameter basis set P consists of K learnable parameter bases, each with a dimension of . Where C is the dimension of the input features. The hidden layer dimension is defined. The set of parameter bases is continuously updated during training, forming a rich adaptive parameter space to support dynamic adjustments for different depths and point cloud inputs.

[0082] Furthermore, the dynamic weighted router performs average pooling on the group tokens of each local point cloud group to obtain the average pooled group descriptors of each group; after processing the group descriptors of each group through two layers of linear networks, it is processed by the Softmax function to obtain the routing coefficient matrix of each group with respect to each parameter basis; the routing coefficients of each group are weighted and merged with the corresponding parameter basis to generate the dynamic projection weights of each group; the dynamic projection weights are used as the projection matrix to project each group token to obtain the projected features; the projected features are then inversely grouped and scaled and superimposed on the input features of the adapter.

[0083] The dynamic weighted router adopts an overlapping sensing sequence neighborhood grouping strategy for local point cloud groups, setting a predetermined overlap rate between each group, and performing feature fusion in the overlapping area of ​​adjacent groups.

[0084] In this embodiment, the dynamic weighted router groups the words of each local point cloud. Perform average pooling to obtain the uniform feature descriptor for this group. Then, the softmax normalized routing coefficients of this group are predicted using MLP for the K parameter basis. Based on this, the parameter basis is weighted and merged to generate the dedicated projection weights for this group. Finally, the parameter basis is weighted and merged using routing coefficients to generate dynamic projection weights. The above process operates according to the following formula:

[0085] (9)

[0086] (10)

[0087] (11)

[0088] in, For local lexical units after grouping, The group descriptor is obtained after average pooling. The MLP is a two-layer linear network. The softmax temperature coefficient is... This is the routing coefficient matrix for each group with K parameter bases. For the first A parameter basis, For the first Dynamic projection weights for groups. After obtaining the dynamic weights, they are used as projection matrices to project the grouped tokens. After inverse grouping and scaling, the projections are superimposed onto the input features to achieve dynamic adaptive feature adjustment.

[0089] The Dynamic Weighted Router employs an overlapping-aware sequence neighborhood grouping strategy for local point cloud grouping. A predetermined overlap rate is set between groups, and feature fusion is performed in the overlapping regions of adjacent groups, thereby mitigating grid-like artifacts and improving prediction continuity. Through overlapping window grouping, the ODPP adapter can achieve a balance between local context and cross-window continuity.

[0090] In this embodiment, a predetermined overlap rate of 75% is set between each group.

[0091] Step 4: Input the adapted multi-scale point cloud features into the hierarchical demeshing compensation module for layer-by-layer upsampling and fusion. Use the geometric position deviation information between the current layer and the previous layer to assist feature recovery and eliminate spatial discontinuities in the upsampling process to obtain fused multi-level features. The fused multi-level features are then used for intermediate saliency prediction by an auxiliary prediction head to obtain auxiliary prediction results for intermediate supervision during the training phase. The auxiliary prediction head consists of three linear layers with Gaussian error linear unit activation functions between adjacent linear layers. The last linear layer outputs a single-channel point-level prediction value. On the other hand, after passing through one linear layer, the decoder input features are obtained.

[0092] Specifically, such as Figure 4 As shown, the demeshing compensation module receives three inputs: the coordinate offset between the current layer and the previous layer, the output features of the previous layer's hierarchical demeshing compensation module, and the input features of the current layer, and outputs the compensated output features of the current layer.

[0093] It should be noted that directly using inverse pooling to restore resolution during feature upsampling often leads to discontinuities in the feature space. To address this, this invention provides a Hierarchical Degridding Compensation Module (HDCM). This module employs a strategy based on multi-source information stitching and residual compensation, utilizing geometric positional deviation information to assist feature recovery and ensure the spatial continuity of the features. Figure 4 As shown, the HDCM module receives three inputs: the coordinate offset between the current layer and the previous layer. Output features of the previous layer HDCM and the input features of the current layer The output features are obtained according to the following formula:

[0094] (12)

[0095] (13)

[0096] (14)

[0097] (15)

[0098] (16)

[0099] (17)

[0100] in, This indicates an unpooling upsampling operation.

[0101] Step 5: Input the input features of the decoder into the SAM decoder. The SAM decoder removes the prompt input mechanism and outputs the point-level saliency prediction result after feature interaction through the bidirectional attention mechanism.

[0102] Specifically, in this embodiment, a SAM decoder is introduced into the point cloud salient object detection task, and the SAM cue mechanism is completely removed. The original SAM decoder relies on external cues to guide the selection of segmentation regions, but in the salient object detection task, the model needs to automatically identify salient objects in the scene without relying on any cue input. Therefore, this invention directly removes the cue input from the original SAM and performs fine-tuning, enabling the decoder to automatically locate salient object regions and complete segmentation prediction. The decoder operates according to the following formula:

[0103] (18)

[0104] (19)

[0105] in, It consists of IOU token and mask token. This indicates a two-way attention mechanism. It consists of two linear layers. The input features are the encoder's features, and the mask is the final point-level saliency prediction result. The decoder receives the fused features from the encoder, achieves feature interaction through a bidirectional attention mechanism, and finally outputs the point-level saliency prediction result.

[0106] This embodiment also includes training and evaluating the model described above. This embodiment uses the PCSOD dataset, which contains 2000 training samples and 872 test samples. This dataset was collected from various scenarios and includes some challenging samples, such as those with multiple targets, small targets, and complex backgrounds.

[0107] In this embodiment, multiple evaluation metrics are employed, including mean absolute error (MAE), F-measure (...). E-measure The intersection-over-union ratio (IoU) and the number of trainable parameters (Params) of the model were also evaluated.

[0108] The model is implemented using the PyTorch framework and trained on four NVIDIA RTX 3090 GPUs. The encoder loads pre-trained Sonata weights and freezes some parameters, fine-tuning only through an adapter. The decoder uses the SAM decoder and removes the cueing mechanism. Each input point cloud data is represented as a 6-dimensional vector containing 3D coordinates and RGB color values. We downsample the input points using a normalized grid sampling method with a grid size of 0.02 and perform block testing during the evaluation phase. Data augmentation strategies include random rotation and flipping, color dithering, and automatic contrast adjustment, while color and coordinate normalization are implemented. Model optimization uses the AdamW optimizer with an initial learning rate of [missing value]. The weight decay coefficient was set to 0.05, and the batch size was set to 64. Training consisted of 300 epochs, including an initial 3 epochs of linear warm-up, followed by a cosine annealing scheduling strategy. The overall loss function was composed of cross-entropy loss and cross-union ratio loss.

[0109] Comparative Example

[0110] The comparison provides several typical point cloud segmentation models and point cloud SOD models. Among them, PointNet2 is a classic point cloud deep learning model, PointMLP is a point cloud processing model based on multilayer perceptron, PTv3 (Point Transformer v3) is a Transformer-based point cloud model, PointSal is a point cloud salient object detection model, PSOD-Net is a Transformer-based point cloud salient object detection model, EPFNet is a salient object detection model that utilizes image-enhanced point cloud features, MSV-PCT is a point cloud salient object detection model that enhances Transformer features through multiple views, and DSFormer is a deformable Transformer point cloud salient object detection model.

[0111] Table 1: Performance and efficiency comparison of different methods on the PCSOD dataset.

[0112]

[0113] The quantitative evaluation results of this method on the PCSOD dataset are shown in Table 1. Compared with eight other mainstream methods, this method performs better in MAE, , This method achieves state-of-the-art performance across all four metrics (MAE, IoU, and GLONASS), while maintaining extremely high efficiency with a small number of trainable parameters (5.8M). Specifically, compared to the powerful Transformer-based baseline PTv3, this method reduces MAE by 0.004, improves IoU by 1.8%, and significantly reduces the number of trainable parameters by approximately 86.7% (5.8M vs 43.6M). Compared to the recent state-of-the-art method DSFormer, this method significantly reduces MAE by 41.7% (from 0.048 to 0.028) and improves IoU by 9.4%. This fully demonstrates that by introducing the SAM unannounced decoder, adapter mechanism, and HDCM module, this method effectively eliminates mesh artifacts while achieving efficient parameter fine-tuning, and leverages the powerful segmentation prior of SAM to achieve more accurate and continuous target segmentation.

[0114] To further verify the superior performance of this method, such as Figure 5 As shown, this embodiment visualizes and compares the prediction results with other representative methods. Observations reveal that, regardless of the diverse test cases—simple targets, small targets, multi-target scenarios, or complex backgrounds—this method consistently generates accurate saliency maps that closely match the ground truth. Compared to other comparative methods that are prone to edge blurring, background noise interference, or detail loss, the salient targets generated by this method not only have clearer and sharper boundaries but also effectively suppress false detections of non-salient regions. This precise capture of fine-grained details and effective utilization of global contextual information fully demonstrates that this method possesses stronger robustness and superior segmentation quality in complex scenarios.

[0115] To verify the effectiveness of each core component in this method, detailed quantitative ablation experimental results are shown in Table 2 in this embodiment. By removing the ODPP adapter, HDCM module, or SAM decoder, we observed that the model performance decreased to varying degrees, which fully demonstrates the necessity of the design of each module.

[0116] Table 2: Effective ablation experiments of OODP adapter, HDCM module and SAM silent decoder.

[0117]

[0118] Effectiveness of ODPP: As shown in Table 2, the full model has a MAE of 0.028 and an IoU of 0.847. After removing the ODPP adapter, although the number of parameters decreased by only 0.2M, the MAE increased to 0.029 and the IoU decreased to 0.844. This performance decline indicates that directly utilizing the pre-trained encoder without targeted fine-tuning is insufficient. ODPP achieves improved accuracy with extremely low parameter costs through a dynamic parameter projection mechanism.

[0119] Effectiveness of the HDCM module: Removing HDCM resulted in a greater performance degradation. This indicates that simple unpooling operations introduce spatial discontinuities during feature upsampling. The HDCM module, through its unique structural design, smooths out high-frequency noise during feature recovery, ensuring the spatial consistency of the prediction results, which is essential for obtaining high-quality saliency maps.

[0120] The effectiveness of the SAM no-hint mechanism: The model performance degrades most when the SAM decoder and its hint mechanism are completely removed. This performance gap strongly demonstrates the value of introducing the powerful prior knowledge of SAM. Our no-hint decoding strategy successfully transfers the segmentation capabilities of SAM to the point cloud salient object detection task, enabling the model to maintain high-precision localization and segmentation of salient objects without external hints.

[0121] Table 3: Comparison experiment of different overlap rates of DPP adapters.

[0122]

[0123] The impact of overlap rate on performance: such as Figure 3 As shown, the choice of overlap rate has a significant impact on the performance of the DPP adapter. When the overlap rate is too low (25%), the neighborhood coverage between windows is insufficient, and the feature fusion of the boundary region is not sufficient, resulting in a slight advantage in the IoU metric but a lower F... m and E m The performance metrics are not as good as 75% overlap. When the overlap rate is too high (80%), the excessive overlapping areas introduce redundant information, which reduces the prediction accuracy.

[0124] Optimal overlap rate selection: A 75% overlap rate achieved optimal or near-optimal performance across all evaluation metrics, demonstrating that this overlap rate strikes the best balance between sufficient local context coverage and cross-window continuity. This result indicates that moderate overlap coverage ensures sufficient fusion of neighborhood features while avoiding information interference from excessive redundancy, making it the optimal configuration for overlap-aware grouping strategies.

[0125] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A point cloud salient target detection method based on a SAM (Significant Awareness Detector) decoder, characterized in that, Includes the following steps: Step 1: Obtain the raw point cloud data, serialize the raw point cloud data using a space filling curve, and then process it through an embedding layer to obtain the initial point cloud features; Step 2: Input the initial point cloud features into a pre-trained encoder. The encoder includes multiple feature extraction stages. Each feature extraction stage uses several attention blocks to extract features and performs downsampling through grid pooling to output multi-scale point cloud features. Step 3: For each attention block in the encoder, introduce an adapter to adapt the multi-scale point cloud features output by the encoder. The adapter is an adapter based on an overlap-aware dynamic parameter projector. The adapter based on the overlap-aware dynamic parameter projector includes a normalization module, an overlap-aware dynamic projection layer, and a projection output layer. The overlap-aware dynamic projection layer dynamically generates projection weights based on input features to achieve parameter fine-tuning. A ReLU activation function is provided between the overlapping sensing dynamic projection layer and the projection output layer. The output of the projection output layer is activated by the Sigmoid activation function to obtain the output features of the adapter. The overlapping sensing dynamic projection layer includes a parameter basis set and a dynamic weight router; the parameter basis set consists of multiple learnable parameter bases, and the dimension of each parameter base is the input feature dimension multiplied by the hidden layer dimension. The dynamic weighted router performs average pooling on the grouping terms of each local point cloud group to obtain the average pooled group descriptors of each group; after processing the group descriptors of each group through two layers of linear network, it is processed by the Softmax function to obtain the routing coefficient matrix of each group to each parameter basis; the routing coefficients of each group are weighted and merged with the corresponding parameter basis to generate the dynamic projection weight of each group. The dynamic projection weights are used as the projection matrix to project each group of words to obtain the projected features; the projected features are then superimposed onto the input features of the adapter after inverse grouping and scaling. The dynamic weighted router adopts an overlapping sensing sequence neighborhood grouping strategy for local point cloud groups, sets a predetermined overlap rate between each group, and performs feature fusion in the overlapping area of ​​adjacent groups. Step 4: Input the adapted multi-scale point cloud features into the hierarchical demeshing compensation module for layer-by-layer upsampling and fusion. Use the geometric position deviation information between the current layer and the previous layer to assist feature recovery and obtain the fused multi-level features. The multi-level features are respectively input into the auxiliary prediction head and the linear layer to obtain the auxiliary prediction result and the decoder input features; Step 5: Input the input features of the decoder into the SAM decoder. The SAM decoder removes the prompt input mechanism and outputs the point-level saliency prediction result after feature interaction through the bidirectional attention mechanism.

2. The method according to claim 1, characterized in that, In step S2, the encoder is a Sonata encoder, which includes five attention blocks, and adjacent attention blocks are downsampled through grid pooling.

3. The method according to claim 1, characterized in that, In step 2, the attention block adopts the FlashAttention mechanism. In the FlashAttention mechanism, the query matrix is ​​multiplied by the transpose of the key matrix and divided by the square root of the attention head dimension. After processing by the Softmax function, it is multiplied by the value matrix to obtain the attention output. The FlashAttention mechanism adopts a block strategy to integrate the calculation of the attention matrix and the soft maximum normalization processing online.

4. The method according to claim 3, characterized in that, The attention block includes conditional location encoding and a feedforward network; the data processing flow of the attention block is as follows: First, add the input feature to the output of the conditional positional encoding to obtain the first intermediate feature; The first intermediate feature is normalized by the layer and then input into the Flash Attention mechanism to obtain the first attention output. The first attention output is added to the first intermediate feature to obtain the second intermediate feature. The second intermediate feature is normalized by the layer and then input into the feedforward network to obtain the feedforward output. The feedforward output, the output of the second intermediate feature processed by the adapter based on the overlapping sensing dynamic parameter projector, and the second intermediate feature are added together to obtain the output feature of the attention block.

5. The method according to claim 1, characterized in that, In step 2, the grid pooling process uses a feature summation method to fuse two layers of features during downsampling, and the depooling layer is followed by a normalization layer and a Gaussian error linear unit activation function.

6. The method according to claim 1, characterized in that, The data processing flow of the adapter is as follows: the normalization module performs layer normalization on the input features to obtain normalized features; The normalized features are input into the overlapping perceptual dynamic projection layer to obtain the dynamic projection features; The dynamic projection features are activated by the ReLU activation function and then input into the projection output layer to obtain the projection output; the projection output is activated by the Sigmoid function and then used as the output feature of the adapter.

7. The method according to claim 1, characterized in that, In step 3, the adapter based on the overlap-aware dynamic parameter projector is only set after each attention block; After the remaining attention blocks, a static adapter is set. The static adapter projects the input features through a projection layer, activates them through a modified linear unit, and then outputs them through another projection layer.

8. The method according to claim 1, characterized in that, In step 4, the hierarchical demeshing compensation module receives three inputs: the coordinate offset between the current layer and the previous layer, the output features of the hierarchical demeshing compensation module of the previous layer, and the input features of the current layer, and outputs the compensated output features of the current layer. The hierarchical demeshing compensation module sequentially processes the coordinate offset between the current layer and the previous layer through linear transformation, Gaussian error linear unit activation, and layer normalization to obtain the first branch feature; The output features of the hierarchical demeshing compensation module of the previous layer are sequentially processed by linear transformation, Gaussian error linear unit activation and layer normalization to obtain the second branch features; the input features of the current layer are sequentially processed by linear transformation, Gaussian error linear unit activation and layer normalization to obtain the third branch features. The first branch feature, the second branch feature, and the upsampled third branch feature are concatenated to obtain the concatenated feature. The concatenated feature is then subjected to linear transformation, Gaussian error linear unit activation, and linear transformation in sequence to obtain the compensation feature of the current layer. The compensation feature of the current layer is added to the upsampled input feature of the current layer, and then concatenated with the output feature of the hierarchical demeshing compensation module of the previous layer to obtain the compensated output feature of the current layer. The hierarchical demeshing compensation module performs weighted fusion of overlapping regions during each feature merging process.

9. The method according to claim 1, characterized in that, In step 5, the SAM decoder includes learnable intersection-over-union tokens and mask tokens; the learnable intersection-over-union tokens and mask tokens interact with the decoder input features through a bidirectional Transformer, which employs a bidirectional attention mechanism. The interacted mask token is processed by a multilayer perceptron to obtain the mask token processing result; The decoder input features after interaction are processed by the output upsampling layer, which consists of two linear layers, to obtain the decoder input feature processing result. The result of the mask token processing is multiplied by the result of the decoder input feature processing to obtain a point-level saliency prediction result.

10. The method according to claim 1, characterized in that, In step 1, the embedding layer uses a 3D submanifold convolution with a kernel size of 5; the input point cloud data of the embedding layer is convolved by the 3D submanifold and then sequentially processed by batch normalization and Gaussian error linear unit activation, resulting in a point cloud feature with 32 output channels.

11. The method according to claim 2, characterized in that, In step 2, the Sonata encoder loads pre-trained weights, and the encoder's parameters, excluding the adapter, are frozen during training.

12. The method according to any one of claims 1 to 11, characterized in that, The overall loss function for model training consists of cross-entropy loss and cross-union ratio loss.

Images