A weakly supervised semantic segmentation method for point clouds based on dual self-attention mechanism

Abstract

This application provides a weakly supervised semantic segmentation method for point clouds based on a dual self-attention mechanism, including: obtaining partially annotated three-dimensional point cloud data, the point cloud data containing spatial coordinate information; encoding the local coordinate features of the point cloud data through the GNE module to generate a discriminative local information representation; using the DLA module to calculate the feature similarity between points in the point cloud data, generate attention weights and adjust the point feature expression; performing weighted aggregation of the adjusted point features through the AFAP module to generate a global feature representation; stacking the DLA module and the AFAP module in series to form a DARF module to improve the modeling capability of complex geometric shapes; performing semantic segmentation on the point cloud data based on a weakly supervised training strategy and outputting the segmentation results. This application significantly reduces the reliance on annotated data and improves segmentation performance when limited annotation is available.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a point cloud weakly supervised semantic segmentation method based on a dual self-attention mechanism. Background Art

[0002] Semantic segmentation of 3D point clouds faces a core technical challenge in large-scale applications: how to achieve high-precision and high-generalization semantic segmentation in the face of insufficient annotated data and complex geometric structures? This problem involves multiple aspects. First, the data annotation dilemma. Large-scale point clouds are massive, making comprehensive annotation expensive. Furthermore, limited annotations fail to capture the complexity of the scene. Second, there is the challenge of geometric feature extraction. Point cloud data is discrete and sparse, lacking a regular structure, making it difficult to directly apply traditional operations like convolution. Third, there is the need to balance local and global semantic understanding, capturing both detailed features and grasping the overall semantics. Furthermore, model generalization is a major challenge: how to adapt trained models to changing scenes. These interrelated issues together constitute a complex technical challenge. Designing a point cloud semantic segmentation method that can effectively model complex geometric shapes, balance local details with global semantics, and achieve good generalization, within the constraints of limited annotation, has become a key issue in this field. Summary of the Invention

[0003] This application provides a point cloud weakly supervised semantic segmentation method based on a dual self-attention mechanism, which mainly includes:

[0004] Acquire partially annotated 3D point cloud data, which contains spatial coordinate information; encode the local coordinate features of the point cloud data through the Geometric Neighborhood Encoding (GNE) module to generate a discriminative local information representation; use the Paired Local Attention (DLA) module to calculate the feature similarity between points in the point cloud data, generate attention weights and adjust the point feature expression; perform weighted aggregation of the adjusted point features through the Adaptive Feature Aggregation and Pooling (AFAP) module to generate a global feature representation; stack the Paired Local Attention (DLA) module and the Adaptive Feature Aggregation and Pooling (AFAP) module in series to form a Dual Attention Residual Fusion (DARF) module to improve the modeling ability of complex geometric shapes; perform semantic segmentation on the point cloud data based on a weakly supervised training strategy and output the segmentation results. Furthermore, the local coordinate features of the point cloud data are encoded through the geometric neighborhood encoding (GNE) module, including: for each center point in the point cloud data, searching for its neighboring points based on the Euclidean distance using the nearest neighbor algorithm; obtaining the relative spatial position information and Euclidean distance information between the center point and the neighboring points; enhancing the geometric representation of the local area based on the relative spatial position information and Euclidean distance information; inputting the enhanced geometric representation into the convolutional layer for linear transformation to generate a discriminative local information representation; reducing excessive reliance on spatial information through local information representation and improving robustness to noise and irregular distribution. Furthermore, a paired local attention (DLA) module is used to calculate the feature similarity between points in point cloud data, including: obtaining the center point feature and its neighborhood point features in the point cloud data; extracting point cloud representations of different resolutions through multi-scale convolution operations to alleviate the information loss caused by sparsity; calculating the similarity between the center point feature and the neighborhood point features based on the query-key-value structure to generate dynamic attention weights; adjusting the point feature expression based on the dynamic attention weights to enhance the understanding of local and global semantics; incorporating local position encoding information into the attention weights to improve the perception of geometric structure. Furthermore, the adjusted point features are weighted and aggregated through the adaptive feature aggregation pooling (AFAP) module, including: obtaining the fused neighborhood context features output by the paired local attention (DLA) module; concatenating the fused neighborhood context features with the spatial encoding information; weighting the importance of each neighborhood point through channel-independent attention pooling to generate a global feature representation; and enhancing the interactive expression of local and global information based on the global feature representation.Furthermore, the paired local attention (DLA) module and the adaptive feature aggregation and pooling (AFAP) module are stacked in series to form a dual attention residual fusion (DARF) module, which includes: obtaining the spatial information and feature vector of the point cloud data as input; deeply fusing the spatial information and feature vector through the paired local attention (DLA) module to generate fused neighborhood context features; re-joining the fused neighborhood context features with the spatial encoding information; performing secondary attention pooling on the joined features through the adaptive feature aggregation and pooling (AFAP) module; and introducing the original input features into the model using a residual connection structure, adjusting the feature channels to avoid information loss. Furthermore, semantic segmentation of point cloud data is performed based on a weakly supervised training strategy, including: obtaining a small number of labeled points in the point cloud data, while keeping the remaining points unlabeled; pre-processing the point cloud data through grid downsampling to preserve spatial distribution information; randomly selecting some points according to a preset ratio to add labels; extending the labeled information to unlabeled points through a label propagation mechanism; and combining a semantic similarity mechanism to enhance the effectiveness of the training data and improve the model's generalization ability. Furthermore, the segmentation results are output, including: obtaining point cloud features processed by the dual attention residual fusion (DARF) module; predicting the category of unlabeled points based on the point cloud features; generating segmentation results based on the category prediction to represent the semantic category of each point in the point cloud data; and achieving efficient semantic understanding of large scene point clouds through the segmentation results.

[0005] The technical solution provided by this application may have the following beneficial effects:

[0006] The present application discloses a weakly supervised semantic segmentation method for point clouds based on a dual self-attention mechanism. To address the problems of insufficient labeled data and complex geometric structures in semantic segmentation of large-scale point clouds, this method extracts local features through geometric neighborhood coding, uses paired local attention modules to calculate the similarity between points and dynamically adjust feature weights, and connects it in series with an adaptive feature aggregation and pooling module to form a dual attention residual fusion structure to achieve multi-round feature fusion. A weakly supervised strategy is adopted, only a small number of points are labeled and extended to unlabeled areas through label propagation. The present application effectively improves the modeling capability of complex geometric shapes, enhances local and global semantic understanding, and achieves high-precision semantic segmentation of large-scale point clouds under limited labeling conditions, with good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a flowchart of a point cloud weakly supervised semantic segmentation method based on a dual self-attention mechanism in this application.

[0008] Figure 2 This is the network structure diagram of the weakly supervised semantic segmentation network (DDLA-Net);

[0009] Figure 3It is the Geometric Neighborhood Encoding (GNE) module;

[0010] Figure 4 It is a Dual Local Attention (DLA) module;

[0011] Figure 5 It is the Adaptive Feature Aggregation Pooling (AFAP) module;

[0012] Figure 6 It is the Dual Attention Residual Fusion (DARF) module. DETAILED DESCRIPTION

[0013] To further understand the content of this application, the present application is described in detail with reference to the accompanying drawings and examples. The present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0014] In order to make the purpose, technical solutions and advantages of this application more clear, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and specific implementation methods:

[0015] like Figure 1 As shown in FIG, a point cloud weakly supervised semantic segmentation method based on a dual self-attention mechanism includes the following steps:

[0016] S101: Acquire partially annotated three-dimensional point cloud data, where the point cloud data includes spatial coordinate information;

[0017] S102: Encode the local coordinate features of the point cloud data through the GNE module to generate a discriminative local information representation;

[0018] S103: Use the DLA module to calculate the feature similarity between points in the point cloud data, generate attention weights and adjust the point feature expression;

[0019] S104: Perform weighted aggregation on the adjusted point features through the AFAP module to generate a global feature representation;

[0020] S105: Stacking the DLA module and the AFAP module in series to form a DARF module to improve the modeling capability of complex geometric shapes;

[0021] S106: Perform semantic segmentation on the point cloud data based on the weakly supervised training strategy and output the segmentation results.

[0022] To further describe the above content in detail, the following is explained: The DDLA-Net network proposed in this application adopts an encoder-query network architecture and consists of two core components: a point local feature encoder and a point query network. Among them, the point local feature encoder is used to extract local features for all annotated points. It extracts hierarchical multi-scale local features by stacking multiple encoding layers and downsampling operations. The point query network is used to predict unannotated points. It uses unlabeled 3D query points as input, and then uses a trilinear interpolation algorithm to obtain a set of learning point features related to the point. Because the query point and the collected point features share similar semantic information, the training signal of the query point can be shared and back-propagated to related points, and finally the category of the unlabeled point is predicted. As Figure 2 As shown in Figure 1, the input of DDLA-Net is a large-scale point cloud of size N×dim, where N is the number of points and dim is the input feature dimension. Each point is represented by its 3D coordinates and color information. The network consists of four encoder layers and a query network. The output of DDLA-Net is the semantic labels of all unlabeled points, of size ,in is the number of unlabeled points, is the number of semantic categories.

[0023] The network consists of four key modules: the GNE module, the DLA module, the AFAP module, and the DARF module. The GNE module encodes the local coordinate features of some annotated point clouds, allowing the network to focus on point cloud location information. The DLA module calculates the feature similarity between two points and uses multi-scale convolution operations to extract point cloud representations of different resolutions, effectively alleviating the information loss caused by point cloud sparsity and significantly enhancing the model's perception of small objects and complex geometric structures. Finally, the DLA module and the AFAP module are stacked in series to form the DARF module, which improves the modeling ability of complex geometric shapes.

[0024] (1) Geometric Neighborhood Encoding Module

[0025] Point attributes of a point cloud include coordinates, color, and other attributes. Because in the task of semantic segmentation of point clouds, the network is more sensitive to position information than other information. For example, when performing semantic segmentation on a green wall and grass, it is obvious that the coordinate feature can better show the difference between the two. In addition, many point cloud semantic segmentation networks introduce too much spatial information in position encoding, which causes the network to be overly dependent on spatial relationships and reduces sensitivity to complex semantics. Especially when there are errors or irregular distributions in the point cloud, too much spatial information encoding will amplify the impact of certain abnormal points, increase sensitivity to noise and outliers, and disrupt feature aggregation. For example, the position encoding module of the serial code network splices the spatial position information of the center point, the spatial position information of the neighborhood points, the relative spatial position information of the point pairs, and the Euclidean distance of the point pairs to form the spatial information of the local points.

[0026] Therefore, this application proposes a GNE module, which does not introduce too much spatial information, but only uses the relative spatial position information and the Euclidean distance information of the point pairs to enhance the geometric representation of the local point cloud. Figure 3 As shown in the figure. This module explicitly inputs the encoded coordinate features into a convolution layer with a convolution kernel size of 1×1 for linear transformation, learning useful features and avoiding the mutual interference of excessive spatial information, so that the corresponding point features can know their position in space. This enables the GNE module to clearly observe local information, which is very helpful for learning complex scene structures. Subsequently, this module will be embedded in the DLA module and AFAP module to integrate other features and help learn more important point features. The operation steps of this module are as follows:

[0027] Find adjacent points. In order to quickly retrieve adjacent points, this paper uses the K-nearest neighbor (KNN) algorithm to search for each point based on the Euclidean distance. Neighboring points , which represents a target point in the point cloud using the K-nearest neighbor (KNN) algorithm The K nearest neighbor points are searched. These nearest neighbor points are in accordance with The Euclidean distances are sorted from nearest to farthest, where is the target query point in the original point cloud, usually a three-dimensional coordinate vector (x, y, z), and the superscript {1, 2, 3, ..., k} represents the sorting index of the neighboring points. It's distance The nearest point, and so on, is the k-th closest point.

[0028] Point coordinate encoding. Neighboring points The local area composed of the points is encoded. The process of encoding the local point coordinates is shown in formula (1):

[0029] (1)

[0030] in, is a multi-layer perceptron (MLP), is the center point, k is the number of nearest neighbors to search, The neighboring points of the center point. Relative position information, represents the Euclidean distance between points, Represents a splicing operation, is the encoded coordinate feature.

[0031] (2) Dual Local Attention Module

[0032] The spatial information of a point affects the importance of its features. The closer the local center point is to its neighboring points, the higher their feature similarity and the greater their semantic relevance. For example, among the neighboring points of a center point near the edge of an object, the closer the neighboring points are to the center point, the greater their contribution to defining the object's shape and region segmentation. Consequently, these points receive higher weight in the attention mechanism or weight allocation.

[0033] Therefore, in order to explore the deeper association between point spatial information and point feature information, a dynamic relationship between the two is established, such as Figure 4 As shown in the figure, this application proposes a DLA module, which integrates spatial relationships and semantic information in the process of point cloud feature aggregation. It realizes dynamic weighted aggregation of local neighborhoods of point clouds through Query-Key-Value structure and multi-scale convolution operation, which significantly improves the model's perception ability and generalization performance of complex structures. Figure 4 As shown. Among them, Center point The features of its adjacent points The characteristic representation of . , (d is the characteristic channel), where is the output feature, It is Hadamard. , and They are the first multilayer perceptron with one linear layer, the second multilayer perceptron, and the third multilayer perceptron. From formula (1), it represents enhanced position encoding. This module focuses more on the interaction and dependency between points, rather than just the weighted aggregation of single-point features. This enables the model to dynamically adjust the aggregation weight according to the importance of points in different scenarios. It also incorporates the position encoding information of local points into the neighborhood point features and attention scores, which effectively enhances the model's understanding of the geometric structure of the point cloud. The operation steps of this module are as follows:

[0034] First, input features pass The operation replicates the elements of a tensor along a specified dimension, with the input tensor and the number of replications along each dimension as parameters, as shown in Formula (2). A query vector (Query) is then generated through a set of convolutional layers, as shown in Formula (3).

[0035] (2)

[0036] (3)

[0037] in, Center point The feature vector of is used as the input of the Tile function. The Tile function is used to copy the tensor along each dimension to generate a larger tensor. The vector is copied K times to get a larger tensor , K is the number of neighboring points set in the geometric neighborhood encoding module. By copying the feature vector of the center point, the feature vector of each center point can maintain the same dimension as the feature vectors of all its neighboring points, which facilitates subsequent element-by-element operations (such as subtraction and concatenation), thus facilitating subsequent feature interaction and attention calculation. is the first multilayer perceptron with one linear layer that performs a 1x1 convolution operation, resizing the tensor Linear combination of feature channels to generate query vector (Query) .

[0038] At the same time, the neighborhood features of each point are extracted using the given neighbor index, and the key vector (Key) and value vector (Value) are generated through the multi-scale convolutional network, as shown in formulas (4) and (5), respectively.

[0039] (4)

[0040] (5)

[0041] in, Center point The features of its adjacent points The characteristic representation of , K represents the center point The number of nearest neighbors. and The second and third multilayer perceptrons have a linear layer, which are used to perform 1x1 convolution operations to adjust the feature vectors of neighboring points. The linear combination of feature channels generates a new feature representation. represents the generated key vector, Represents the generated value vector.

[0042] In order to enhance the spatial perception ability of the model, spatial position information is introduced and the encoded local position information is converted into Added to the value vector (Value) to form a feature representation with position information , as shown in formula (6):

[0043] (6)

[0044] in, Derived from formula (5), it represents the generated value vector. From formula (1), it represents the enhanced position encoding, It is a feature representation that combines the value vector of neighboring points and coordinate information, which helps the model to use both feature and spatial location information when calculating attention, thereby enhancing the model's expressive power.

[0045] Next, by calculating the feature difference between the query vector (Query) and the key vector (Key) and combining it with the local position information of the point cloud, a basic vector for attention weight is constructed. , the vector not only contains the semantic feature differences, but also integrates the geometric position relationship between the points, as shown in formula (7):

[0046] (7)

[0047] in, Derived from formula (3), it represents the generated query vector (Query), Derived from formula (4), it represents the generated key vector. From formula (1), it represents the enhanced position encoding. Used to measure the feature difference between the query vector and the key vector, and then introduce position encoding , get the basis vector for constructing attention weights , this operation enables the subsequent attention weights to take into account both feature similarity and spatial proximity.

[0048] Afterwards, the basis vectors used to construct the attention weights After processing by the multi-layer perceptron and the normalized function Softmax layer, the attention weight of each neighboring point is obtained, as shown in formula (8):

[0049] (8)

[0050] in, Represents the raw input of the attention weights, i.e., the basis vectors used to construct the attention weights, which are used to generate the attention scores. Represents a multilayer perceptron with one linear layer that performs a 1x1 convolution operation, adjusting the basis vector Linear combination between channels to generate new feature representations, It means that a multilayer perceptron with a linear layer performs 1x1 convolution operations multiple times, and the method in this paper does it twice. Represents the normalization function, which is used to convert Processed basis vectors Normalized to a probability distribution, that is, the value of each position represents the corresponding probability, so as to facilitate the subsequent weighted aggregation of neighbor features. Represents the attention weight, reflecting the importance of the neighborhood points to the center point.

[0051] Finally, the score is multiplied by the local corresponding point, and the symmetric function Sum is used to realize the dynamic aggregation of the neighborhood features of each point, as shown in formula (9):

[0052] (9)

[0053] in, Derived from formula (8), it represents the attention weight, which is used to weightedly aggregate neighbor features. Derived from formula (6), it is a feature representation that combines the neighboring point value vector and coordinate information. Represents the Hadamard Product, which is the element-by-element multiplication of two matrices or tensors of the same shape. Represents the sum function Sum, K represents the number of neighboring points. With the feature tensor Multiply and then pass the function Sum and generate the final feature representation of the current center point , i represents the number of center points.

[0054] The entire process effectively integrates the spatial geometric relationship and semantic features of the point cloud, improving the model's perception and expression capabilities of complex structures.

[0055] (3) Adaptive Feature Aggregation Pooling Module

[0056] The traditional attention pooling module first concatenates the spatial position information enhanced by the local spatial encoding module with the original features of the corresponding point (such as color features) to obtain a fused feature vector. Next, a multi-layer perceptron is used to model this fused feature and learn the corresponding attention scores, enabling the network to automatically focus on key features. Finally, the feature vector is weighted and summed with the corresponding attention scores to obtain a feature representation that contains rich local structural information. However, this method of directly concatenating spatial encoding with original features that have not undergone neighborhood interaction means that the attention network can only calculate weights based on local static features (the semantics of the neighboring points themselves) and position encoding information. It lacks awareness of higher-level contextual relationships within the neighborhood, cannot distinguish the importance of neighborhoods in the channel dimension in a fine-grained manner, and has relatively limited representational capabilities.

[0057] Therefore, in order to enable the model to accurately capture the most representative local patterns when the geometric structure of the point cloud is complex and the semantic relationships between different neighborhoods vary greatly, such as Figure 5 As shown in the figure, this application proposes an AFAP module, which uses spatial position encoding information and self-attention features for splicing, and then uses channel-independent attention for more detailed weighting, which can more accurately focus on important information in different channel dimensions within the neighborhood. Figure 5 This module concatenates the features of the fused neighborhood context obtained by the paired local attention module with the spatial encoding, and then performs weighted aggregation of the importance of each neighborhood point through channel-independent attention pooling, thereby generating a global representation of each point to preserve the transmission and influence of global information, which helps the network further distinguish finer-grained spatial relationships. The formula is as follows:

[0058] (10)

[0059] in, is the output of the paired local attention module, representing the feature representation of the i-th center point, It is the position feature after the geometric neighborhood module is coded, i represents the number of center points, and K is the number of neighboring points of the point set in the geometric neighborhood coding module. To aggregate them, in the adaptive feature aggregation pooling module, use Join as a Relationship Function . Represents the splicing operation, and location information The channel dimension is merged to obtain the aggregated feature with increased feature dimension .

[0060] To obtain the aggregated features Attention weight, design a function consisting of a multi-layer perceptron (MLP) and a normalized function softmax , the learnable weights in the MLP Assign a unique attention score to each feature. The attention score can be regarded as a mask that automatically selects important features. is the Hadamard product. Then these features are weighted and summed to generate a new feature vector with rich semantic information. The formula is as follows:

[0061] (11)

[0062] in, represents the aggregated features of the Kth neighborhood point of the i-th center point, is the learnable weight in the MLP, which can be used for each The feature is given a unique attention score, Represents a function consisting of MLP and normalization function softmax, which is implemented by weight matrix Learn the correlation between features and generate attention scores, which are then combined with the aggregated features Weighted summation to obtain the final aggregate feature , Represents the Hadamard Product, which is the element-by-element multiplication of two matrices or tensors of the same shape. It is used here to combine attention weights with features. Represents the summation function Sum.

[0063] (4) Dual Attention Residual Fusion

[0064] In view of the fact that a single attention pooling module is limited to the local neighborhood, lacks the ability to perceive long-distance point pair relationships and global dependencies, and the weights of the attention pooling module tend to be averaged in areas where features are too similar or relatively uniform, ignoring the importance of key points, this application stacks the DLA module and AFAP module with jump connections in series into a DARF module, so as to better capture the key local features in the point cloud, so that the aggregated features can fully represent the local geometric relationships, and further improve the weakly supervised semantic segmentation effect.

[0065] The structure of this module is as follows Figure 6As shown. The input of this module includes point cloud spatial information and feature vectors. The spatial information is used to construct the GNE module embedded in the DLA module and AFAP module. The DARF module first deeply fuses the point cloud spatial information and feature information in the paired local attention module, obtains the features of the fused neighborhood context, and then re-splices them with the spatial encoding and performs secondary attention pooling in the AFAP module. This layered splicing and multi-round fusion method improves the network's ability to model complex geometric shapes. The operation steps of this module are as follows:

[0066] First, input features The output dimension is reduced to half of the original dimension by passing it through a shared MLP1 layer and a ReLU activation function is applied to obtain the transformed feature vector As shown in formula (12).

[0067] (12)

[0068] in, Center point The eigenvector of Represents a list of the number of output channels in the model configuration, Indicates the number of output channels of the specified convolutional layer, that is, the number of output channels half, represents a multilayer perceptron A with one linear layer, Refers to the eigenvector After dimensionality reduction The processed feature tensor.

[0069] Then, the position encoding and the transformed eigenvectors Input into the DLA module for processing to obtain the local attention feature vector , then the eigenvector and The input is further processed into the AFAP module to generate a feature vector containing rich semantic information. , as shown in formulas (13) and (14) respectively.

[0070] (13)

[0071] (14)

[0072] in, Derived from formula (12), the input feature Through the dimension reduction of MLP1, Derived from formula (1), it represents the encoded position information, and the transformed feature vector and location information Input The local attention feature vector is processed in the module to obtain . The local attention feature vector and location information Input Further processing is performed in the module to generate a feature vector containing rich semantic information .

[0073] In order to perform more complex feature learning, the feature vector is then convolved to double the number of channels. The formula is as follows:

[0074] (15)

[0075] in, Derived from formula (14), it represents the feature vector containing rich semantic information. Represents a list of the number of output channels in the model configuration, Indicates the number of output channels of the specified convolutional layer, that is, the number of output channels twice as much, Represents a multilayer perceptron B with one linear layer, and a feature vector containing rich semantic information go through After processing, the number of feature channels is the number of output channels twice, aligned with the number of feature channels of the subsequent residual connection.

[0076] Subsequently, the original input features are introduced into the model using residual connections, and their feature channels are adjusted through convolution transformation to obtain the target features. Channel-aligned feature vectors The formula is as follows:

[0077] (16)

[0078] in, Center point The eigenvector of Represents a list of the number of output channels in the model configuration, Indicates the number of output channels of the specified convolutional layer, that is, the number of output channels twice as much, Represents a multilayer perceptron B with one linear layer, the feature vector of the center point go through After processing, a feature vector containing the original information is generated , the number of characteristic channels is the number of output channels twice as much.

[0079] Finally, the feature vector containing the original information and feature vectors containing rich semantic information The merge is performed and the Leaky ReLU function is activated to further enhance the expressive power of the model. The formula is shown in (17).

[0080] (17)

[0081] in, Derived from formula (15), it represents the feature vector containing rich semantic information in the main path, Derived from formula (16), it represents the feature vector containing the original information in the residual branch. It is a nonlinear activation function that can help the model learn complex feature relationships. The output of the residual branch Perform element-by-element addition and introduce nonlinear activation function , generate the final residual features .

[0082] Through this series of steps, the entire process can effectively extract and optimize 3D point cloud features. In particular, with the help of residual connections, the model can avoid information loss and accelerate the training process.

[0083] 1. Dataset

[0084] This application verifies the effectiveness of the DDLA-Net network on the S3DIS dataset. S3DIS is a high-quality indoor scene dataset, approximately 14.2GB in size. It contains six different indoor areas, covering a total area of ​​approximately 6020 square meters, including 272 rooms. The dataset has 13 categories (such as walls, floors, and chairs). A 1m×1m area is randomly selected from the plane defined by the X and Y axes.

[0085] Unlike fully supervised learning settings, weakly supervised approaches assume that only a subset of points in the training set are labeled, while a large number of points remain unlabeled. However, for performance testing, all points in the test / validation set are assumed to be labeled. Therefore, the S3DIS dataset needs to be processed for weakly supervised training.

[0086] In this application, a weakly supervised strategy is employed on the S3DIS dataset, labeling only a small portion of the point cloud, while the remaining points remain unlabeled. Following the random labeling principle of SQN (a classic large-scale point cloud weakly supervised semantic segmentation network model), a binary mask is maintained for each point, where 1 indicates labeled and 0 indicates unlabeled. The entire process consists of two steps: First, the original input point cloud is downsampled to a grid with a side length of 0.04m according to the dataset preprocessing method based on PseudoGrid (in 3D point cloud processing, PseudoGrid is a spatial grid-based preprocessing strategy whose core idea is to sparsify and structure the original point cloud by constructing a virtual grid structure). Compared with pure random downsampling, grid downsampling better preserves spatial information and makes the output point cloud more uniformly distributed, thus providing a stable learning input for the model. Second, within the downsampled training point set, a subset of randomly selected points are labeled according to a preset ratio, while the remaining points are not subject to supervision. The advantage of this approach is that the labeling tool is easier to use and is more suitable for real-world scenarios where manual intervention is required. Through this strategy, the network can achieve good 3D point cloud semantic segmentation results on the S3DIS dataset using only very little labeled data.

[0087] The invention was experimented on a server equipped with an NVIDIA RTX2080Ti GPU and 43GB of memory, using the TensorFlow framework. TensorFlow is Google's open-source machine learning framework that provides a computational graph mechanism, supports tensor operations and automatic differentiation, and is suitable for building and training neural network models.

[0088] 2. Experimental parameter settings

[0089] The network was trained for a total of 100 epochs. In this application, the batch size was set to 3. Batch size represents the number of samples used in each model parameter update. In this experiment, batch size = 3 means that only 3 point cloud samples were used for forward and backward propagation in each training iteration. The number of nearest neighbors K was set to 3, the initial learning rate was set to 0.01, and the AdamW optimizer was used. AdamW is an optimization algorithm based on the adaptive moment estimation (Adam). Its core concept is to decouple weight decay, that is, to separate the regularization term (weight decay) from the learning rate adjustment during the parameter update process. This design solves the problem of weight decay being affected by the learning rate in the traditional Adam optimizer, improving the model's generalization ability in large-scale deep learning tasks. In layman's terms, AdamW dynamically adjusts the "learning strength" of each parameter during training, allowing the model to learn the task faster and more accurately. Finally, a cosine annealing function was used to gradually decay the learning rate. The DDLA-Net was trained end-to-end using 0.1% and 0.01% randomly annotated points.

[0090] 3. Experimental Results

[0091] The Stanford 3D Indoor Scene Dataset (S3DIS) is a benchmark dataset for 3D point cloud semantic segmentation. It consists of six independent indoor scene areas (Areas 1-6) captured by Stanford University using a Matterport scanner. Area 5, the default test area, is widely used for model performance evaluation. Table 1 shows that the DDLA-Net network performs semantic segmentation on the point cloud within Area 5 in the S3DIS dataset. Among the segmentation metrics, mIOU (miounity under One-Unit Union) is 59.05% and OA (Overall Accuracy) is 86.35%. OA (Overall Accuracy) represents the overall accuracy, which is the proportion of correctly predicted points out of the total number of points, reflecting the model's global classification accuracy. mACC (Mean Accuracy) represents the average accuracy, which is the average accuracy of each class, highlighting the model's balanced learning ability across all classes.

[0092] Table 1 shows that when only 0.1% of the training labels are used, the segmentation performance index mIoU of the proposed method is 59.05%, which is better than the weakly supervised methods Π-model, MT and Xu. Compared with the classic fully supervised semantic segmentation methods PointNet and PointCNN, the mIoU performance is improved by 17.95% and 1.75% respectively.

[0093] Table 2 shows that DDLA-Net achieves 11.34 mIoU (mean intersection over union) higher semantic segmentation accuracy than SQN (a classic network model for weakly supervised semantic segmentation of large-scale point clouds) with 0.01% training labels. mIoU (mean intersection over union) is a core metric for measuring the overlap between a model's predicted regions and ground-truth regions in 3D point cloud semantic segmentation. It is calculated based on the mean intersection over union (IoU) of each category and comprehensively reflects the model's ability to discern semantic boundaries and class balance. Furthermore, DDLA-Net does not utilize the retraining method used in SQN (a classic network model for weakly supervised semantic segmentation of large-scale point clouds). Compared to WSSS-ST, DDLA-Net achieves a 2.54% performance improvement, demonstrating its effectiveness. Furthermore, with 0.1% training labels, DDLA-Net achieves comparable performance to fully supervised methods such as RandLA-Net and PseudoGrid, only 4-5 mIoU lower than these methods.

[0094] Table 1 Quantitative results of different methods on S3DIS dataset Area5

[0095]

[0096] Table 1. Semantic segmentation results of different networks on point clouds within Area 5 in the S3DIS dataset, including fully supervised methods (PointNet, PointNet++, and RandLA-Net) and the proposed DDLA-Net. Both mean Intersection over Union (mIoU) and per-class Intersection over Union (IoU) are reported. The IoU (Intersection over Union) metric, or intersection over union (IoU), is a commonly used standard metric in semantic segmentation. It measures the ratio of the intersection over union (IoU) between the predicted result and the ground-truth label, i.e., the degree of overlap between the predicted region and the ground-truth region of a 3D point cloud semantic segmentation model. IoU is calculated as: IoU = TP / (TP + FP + FN), where TP represents true positives (1 positive), FP represents false positives (0 positive), and FN represents false negatives (0 negative). In the table, bold indicates the best results in the weak label rate setting, while underlined indicates the best results in the full label rate setting.

[0097] Table 2 Quantitative results of different methods on the S3DIS dataset Area5 (including comparison results at a 0.01% annotation rate)

[0098]

[0099] Table 2. Semantic segmentation results of different networks on point clouds within Area 5 in the S3DIS dataset, including fully supervised methods (RandLA-Net and PseudoGrid), weakly supervised methods, and the proposed DDLA-Net. Mean Intersection over Union (mIoU) is reported. Bold indicates the best results in the weak label rate setting.

[0100] In summary, this application significantly reduces the dependence on labeled data by designing a variety of innovative modules, and improves the segmentation performance under limited annotation conditions. First, the GNE module is used to encode the local coordinate features of some labeled point clouds, thereby strengthening the local geometric information representation of the point cloud. Secondly, the DLA module is used to calculate the similarity of point cloud features, and point cloud features of different resolutions are extracted through multi-scale convolution operations, thereby enhancing the perception of small objects and complex geometric shapes. In order to better aggregate multi-scale features, this application introduces the AFAP module, so that the model can accurately focus on the most important features. Finally, the DARF module stacks the DLA module and the AFAP module in series through jump connections, further improving the modeling ability of complex geometric shapes. Experimental results show that on the S3DIS dataset, when only 0.1% training labels are used, the performance of this method is close to that of the fully supervised method, and when 0.01% training labels are used, the performance is better than the existing weakly supervised method, verifying the effectiveness of this application.

[0101] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions that are not conceived through creative work should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection defined in the claims.

Claims

1. A point cloud weakly supervised semantic segmentation method based on a dual self-attention mechanism, characterized by: include: Acquire partially annotated 3D point cloud data containing spatial coordinate information; encode the local coordinate features of the point cloud data using the GNE module to generate a discriminative local information representation; use the DLA module to calculate the feature similarity between points in the point cloud data, generate attention weights, and adjust the point feature representation; and perform weighted aggregation of the adjusted point features using the AFAP module to generate a global feature representation. The DLA module and the AFAP module are stacked in series to form the DARF module, which improves the modeling ability of complex geometric shapes. The point cloud data is semantically segmented based on a weakly supervised training strategy and the segmentation results are output. The GNE module encodes the local coordinate features of the point cloud data, including: for each center point in the point cloud data, using the nearest neighbor algorithm to search for its neighboring points based on Euclidean distance; obtaining the relative spatial position information and Euclidean distance information between the center point and the neighboring points; enhancing the geometric representation of the local area based on the relative spatial position information and Euclidean distance information; inputting the enhanced geometric representation into the convolution layer for linear transformation to generate a discriminative local information representation; using the local information representation to reduce excessive reliance on spatial information and improve robustness to noise and irregular distribution; Semantic segmentation of point cloud data is performed based on a weakly supervised training strategy, including: obtaining a small number of labeled points in the point cloud data, and keeping the remaining points unlabeled; preprocessing the point cloud data through grid downsampling to retain spatial distribution information; randomly selecting some points according to a preset ratio to add labels; extending the labeled information to unlabeled points through a label propagation mechanism; combining the semantic similarity mechanism to enhance the effectiveness of training data and improve the model generalization ability.

2. The point cloud weakly supervised semantic segmentation method according to claim 1, characterized in that The DLA module is used to calculate the feature similarity between points in point cloud data, including: obtaining the center point features and their neighborhood point features in the point cloud data; extracting point cloud representations of different resolutions through multi-scale convolution operations to alleviate the information loss caused by sparsity; calculating the similarity between the center point features and the neighborhood point features based on the query-key-value structure to generate dynamic attention weights; adjusting the point feature expression according to the dynamic attention weights to enhance the understanding of local and global semantics; and incorporating local position encoding information into the attention weights to improve the perception of geometric structures.

3. The point cloud weakly supervised semantic segmentation method according to claim 1, characterized in that The adjusted point features are weightedly aggregated through the AFAP module, including: obtaining the fused neighborhood context features output by the DLA module; splicing the fused neighborhood context features with the spatial encoding information; weighting the importance of each neighborhood point through channel-independent attention pooling to generate a global feature representation; and enhancing the interactive expression ability of local and global information based on the global feature representation.

4. The point cloud weakly supervised semantic segmentation method according to claim 1, characterized in that The DLA module and the AFAP module are stacked in series to form a DARF module, which includes: obtaining the spatial information and feature vector of point cloud data as input; deeply fusing the spatial information and feature vector through the DLA module to generate fused neighborhood context features; re-splicing the fused neighborhood context features with the spatial encoding information; performing secondary attention pooling on the spliced ​​features through the AFAP module; and introducing the original input features into the model using a residual connection structure, adjusting the feature channels to avoid information loss.

5. The point cloud weakly supervised semantic segmentation method according to claim 1, wherein: Output segmentation results, including: obtaining point cloud features processed by the DARF module; predicting the category of unlabeled points based on point cloud features; generating segmentation results based on category prediction to characterize the semantic category of each point in the point cloud data; and achieving efficient semantic understanding of large scene point clouds through segmentation results.

Images