Point cloud semantic segmentation method and model based on density perception and feature enhancement

Abstract

The invention relates to a point cloud semantic segmentation method and model based on density perception and feature enhancement, and the method comprises the steps: firstly, carrying out the dynamic density perception sampling in an input point cloud, and carrying out the grouping according to the semantic feature distance; then, after multi-level extraction of deep semantic information, enhancement of relative position information is carried out, and local features of different areas are enhanced; and finally, guiding to recover sampling points according to semantic homogeneity in the up-sampling of the decoder, and fusing the point-level semantic features and the channel-level semantic features by using a bidirectional self-attention mechanism. The model comprises a point cloud input module, a down-sampling module, a cross attention module, a position feature enhancement module, an up-sampling module, a point cloud output module and a trigger threshold judgment module. The method is tested on public data sets S3DIS and Scannet, mIoU reaches 73.0% and 74.8% respectively and is increased by 2.6% and 4.2% respectively relative to a baseline model, and the effectiveness of the method is proved.

Description

Technical Field

[0001] This patent application belongs to the field of point cloud semantic segmentation technology, and more specifically, it relates to a point cloud semantic segmentation method and model based on density perception and feature enhancement. Background Art

[0002] In recent years, point clouds have developed rapidly in the fields of environmental perception and autonomous driving, thus attracting widespread attention from researchers. However, due to the characteristics of the sensor itself, point clouds are sparse and disordered, making it challenging to identify targets from point clouds. Compared with two-dimensional images, point clouds retain the geometric and topological structure of objects in three-dimensional space as a whole. The ability to understand and analyze three-dimensional data is becoming increasingly important in computer vision and computer graphics. The use of deep learning methods to analyze three-dimensional objects is inspired by the successful experience of these technologies in two-dimensional images and one-dimensional text. Three-dimensional point cloud semantic segmentation can achieve accurate understanding and analysis of the three-dimensional environment by semantically labeling point clouds, thereby solving problems such as object recognition, scene analysis and decision-making, and promoting the development of three-dimensional perception and applications.

[0003] The semantic segmentation process of point cloud is: first downsample, gradually extract features, then upsample to restore point cloud, and segment point cloud by features. Therefore, in the field of point cloud semantic segmentation research, downsampling is an important operation for processing point cloud. Usually, a point cloud scene contains hundreds of thousands of points and their label data, but it contains a large amount of redundant data with repeated semantic features. The purpose of downsampling is to reduce the size of point cloud as much as possible while avoiding losing too much semantic information of point cloud, so as to improve the efficiency of processing point cloud. Most existing methods use farthest point sampling (FPS). The FPS method first randomly selects the initial sampling point, then calculates the point with the largest Euclidean distance from all points to the sampling point as the new sampling point, and finally repeats this operation until the sampling is completed. However, FPS only considers the Euclidean distance information between points, ignoring the semantic feature relationship between points. Later, some researchers proposed a sampling method based on point cloud density to calculate point cloud density, which can control the number of search points to calculate the average distance to the center point, and the smaller the distance, the higher the density; it can also control the number of points in a certain range, and the more points there are, the higher the density. Most of the existing methods are based on the former, which will lead to the problem of "density deception", that is, no matter which neighborhood point has the same influence on the density of the center point, resulting in a larger density value for each center point; in order to highlight that the points close to the center point have a greater influence, it is necessary to combine the above two ideas and introduce a dynamic adjustment parameter, which is defined as the ratio of the number of point clouds within the average distance range to the total number of point clouds in the neighborhood; finally, the density of the point cloud is calculated within a certain range, and a density scoring strategy is proposed to determine the sampling set based on the score and sampling rate.Figure 1 As shown in the figure, the uniform type is the situation where the spatial distribution of neighborhood points is relatively uniform, the concentrated type reflects that the spatial distribution of neighborhood points is mostly close to the center point, and these close points have a greater impact on the density of the center point. Conversely, the discrete type reflects the situation where there are too many neighborhood points far from the center point, such as Figure 2 As shown in the figure, the purple dotted edge is the calculated average distance. The ratio of the number of points within this distance to the total number is obviously different, resulting in different degrees of "density deception". In order to solve the above problems, it is necessary to propose a dynamic-scale sampling and grouping model (DSG), which determines whether to scale the point cloud by triggering a threshold, determines the point cloud mapping rules, and guides the subsequent upsampling process.

[0004] As the level of sampling point feature extraction deepens, the number of points decreases, and the size of the feature map continues to decrease. Each feature map contains rich deep semantic information, but at the same time loses a lot of relative position information. These sparse point clouds are difficult to fully represent the three-dimensional structure of the entire scene by lacking semantic information of position features. Convolution and pooling may further cause the loss of precise position information. To solve this problem, the model of the present invention proposes a position feature enhancement model (PFEM) module, which obtains the global position dependency relationship between the current point and the input point through a cross-attention mechanism, establishes a connection between the aggregated sampling points and the input features, enhances the sampling point features, and aligns the feature scale with a 1×1 convolution kernel (alignment loss refers to the length of the feature vector of the extracted point cloud to unify the length so as to process it at a uniform size). Finally, while performing a dual self-attention mechanism, position encoding is introduced to optimize the matching ability with the feature map.

[0005] The Transformer-based 3D point cloud semantic segmentation solution adopts an end-to-end "encoding-decoding" structure. Transformer can help the network capture global and local features and learn the importance of each point in semantic segmentation. Point Transformer introduces the self-attention mechanism into the task of 3D point cloud semantic segmentation, encodes and decodes point cloud features in a local range, and ignores global features. PCT extracts global features of point clouds through maximum pooling and average pooling. When performing maximum pooling, it selects a multi-neighborhood feature encoding structure, which has a good classification and segmentation effect. Although different Transformer-based methods have shown strong feature learning capabilities in point cloud applications, due to the large number of point clouds, the modeling efficiency of semantic points of such methods is still insufficient.

[0006] Therefore, it is necessary to propose a double transformer block (DTB) method that integrates the point level transformer (PLT) and the feature channel level transformer (CLT), so that the Transformer structure can learn semantic features of different levels of deep sampling points based on density perception. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a point cloud semantic segmentation method and model based on density perception and feature enhancement, so that the Transformer structure can learn semantic features of different levels of deep sampling points based on density perception, thereby solving the problems mentioned in the background.

[0008] In order to solve the above problems, the technical solution adopted by the present invention is:

[0009] A point cloud semantic segmentation method based on density perception and feature enhancement is proposed. First, dynamic density-aware sampling is performed in the input point cloud and grouped according to the semantic feature distance. Secondly, the relative position information is enhanced after multi-level extraction of deep semantic information to strengthen the local features of different regions. Then, the sampling points are restored according to the semantic homogeneity guidance in the decoder upsampling, and the point and channel-level semantic features are fused using a bidirectional self-attention mechanism. Finally, experimental verification is performed.

[0010] At the same time, the present invention also discloses a point cloud semantic segmentation model based on density perception and feature enhancement, which is used to implement the above method. The model includes a point cloud input module, a downsampling module, a cross attention module, a position feature enhancement module, an upsampling module, and a point cloud output module, wherein

[0011] Point cloud input module, used to input three-dimensional point cloud;

[0012] A downsampling module, used to downsample the input 3D point cloud;

[0013] The cross attention module is connected to the input of the downsampling module and the input of the upsampling module, corresponding to the downsampling operation and the upsampling operation, which are used to reduce the number of 3D point clouds and restore semantic point clouds after segmentation respectively;

[0014] The position feature enhancement module is connected to the output of the downsampling module and the input of the upsampling module, and is located between the downsampling operation and the upsampling operation;

[0015] The point cloud output module is connected to the output end of the upsampling module and is used to output semantic point clouds.

[0016] It also includes a trigger threshold judgment module, which is connected to the downsampling module and dynamically adjusts the density of the three-dimensional point cloud through the trigger threshold to cooperate with the operation of the downsampling module.

[0017] Due to the adoption of the above technical solution, the beneficial effects achieved by the present invention are:

[0018] (1) A dynamic scale sampling and grouping (DSG) method is proposed. Through the dual parameter constraints of distance and density, the fine-grained features of the local area of ​​each point are dynamically perceived, and the point cloud sampling rate is updated using the distance between points and the number of points within the distance. At the same time, a point cloud sampling scoring mechanism is introduced to effectively improve the effect of point cloud sampling and grouping;

[0019] (2) A position feature enhancement PFEM method is proposed. Through the Transformer module, the relative position relationship between the original point cloud containing rich position structure information and the sparse point cloud containing rich semantic feature information is established, so that the sparse point cloud has rich semantic features while enhancing the ability to express the scene structure.

[0020] (3) The two-level features of PLT and CLT are integrated in the bidirectional Transformer model to improve the modeling efficiency of the algorithm for global and local geometric features. The model proposed in this paper is experimentally tested in two datasets, S3DIS and ScanNet, and the improvements are 2.6% and 4.2% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 These are three morphological diagrams of the point cloud density of the method of the present invention.

[0022] Figure 2 It is the local density dynamic contraction of the method of the present invention.

[0023] Figure 3 It is a network structure diagram of the method of the present invention.

[0024] Figure 4 It is a cluster diagram of KNN and semantic feature distance of the method of the present invention.

[0025] Figure 5 It is a DTB network model diagram of the method of the present invention.

[0026] Figure 6 This is a visualization result diagram of the S3DIS data set of the method of the present invention.

[0027] Figure 7 This is a visualization result diagram of the ScanNet data set of the method of the present invention.

[0028] Figure 8 It is a connection block diagram of the model of the present invention.

[0029] Fig. 9 It is a connection structure diagram of MLP in the model of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below with reference to the embodiments.

[0031] A point cloud semantic segmentation method and model based on density perception and feature enhancement, as described below.

[0032] 1 Related Work

[0033] According to the different ways of processing data in point cloud semantic segmentation, the segmentation methods use a multi-layer perceptron (MLP) method for point-by-point feature extraction, a point convolutional neural network-based method, and an attention mechanism-based method.

[0034] As a pioneering work in semantic segmentation of point clouds, the PointNet method uses MLP to increase the dimension of data features in order to enrich the features of each point. At the same time, the method uses the maximum pooling method to aggregate the global features. When performing semantic segmentation tasks, the global features are spliced ​​with the features of each point. In addition to its own features, each point also contains global features. This method is applied to the properties of discrete points and global attributes, but ignores the contextual information between points, that is, it lacks the ability to mine local features. To solve this problem, PointNet++ adds local feature extraction modules of different scales on the basis of PointNet, collects multiple points as center points through FPS, selects multiple different spherical radii to divide multiple regions, and splices features of different scales to obtain local features. In order to realize the direction encoding and scale perception of point clouds, the PointSIFT module uses three-stage ordered convolution stacking to encode information from eight spatial directions, connects multi-scale features to improve adaptability to different scales. This method lacks modeling of the mutual interaction between different points. To this end, PointWeb densely constructs a local fully connected network to explore the relationship between all point pairs in the local area, and proposes an adaptive feature adjustment (adaptive feature Adjustment, AFA) module is used to realize information exchange and feature refinement.

[0035] Methods based on point convolution tend to design more efficient convolution operators. PCNN, as a point-by-point convolution method, uses a fixed-size convolution kernel. The points in the convolution kernel are multiplied by the weights to obtain the average value of the convolution kernel, which can be applied to each point in the point cloud to learn point features. Each point in the point cloud has several sub-neighborhoods. The gradient and loss function are calculated based on the weights and data to train the network. In the KPConv kernel point full convolution network, the convolution weight is determined by the Euclidean distance to the kernel point, and the position of the kernel point is expressed as an optimization problem with the best coverage in the spherical space. The radius neighborhood is used to maintain a consistent receptive field, and each layer uses grid sampling to achieve high robustness under different point cloud densities. In MCCN, convolution relies on the Monte Carlo estimation result of the sample density function, and Poisson disk sampling is used to construct a point cloud hierarchy. The A-CNN annular convolution operator method can better capture the local neighborhood information of each point by specifying (regular and dilated) annular structures and directions. It can adapt to the geometric variability and scalability at the signal processing level and be used in semantic segmentation tasks in large-scale scenes.

[0036] The attention-based method mainly adopts the encoder-decoder structure in Transformer; PointTransformer is a structure based on vector attention, and the encoder only uses Point Transformer blocks, point-by-point transformation and pooling operations to build and use for point cloud segmentation tasks; Point Transformer proposes an upsampling module. As mentioned above, the point cloud segmentation process is: downsampling first, then upsampling. Both processes are indispensable. The module consists of a linear layer, a batch normalization layer, a ReLU, and a trilinear interpolation for feature mapping. The model introduces jump connections to facilitate back propagation. PCT, as a global Transformer network, applies a neighbor embedding architecture to map the point cloud to a high-dimensional feature space with three-dimensional coordinates as input. This operation can also incorporate local information into the embedded features. These features are then input into four stacked global Transformer blocks to learn semantic information for classification and segmentation. In addition, inspired by the Laplacian matrix in the graph convolutional network, PCT proposes an improved self-attention module that can improve attention weights and reduce the impact of noise. PVT is a point cloud learning backbone network based on Transformer with three-dimensional voxels as input. Inspired by the recent SwinTransformer, a sparse window attention operation is designed to perform self-attention within non-overlapping three-dimensional voxel windows in a shifted window configuration. A relative attention operation is also introduced to calculate fine-grained features of points. Stratified Transformer uses dense local points and sparse distant points as the key vectors of each query vector, which can pass messages between cubic windows and capture global information; but the above models are only limited to the point-by-point Transformer module; channel-by-channel Transformer focuses on measuring the similarity of different feature channels; they can improve contextual information modeling by highlighting the interaction between different channels.

[0037] Inspired by bidirectional Transformer and Gaussian kernel density estimation, this paper proposes a density estimation method based on dynamic scale sampling. The bidirectional Transformer fuses the position encoding information of the local area to perform fine-grained local feature extraction. The results of the datasets S3DIS and ScanNet-v2 show that the method proposed in this paper has improved the semantic segmentation task.

[0038] 2 Model approach

[0039] 2.1 Overall Framework

[0040] The network structure of this model is as follows Figure 3As shown in the figure, an "encoder-decoder" structure is adopted; first, the original point cloud is input, and a layer of MLP is used to upgrade the dimension of the feature to increase the number of feature channels of each point. The dimension after dimensionality upgrade is d0. Then, in the encoder, dynamic sampling grouping DSG and bidirectional Transformer based on self-attention mechanism are used to extract deep semantic features of the point cloud; then, the contextual point cloud position information is fused in PFEM, and the enhanced relative position information is aggregated through graph convolution and maximum pooling; finally, the point cloud-level self-attention PLT and channel-level self-attention CLT in the DTB module are fused with the local features of DSG to efficiently solve the semantic segmentation task of the point cloud.

[0041] 2.2 Dynamic Sampling and Grouping

[0042] Input point cloud P = {p i |i=1,2,…,n}, the entire set contains n points, among which p={x,y,z,r}, x,y,z are the three-dimensional coordinates of the point respectively, and r is the reflectivity of the point; when grouping point clouds, this model adopts a grouping scheme based on semantic feature distance, so that points with high semantic homogeneity will be grouped into the same group, thus avoiding the misclassification of sample points belonging to other classes; inspired by GAM, the present invention designs a lightweight gradient descent scheme to extract local semantic features and performs fine-grained local feature analysis on each group of point clouds; this section explains the design ideas of DSG in two parts.

[0043] 2.2.1 Density Perception and Feature Distance

[0044] For a region of interest, its density is determined by the number of points in the region; for a point, its density is determined by the number of surrounding points and the distance between the points. First, the K nearest neighbor algorithm is used to initialize the neighborhood range of each point, select k points closest to each other, and calculate the average distance from each point to the center point, as shown in formula (1):

[0045]

[0046] Among them, ρ(p i ) represents the neighborhood range of the i-th point, pi represents the i-th point; p j is a neighborhood point in the range, then j∈[1,k] and j is an integer; g i is the average distance; therefore, there is a Gaussian kernel density u i :

[0047]

[0048] Where σ is the bandwidth, which can control the local range of the Gaussian function; the set U = {u1,u2,...,u n} is the point cloud density of each point; in order to further highlight the importance of points close to the center point, the average distance is used as the ball query radius of the center point, and the number of points within the ball query radius m is counted i ; The importance of a point is calculated as follows:

[0049]

[0050] Among them, η i is the dynamic coefficient, indicating the importance of points closer to the center point; Combining equations (2) and (3), point p i The Gaussian kernel density is:

[0051] s i =u i *η i (4)

[0052] Set S = {s1,s2,...,s n} is the point cloud density for each point; according to the similarity between the point cloud densities, for each point p i Calculate the semantic feature distance v i ; Find a density set α for each point i , satisfying α i ={s j |s j ≥s i ,s j ∈S}, s j is to find a point greater than p i The corresponding s i Point cloud.

[0053] but:

[0054]

[0055] Among them, v i It represents the minimum semantic feature distance between the point and any other point with (relatively) high local density; the distance index of the point with the highest local density is the maximum semantic feature distance between the point and any other point; for each point p i Get a comprehensive score κ i :

[0056] κ i =s i *v i (6)

[0057] The higher the score is, the greater the neighborhood density of the point is, the richer the semantic information of the point is, and the more representative it is, which increases the probability of it being retained during the sampling process.

[0058] For edge points of a certain category, simply grouping them according to the Euclidean distance of K clustering or ball query often results in misclassification, that is, they are classified into the neighborhood range of other categories of points. To address this problem, the present invention proposes a localization scheme based on semantic feature distance, which groups points with high semantic homogeneity into one group. Figure 4 It shows the difference between using K clustering and semantic feature distance; the enlarged image on the left is the result of K clustering, the center point of the blackboard type is represented by a larger red dot, other smaller red dots represent neighborhood points of this type, and the remaining gray points are not covered by the neighborhood. It can be found that some blue points belonging to the wall are also divided into the neighborhood, which shows a deviation; the enlarged image on the right is the result of grouping according to the semantic feature distance, the center point of the blackboard type is represented by a larger red dot, other smaller red dots represent neighborhood points of this type, and it can be found that the gray points belonging to the wall are almost not divided into the neighborhood; in this way, by selecting k points with the closest semantic feature distance, the neighborhood points of the center point are obtained.

[0059] 2.2.2 Local Feature Descriptors

[0060] Through dynamic sampling and grouping, the neighborhood range of each center point is obtained. The sampling points are represented by the set Q = {q1, q2, ..., q t} represents that the neighborhood range relationship corresponding to each point is represented by R, then Among them, r ij represents the jth point in the i-th group, t is the number of sampling points; at the same time, there are corresponding group features Where C is the number of feature channels, G is the representation of group features, and f is the representation of features. To aggregate the features of each neighborhood point, an intuitive method is to directly average the features of the neighborhood points in the cluster. However, even if different points have similar semantic labels, their importance is different. The importance parameter y i Inspired by this, the present invention introduces a parameter δ to explicitly indicate the importance of each neighborhood point, ξ i Represents a local feature descriptor, averaging neighborhood features under the guidance of importance:

[0061]

[0062] Among them, θ j It is the rth ij The learnable attention parameter matrix, ξ i is an aggregated feature that can more accurately describe local semantic information; however, the feature aggregation process only relies on the Euclidean distance to constrain, cannot fully utilize the information in the neighborhood of the center point, and is easily affected by outliers, and cannot fully fit the original geometric shape of the point cloud; in order to extract fine-grained features, the present invention proposes an A matrix with the same dimension as the A matrix in GAM and suitable for this topic* Matrix, the rest of the mathematical expressions remain unchanged:

[0063] A * =MLP(concat(ξ,A)) (8)

[0064] Concatenate ξ and A and use a layer of MLP to adjust them to the same dimension as ξ, outputting feature f c :

[0065]

[0066] Among them, λ is the balancing weight, · represents element-wise multiplication, and ψ represents the use of PointNet++’s local feature extractor to extract deep aggregation features; Obtained through maximum pooling At this time, use N c represents the sampling point, and its feature dimension is represented by d; k is the number of neighbor nodes, f d is the feature after maximum pooling.

[0067] 2.3 Location Information Enhancement Method

[0068] After deep semantic feature extraction through multi-layer encoders, each sampling point contains richer features; however, the feature points become increasingly sparse, and the overall structure of the point cloud changes, making it difficult to ensure that these sparse point sets can fully display the structure of the original point cloud; in order to reduce the loss of relative position information caused by the aggregation process and further enhance the features of the sampling points, the present invention designs a cross-attention mechanism to establish a connection between deep points and original input features.

[0069] According to the deep sampling point N c Get the Q matrix, and get the K matrix and V matrix through the original points containing rich position information: W Q , W K , W V The meanings correspond to the weight parameter matrices of Q, K, and V, that is, the parameter matrices that need to be trained; X is the feature matrix, and W is the coefficient to be solved.

[0070] Q=N c W Q

[0071] K=XW K

[0072] V=XW V (10)

[0073] A calculated in the previous section * With QK T Features Figure 1Perform softmax normalization to get the matrix M: d is the feature dimension, QK T Transpose to matrix multiplication and calculate vector inner product;

[0074]

[0075] Multiply the new feature map with V, concatenate the result with the deep point set feature, and aggregate the features through the graph convolution network and maximum pooling: N c Represents the sampling point.

[0076]

[0077] Among them, maxpool is the maximum pooling method, so that the obtained output L participates in the subsequent first layer decoder for upsampling.

[0078] 2.4 Bidirectional Self-Attention Module

[0079] The bidirectional self-attention Transformer module designed in the present invention includes two sub-blocks, namely, a point-level Transformer and a channel-level Transformer. Figure 5 The interaction and relationship between the two sub-blocks are shown in Figure 1. The output of the DSG model is expressed as f d ; In order to highlight the role between points and various channels to improve contextual information modeling and integrate global and local semantic information, this section will introduce the network architectures of PLT and CLT respectively.

[0080] 2.4.1 Point-level Transformer

[0081] As a point-by-point self-attention mechanism, the point-level Transformer aims to learn not only by utilizing the sampled points but also their neighborhood information. First, f d Mapped to two different spaces, we get two matrices Q of point-level queries and primary keys pl With K pl :

[0082] Q pl With K pl They are the point-level query matrix and the point-level primary key matrix respectively.

[0083] Q pl =W q f d

[0084] K pl =W k f d (13)

[0085] Wq With W k Is a trainable parameter matrix; using Q pl With K pl The feature map M can be calculated p :

[0086]

[0087] In formula (14), d is the feature dimension, Represents the transpose of a matrix.

[0088] Among them, P E is the position encoding information of Point Transformer; in order to fully integrate local information, this model selects This three-dimensional matrix is ​​used as V in self-attention pl , where V pl represents the value vector, and forms an attention mechanism with K and Q. Figure 5 As shown, f c Covering the d-dimensional features contained in the k different neighborhood points corresponding to each center point, the features of each center point are horizontally spliced ​​and vertically arranged k times. This process is represented by reshape, and we can get a Finally, after the maximum pooling, the two-dimensional matrix is ​​obtained. N c In k*kd, kd represents the product of k and d, N c Represents the number of sampling points.

[0089] V pl =maxpool(reshape(f c )) (15)

[0090] The elements in the attention map are regarded as the weights of the corresponding neighborhood feature set, and the output neighborhood feature set is obtained by calculating the weighted sum of the input set; all points including sampling points and neighborhood points are used to improve the feature learning ability; the output of the PLT model is:

[0091]

[0092] in, Represents the output features of the point-level Transformer; Mp represents the result of the operation of the key matrix and the query matrix.

[0093] 2.4.2 Channel-Level Transformer

[0094] The main focus of the channel-level Transformer sub-block is the channel space, so it is necessary to reduce the computational cost caused by the point space complexity; inspired by the geometric back-projection network, for a given fd , the context in each channel is simplified by a shared MLP operating on the channel vectors while fully preserving the information of each channel:

[0095] Q cl =MLP q (f d )

[0096] K cl =MLP k (f d )

[0097] V cl =MLP v (f d ) (17)

[0098] Among them, MLP q (·), MLP k (·) and MLP v (·) is the shared MLP layer, and the obtained Q cl , K cl With V cl Corresponding to the Query matrix, Key matrix and Value matrix of the self-attention mechanism; according to the Key matrix and the Query matrix, the relative relationship feature map is obtained

[0099]

[0100] Represents the transpose of the key matrix.

[0101] right Perform the maximum pooling operation, take the maximum value of each feature, and get vector; by continuous splicing, Repeat d times to restore the matrix size, becoming Compare this result with Subtract and add to V cl Multiply to get

[0102]

[0103] Select the maximum similarity along the channel direction, and obtain the similarity measure by calculating the difference. Channels with higher similarity have lower affinity. Add softmax to normalize the values. Used as a weight matrix for refinement; each channel can give higher weights to other different channels to avoid aggregating redundant information; the results of the two submodules are fused to obtain the output f t :

[0104]

[0105] in, is the result of formula (16).

[0106] LBR is a combination of linear layer, BatchNorm and ReLU layer; ft is the output result of DTB; finally, MLP is used for point cloud segmentation task, such as Fig. 9 As shown in the figure, it consists of two fully connected layers FCN (Fully Connected Network), a batch normalization layer BN (Batch Normalization) and a ReLU activation function (Rectified Linear Unit, also called linear rectification activation function) connected in sequence.

[0107] According to the above method, the present invention also discloses a point cloud semantic segmentation model based on density perception and feature enhancement, which can implement the above method. Figure 8 As shown, the model includes a point cloud input module, a downsampling module, a cross-attention module, a position feature enhancement module, an upsampling module, and a point cloud output module.

[0108] Point cloud input module, used to input three-dimensional point cloud;

[0109] A downsampling module, used to downsample the input 3D point cloud;

[0110] The cross attention module is connected to the input of the downsampling module and the input of the upsampling module, corresponding to the downsampling operation and the upsampling operation, which are used to reduce the number of 3D point clouds and restore semantic point clouds after segmentation respectively;

[0111] The position feature enhancement module is connected to the output of the downsampling module and the input of the upsampling module, and is located between the downsampling operation and the upsampling operation;

[0112] The point cloud output module is connected to the output end of the upsampling module and is used to output semantic point clouds.

[0113] It can also include a trigger threshold judgment module, which is connected to the downsampling module, and dynamically adjusts the density of the three-dimensional point cloud through the trigger threshold to cooperate with the operation of the downsampling module. The following experimental exploration can be achieved using this model.

[0114] 3 Experimental Exploration

[0115] In this section, we use S3DIS and ScanNet-v2 datasets to conduct experiments on semantic segmentation of 3D point clouds of indoor scenes; in the experiments of this invention, the hardware environment of the experimental platform uses 2 24GB NVIDIA RTX 3090 graphics cards, the operating system is Ubuntu 1.04 version, Python version is 3.7, Pytorch version is 1.7, cuda version is 11.1, Intel 11900k CPU, and 128GB memory; the number of sampling points is set to 64, and the network is trained using Adam Optimizer, the weight decay is 0.0001, and the initial learning rate is set to 0.001; the network is trained for 250 epochs; the batch size is set to 8.

[0116] 3.1 Datasets and Indicators

[0117] The S3DIS dataset for semantic scene parsing consists of 271 rooms in 6 areas of 3 different buildings; each point in the dataset is assigned a semantic label containing 13 categories (ceiling, floor, table, etc.); the ScanNet-v2 dataset contains a total of 1513 acquisition scene data, a total of 21 categories of objects; the experiment retains Area 5 in S3DIS for the test set, and the rest of the data is the training set; the experiment uses the overall point-by-point accuracy (OA), average category accuracy (mAcc) and average category intersection over union (mIoU) to analyze and evaluate the model, the formula is as follows:

[0118]

[0119]

[0120] Among them, k′ represents the number of categories, p ij Indicates the number of correctly predicted positive samples, p ji Indicates the number of positive samples predicted incorrectly, p ii Indicates the number of negative samples with incorrect predictions.

[0121] 3.2 Result analysis and visualization

[0122] The semantic segmentation results in Area 5 of the S3DIS dataset are shown in Table 1. The method proposed in the present invention is ahead of other methods in the three evaluation indicators of OA, mAc and mIou; the three indicators reached 92.4%, 79.2% and 73.0% respectively, which are 1.6%, 2.7% and 2.6% higher than the accuracy of Point Transformer respectively; among them, the experimental results of doors, sofas, bookshelves, boards and sundries are higher than other models, especially the accuracy of sundries is higher than 60%, and the accuracy of windows is more than 75%, all of which have achieved relatively high experimental results; the model proposed in the present invention shows good segmentation effect in multiple categories.

[0123] Table 1 Test results of S3DIS dataset

[0124]

[0125] The visualization of the experimental results is as follows Figure 5 As shown in the figure, the experiment compares the difference with the Point Transformer method. It can be seen intuitively that the accuracy of the above categories has been improved, and more points belonging to the same category have been segmented out; at the same time, the misclassification of points has been reduced; the window is marked as light purple, and the blank part of the window in the visualization interface of the model of the present invention tends to shrink compared with Point Transformer, the segmentation boundary of the black debris part is more obvious, and the overall structure is closer to the real label; the marking effect of the yellow door is darker in color, and the observability of the door frame is guaranteed; the black area but the bookshelf part that does not belong to the debris category has a higher confidence under the model proposed by the present invention, and has a higher response to the sensitivity of the boundary points; this shows that the solution designed by the present invention can effectively solve the problem of misclassification of boundary points of the same category, and can retain the obvious boundaries between different categories.

[0126] As shown in Table 2, in the ScanNet dataset, 70% of the dataset is used as a training set, 10% as a validation set, and 20% as a test set. This experiment compares and analyzes the experimental results of the validation set and the test set. The model proposed in this paper exceeds Point Transformer by 4.2% in the validation set and 4.5% in the test set, and is 3.3% ahead of the KPConv model, achieving competitive detection results. The visualization of the ScanNet dataset is shown in Figure 6 As shown in the figure, it has better segmentation results than the Point Transformer method in categories such as chairs, windows, and floors.

[0127] Table 2 ScanNet dataset test results

[0128]

[0129] 3.3 Ablation Experiment

[0130] Combined with the model proposed in the present invention, the ablation experiment mainly discusses dynamic density perception, location information enhancement and the CLT sub-block in the bidirectional Transformer architecture; for the sake of distinction, the dynamic density perception is recorded as ①, the CLT sub-block is recorded as ②, and the location information enhancement module is recorded as ③; ablation experiments are carried out on the three models respectively, and the results obtained are shown in Tables 3 and 4 for the S3DIS dataset and the ScanNet dataset respectively; in the ablation experiment on the S3DIS dataset, the intersection-over-union ratio of the baseline model is 70.4%, which is improved by 0.9% when ① exists alone, and there is a significant improvement when ①② exists at the same time or ①③ exists at the same time, which is 1.5% and 2% respectively, while when there is no ① and only ②③ exists at the same time, the improvement is only 1.6%, which fully illustrates the criticality of the dynamic coefficient proposed in this experiment; at the same time, when there is a CLT sub-block, it can be improved by 0.6%, and when there is ③, it can be further improved by 1.1%, which is enough to illustrate the necessity of introducing the location information enhancement module.

[0131] Table 3 Ablation experiments on S3DIS dataset

[0132]

[0133] Table 4 Ablation experiment of ScanNet dataset

[0134]

[0135] The ablation experiment of the ScanNet dataset can better illustrate the importance of the dynamic density perception module described above. As shown in Table 4, when only modules ② and ③ exist at the same time, the performance can be improved by 2.5%, while when all three modules exist, the performance can be improved by 4.2%, and the overall result can reach 74.8%.

[0136] 4 Conclusion and Outlook

[0137] From the above discussion, we can know that the present invention designs a bidirectional self-attention Transformer point cloud semantic segmentation model based on dynamic density perception and position information feature enhancement; first, through dynamic density perception, the importance of points close to the center point is highlighted, alleviating the problem of "density deception"; then, position information feature enhancement is added between the encoder and decoder to supplement the lost relative position information and strengthen local features; finally, feature fusion is performed under the action of the bidirectional Transformer to obtain the category to which each point belongs; however, this model does not assign different influence scores to each neighborhood point when dynamic density perception is used, and there are still deficiencies in the dual attention module. For example, the point-level self-attention and channel-level self-attention are simply added at the end, which may cause slight deviations in the overall features. In the future, we can continue to pay attention to the problem of point cloud density calculation and the direction of multi-scale and multi-modal feature fusion.

Claims

1. A point cloud semantic segmentation method based on density perception and feature enhancement, characterized in that The steps include: S1, dynamic density-aware sampling is performed in the input point cloud, and grouping is performed according to the semantic feature distance; S2, after extracting deep semantic information at multiple levels, enhance the relative position information and strengthen the local features of different regions; S3, in the decoder upsampling, the sampling points are restored according to the semantic homogeneity guidance, and the point-level and channel-level semantic features are fused using the bidirectional self-attention mechanism; S4. Experimental verification.

2. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 1, characterized in that: S1 includes the following steps: S11, dynamic sampling and grouping: Input point cloud P = {p i |i=1,2,…,n}, the whole set contains n points, where p={x,y,z,r}, x, y, z are the three-dimensional coordinates of the point, and r is the reflectivity of the point; S12, density perception and feature distance: First, the K nearest neighbor algorithm is used to initialize the neighborhood range of each point, select the k nearest points, and calculate the average distance from each point to the center point, as shown in formula (1): Among them, ρ(p i ) represents the neighborhood range of the i-th point, p i represents the i-th point, p j is a neighborhood point in the range, j∈[1,k] and j is an integer; g i is the average distance; therefore, there is a Gaussian kernel density u i : Where σ is the bandwidth, which can control the local range of the Gaussian function. The set U = {u1,u2,...,u n } is the point cloud density of each point, the average distance is taken as the ball query radius of the center point, and the number of points within the ball query radius is counted m i ; The importance of a point is calculated as follows: Among them, η i is the dynamic coefficient, which indicates the importance of points closer to the center point. Combining equations (2) and (3), point p i The Gaussian kernel density is: s i =u i *or i (4) Set S = {s1,s2,...,s n } is the point cloud density for each point; according to the similarity between the point cloud densities, for each point p i Calculate the semantic feature distance v i , find a density set α for each point i , satisfying α i ={s j |s j ≥s i, s j ∈S}, s j It means to find a point greater than p i The corresponding s i Point cloud, then: Among them, v i It represents the minimum semantic feature distance between the point and any other point with high local density; the distance index of the point with the highest local density is the maximum semantic feature distance between the point and any other point; for each point p i Get a comprehensive score value κ i : k i =s i *v i (6) The score k i The higher it is, the greater the neighborhood density of this point is, the richer the semantic information of this point is, and the more representative it is, which increases the probability of it being retained during the sampling process; S13, local feature descriptor: Through dynamic sampling and grouping, the neighborhood range of each center point is obtained. The sampling points are represented by the set Q = {q1, q2, ..., q t }, the neighborhood range relationship corresponding to each point is represented by R, then Among them, r ij Indicates the jth point in the i-th group, and has the corresponding group feature Where C is the number of feature channels, G is the representation of group features, and f is the representation of features; A parameter δ is introduced to explicitly indicate the importance of each neighborhood point, ξ i represents a local feature descriptor, averaging neighborhood features under the guidance of importance: Among them, θ j It is the rth ij The learnable attention parameter matrix, ξ i The aggregated features can describe local semantic information.

3. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 2, characterized in that: In S13, in order to extract fine-grained features, we propose A with the same dimension as the A matrix in GAM * Matrix, the rest of the mathematical expressions remain unchanged: A * =MLP(concat(ξ,A)) (8) Concatenate ξ and A matrix and use a layer of MLP to adjust them to the same dimension as ξ, outputting feature f c : Among them, λ is the balancing weight, · represents element-wise multiplication, and ψ represents the use of PointNet++’s local feature extractor to extract deep aggregation features; Obtained through maximum pooling At this time, use N c represents the sampling point, and its feature dimension is represented by d; k is the number of neighbor nodes, f d is the feature after maximum pooling.

4. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 3, characterized in that: In step S2, "enhancement of relative position information" establishes the connection between deep points and original input features through the cross-attention mechanism, specifically: According to the deep sampling point N c Get the Q matrix, get the K matrix and V matrix through the original points containing rich position information, W Q , W K , W V The weight parameter matrices corresponding to the Q matrix, K matrix, and V matrix are the parameter matrices that need to be trained, X is the feature matrix, and W is the coefficient of the matrix to be solved; Q=N c W Q K=XW K V=XW V (10) A * With QK T The feature maps of are processed with softmax normalization to obtain the matrix M: d is the feature dimension, QK T is transposed into matrix multiplication, calculating the vector inner product Multiply the new feature map with V, concatenate the result with the deep point set features, and aggregate the features through the graph convolutional network and maximum pooling: Among them, maxpool is the maximum pooling method, N c Represents the sampling point, and the obtained output L participates in the subsequent first-layer decoder for upsampling.

5. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 4, characterized in that: In S3, "using a bidirectional self-attention mechanism to fuse point-level and channel-level semantic features" is implemented through point-level Transformers and channel-level Transformers, specifically: S31, point-level Transformer First, f d Mapped to two different spaces, we get the point-level query matrix Q pl With the point-level primary key matrix K pl , Q pl With K pl They are the point-level query matrix and the point-level primary key matrix; Q pl =W q f d K pl =W k f d (13) W q With W k is a trainable parameter matrix; Using Q pl With K pl The feature map M can be calculated p : In formula (14), d is the feature dimension, Represents the transpose of a matrix; Among them, P E is the position encoding information of Point Transformer; in order to fully integrate local information, select As V in self-attention pl , V pl represents the value vector, which forms the attention mechanism with the K matrix and the Q matrix; f c Covering the d-dimensional features contained in the k different neighborhood points corresponding to each center point, the features of each center point are horizontally spliced ​​and vertically arranged k times. This process is represented by reshape, and we can get a Finally, after the maximum pooling, we get In the above expression, N c In k*kd, kd represents the product of k and d, N c represents the number of sampling points, V pl =maxpool(reshape(f c )) (15) The elements in the attention map are regarded as the weights of the corresponding neighborhood feature set, and the output neighborhood feature set is obtained by calculating the weighted sum of the input set; all points including sampling points and neighborhood points are used to improve the feature learning ability; thus the output of the PLT model is: in, Represents the output features of the point-level Transformer, M p Represents the result of the operation between the key matrix and the query matrix; S32, Channel-level Transformer For a given f d , the context in each channel is simplified by a shared MLP operating on the channel vectors while fully preserving the information of each channel: Q cl =MLP q (f d ) K cl =MLP k (f d ) V cl =MLP v (f d ) (17) Among them, MLP q (·), MLP k (·) and MLP v (·) is the shared MLP layer, and the obtained Q cl , K cl With V cl The Q matrix, K matrix and V matrix corresponding to the self-attention mechanism; according to the K matrix and the Q matrix, the relative relationship feature map is obtained : represents the transpose of the bond matrix; right Perform the maximum pooling operation, take the maximum value of each feature, and get vector; by continuous splicing, Repeat d times to restore the matrix size, becoming Compare this result with Subtract and add to V cl Multiply to get Select the maximum similarity along the channel direction, and obtain the similarity measure by calculating the difference; add softmax to normalize the value. Used as a weight matrix for refinement, each channel can give higher weights to other different channels, thereby avoiding the aggregation of redundant information and fusing the results of the two submodules to obtain the output f t : Where LBR is a combination of linear layer, BatchNorm and ReLU layer; f t It is the output result of DTB, and finally MLP is used for point cloud segmentation task.

6. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 5, characterized in that: The MLP includes a fully connected layer, a batch normalization layer, and a ReLU activation function connected sequentially, and the number of fully connected layers is two.

7. The point cloud semantic segmentation method based on density perception and feature enhancement according to claim 6, characterized in that: In S4, "experimental verification" specifically includes: S41. Determine the data set and indicators: The experiment uses the overall point-by-point accuracy OA, the average category accuracy mAcc and the average category intersection over union mIoU to analyze and evaluate the model. The formula is as follows: Among them, k′ represents the number of categories, p ij Indicates the number of correctly predicted positive samples, p ji Indicates the number of positive samples predicted incorrectly, p ii Indicates the number of negative samples with incorrect predictions; S42. Result analysis and visualization: The three evaluation indicators of OA, mAcc and mIou reached more than 92.0%, more than 79.0% and more than 73.0% respectively.

8. A point cloud semantic segmentation model based on density perception and feature enhancement, used to implement the method according to any one of claims 1 to 7, characterized in that: It includes point cloud input module, downsampling module, cross attention module, position feature enhancement module, upsampling module and point cloud output module. Point cloud input module, used to input three-dimensional point cloud; A downsampling module, used to downsample the input 3D point cloud; The cross attention module is connected to the input of the downsampling module and the input of the upsampling module, corresponding to the downsampling operation and the upsampling operation, which are used to reduce the number of 3D point clouds and restore semantic point clouds after segmentation respectively; The position feature enhancement module is connected to the output of the downsampling module and the input of the upsampling module, and is located between the downsampling operation and the upsampling operation; The point cloud output module is connected to the output end of the upsampling module and is used to output semantic point clouds.

9. The point cloud semantic segmentation model based on density perception and feature enhancement according to claim 8, characterized in that: It also includes a trigger threshold judgment module, which is connected to the downsampling module and dynamically adjusts the density of the three-dimensional point cloud through the trigger threshold to cooperate with the operation of the downsampling module.

Images