Block attention point cloud segmentation method based on curve characteristics

Abstract

The invention provides a block attention point cloud segmentation method based on curve features, and relates to the technical field of point cloud segmentation, and the method comprises the steps: obtaining the point cloud coordinate data of a to-be-segmented point cloud; local neighbor features of all points in the point cloud coordinate data are extracted and aggregated based on sampling grouping; performing point-by-point operation on the to-be-segmented point cloud based on an offset attention mechanism, and calculating an offset difference between a self-attention feature and a neighborhood embedded output feature; curve connection is carried out on the point cloud coordinate data according to a certain rule, and obtained curve features and curve point features forming a curve are fused; performing global context information extraction on the curve output features and the offset attention output features based on a block attention mechanism; pooling the point features to obtain pooling features; performing fusion processing on the pooling features and the point features to obtain global features; and performing segmentation processing by using the global features to obtain a segmentation result of the to-be-segmented point cloud. According to the scheme, the point cloud segmentation precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud segmentation, and in particular to a block attention point cloud segmentation method based on curve features. Background Art

[0002] With the widespread application of high-performance hardware equipment and the rapid development of intelligent technology, 3D point cloud registration technology has played a key role in many fields such as face recognition, unmanned driving, and medical imaging, and has been widely used in scientific research and engineering projects. For example, in the indoor object point cloud registration task, point cloud segmentation is the core of separating the indoor foreground point cloud from the background point cloud. However, during the operation, there are many types of objects in the indoor 3D scene and they are easily confused with the background environment point cloud. The point cloud data has the characteristics of discrete distribution, unstructured and disordered, which creates obstacles to the segmentation task.

[0003] In order to solve the above problems, traditional point cloud segmentation methods usually convert point cloud data into other forms of data, such as voxels or images, and then preprocess and align the data through machine learning. However, this method requires more computing resources and time. In order to solve this problem, technicians in this field began to consider processing the original point cloud data, such as directly mapping the coordinates to a high-dimensional space through a PCT (Point Cloud Transformer) network, and introducing a Transformer to retain the permutation invariance of the point cloud while processing the point sequence, which has advanced performance for processing unstructured, disordered, and irregularly spaced point cloud data.

[0004] However, although the above method is an effective global feature extraction network, it does not make sufficient use of local features, especially in indoor scenes, where local domain information is crucial for distinguishing objects of different shapes and understanding scene structures. Existing solutions ignore this information, resulting in poor point cloud segmentation accuracy when performing segmentation tasks in indoor environments where objects have different shapes and the boundaries between foreground objects and background point clouds are not obvious. Summary of the invention

[0005] In view of this, in order to address the above shortcomings, it is necessary to propose a block attention point cloud segmentation method based on curve features to improve the accuracy of point cloud segmentation.

[0006] The present invention provides a block attention point cloud segmentation method based on curve features, comprising:

[0007] Obtain the point cloud coordinate data of the point cloud to be segmented;

[0008] Extracting and aggregating local neighbor features of each point in the point cloud coordinate data based on sampling grouping to obtain neighborhood embedding output features;

[0009] Based on the offset attention mechanism, the point cloud to be segmented is operated point by point, and the offset difference between the self-attention feature and the neighborhood embedding output feature is calculated to obtain the offset attention output feature;

[0010] Connecting the point cloud coordinate data by curves according to certain rules, and fusing the obtained curve features with the curve point features constituting the curve to obtain curve output features;

[0011] Extracting global context information from the curve output feature and the offset attention output feature based on a block attention mechanism to obtain a point feature;

[0012] Performing pooling processing on the point features to obtain pooled features;

[0013] The pooling feature and the point feature are fused to obtain a global feature;

[0014] The global features are used to perform segmentation processing to obtain a segmentation result of the point cloud to be segmented.

[0015] Preferably, extracting and aggregating local neighbor features of each point in the point cloud coordinate data based on sampling grouping includes:

[0016] Down-sampling the point cloud coordinate data using a farthest sampling algorithm to obtain sampling features;

[0017] For each sampling point in the point cloud to be segmented, calculate the feature difference between the current sampling point and its k nearest neighboring points;

[0018] The feature differences corresponding to each sampling point are spliced ​​to obtain local difference features;

[0019] The local difference feature is concatenated with the sampling feature to obtain a difference fusion feature; wherein the sampling feature is repeated k times before concatenation to match the local difference feature dimension;

[0020] The difference fusion features are subjected to feature extraction and transformation by two LBR networks, and the neighborhood embedding output features are output through a maximum pooling operation; wherein the LBR network is composed of a linear layer, a batch normalization layer, and a ReLU activation function layer.

[0021] Preferably, performing point-by-point operations on the point cloud to be segmented based on the offset attention mechanism to calculate the offset difference between the self-attention feature and the neighborhood embedding output feature includes:

[0022] Calculate the attention weight matrix of each attention layer;

[0023] Based on the neighborhood embedding output features, calculating the layer output features of each attention layer;

[0024] The layer output features of each attention layer are fused to obtain the offset attention output features.

[0025] Preferably,

[0026] When calculating the attention weight matrix of each attention layer, the following calculation formula is used to calculate each attention weight in the attention weight matrix:

[0027]

[0028] where α i,j Used to represent the attention weight, To characterize the The attention weights obtained after scaling, is the normalized attention score obtained by calculating the product of the query matrix Q and the transpose of the key matrix K;

[0029] and / or,

[0030] When calculating the layer output features of each attention layer based on the neighborhood embedding output features, the layer output features are calculated using the following calculation formula:

[0031]

[0032] in, Represents the output of the nth attention layer, Characterize the features of the nth attention layer input, when n = 1, Embed the output features of the neighborhood, F sa is the self-attention feature, A n is the attention weight matrix of the nth attention layer, and LBR is a network consisting of a linear layer, a batch normalization layer, and a ReLU activation function layer;

[0033] and / or,

[0034] The fusion of the layer output features of each attention layer includes using the following calculation formula:

[0035]

[0036] Among them, F concat represents the offset attention output feature, N represents the total number of attention layers, and W represents the weight parameter of the linear layer.

[0037] Preferably, the step of connecting the point cloud coordinate data by curves according to certain rules and fusing the obtained curve features with the curve point features constituting the curve comprises:

[0038] Constructing an undirected graph G=(F, E) based on the point cloud coordinate data; wherein the point set F is the discrete point features in the point cloud, and the edge set E is the connection relationship between the point and the adjacent points;

[0039] According to the defined random walk strategy, a random walk is performed on the undirected graph G to form a curve c; wherein the curve c is defined as a point feature sequence;

[0040] Applying attention pooling operation to grouped curves C respectively to learn inter-curve feature vectors and intra-curve feature vectors; wherein the inter-curve feature vectors are used to characterize the mutual relationship between different curves, and the intra-curve feature vectors are used to characterize the internal relationship within each curve; wherein the grouped curves are a set of curves c obtained by applying a random walk strategy to different starting points;

[0041] The curve point features, the inter-curve feature vectors and the intra-curve feature vectors are fused to obtain the curve output features.

[0042] Preferably, the random walk on the undirected graph G according to the defined random walk strategy forms a curve c, including:

[0043] The point set F is grouped by iteratively executing the following calculation formula l times:

[0044] s i+1 =π(s i ),0≤i∈Z + ≤l

[0045] The curve c is defined as c = {s1,…,s l |s∈F},s i is the point feature of the i-th point constituting the curve, and π(·) is the random walk strategy, which is specifically obtained by the following calculation formula:

[0046]

[0047] in, is the state descriptor of the K nearest neighbor algorithm, consisting of point features s i and the curve descriptor r i Together, N s Used to represent the set of neighboring points related to the current state s.

[0048] Preferably, the fusing the curve point features, the inter-curve feature vectors and the intra-curve feature vectors comprises:

[0049] Input the curve point feature, the inter-curve feature vector and the intra-curve feature vector into a first multi-layer perceptron, a second multi-layer perceptron and a third multi-layer perceptron respectively, and output a first output feature, a second output feature and a third output feature respectively;

[0050] Inputting the second output feature into a fourth multi-layer perceptron to obtain a fourth output feature;

[0051] Inputting the third output feature into a fifth multi-layer perceptron to obtain a fifth output feature;

[0052] Fusion the first output feature with the second output feature to obtain a sixth output feature;

[0053] Fusion the sixth output feature with the fourth output feature to obtain a seventh output feature;

[0054] Fusion the first output feature with the third output feature to obtain an eighth output feature;

[0055] Merging the eighth output feature with the fifth output feature to obtain a ninth output feature;

[0056] The seventh output feature and the ninth output feature are fused and output through a sixth multi-layer perceptron to obtain the curve output feature.

[0057] Preferably, the extracting of global context information from the curve output feature and the offset attention output feature based on the block attention mechanism includes:

[0058] The point cloud to be segmented is divided into a set of M instance bases {S1, S2, ..., S M};

[0059] The combined features in each instance basis are fused to obtain an estimated basis; wherein the combined features are composed of the curve output features and the offset attention output features;

[0060] Re-estimating the combined feature using the estimation basis to obtain a re-estimated feature;

[0061] The re-estimated feature and the combined feature are subjected to difference processing, and the point feature is outputted to obtain the point feature.

[0062] Preferably, fusing the combined features in each instance basis includes calculating the estimated basis using the following calculation formula:

[0063]

[0064] Among them, b mis the instance-based feature obtained by fusing the combined features in the mth instance-based, f i For point p i The combined characteristics of i is the normalization factor, B is the estimation basis, and the transformation function is a multi-layer perceptron containing a linear layer and a ReLU activation function;

[0065] The re-estimating the combined feature by using the estimation basis includes calculating the re-estimated feature by using the following calculation formula group:

[0066]

[0067] Among them, A is used to represent the attention graph network structure constructed by the estimated basis, Q is used to represent the important features that A should focus on, D′ represents the dimension of B, and N represents the total number of points in the point cloud to be segmented;

[0068] The performing difference processing on the re-estimated feature and the combined feature includes calculating the point feature using the following calculation formula:

[0069]

[0070] Among them, F output Point features used to characterize the output, For the standard characterization of the combined features, φ(·) is a multilayer perceptron consisting of a linear layer and a ReLU layer.

[0071] Preferably, the performing pooling processing on the point features includes:

[0072] Performing maximum pooling processing on the point features to obtain maximum pooling features;

[0073] Performing average pooling processing on the point features to obtain average pooling features;

[0074] The maximum pooling feature and the average pooling feature are fused to obtain the pooling feature.

[0075] It can be seen from the above technical scheme that in the block attention point cloud segmentation method based on curve features provided by the present invention, the point cloud coordinate data of the point cloud to be segmented is first obtained, and then the local domain features of each point in the point cloud coordinate data are extracted and aggregated based on sampling grouping to obtain the neighborhood embedding output features. In this way, the importance of local domain information in distinguishing objects of different shapes and understanding the scene structure is fully considered, thereby greatly improving the point cloud segmentation accuracy; further, the offset difference between the self-attention feature and the neighborhood embedding output feature is calculated through the offset attention mechanism to obtain the offset attention output feature, so that the influence of the surrounding environment nodes on the result when segmenting the object point cloud can be reduced by sharpening the attention weight, thereby improving the accuracy of segmentation; then the point cloud data is subjected to The curves are connected, and the obtained curve features are fused with the curve point features that constitute the curve to obtain the curve output features. Compared with the traditional local feature aggregation method, the curve feature extraction and aggregation can enhance the geometric information representation of the point cloud and capture the longer-distance geometric relationship in the point cloud, thereby extracting and aggregating more accurate feature information in the point cloud segmentation; further, the global context information is extracted based on the block attention mechanism to obtain point features, so that long-distance dependencies can be captured, and the global context information of the point cloud can be obtained by measuring the correlation between each feature, so as to dynamically pay attention to and learn important features; finally, the pooled features are obtained by pooling the point features, and the segmentation process is performed after fusion with the pooled features to obtain the segmentation result of the point cloud to be segmented. It can be seen that this scheme can greatly improve the accuracy of point cloud segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 An architecture diagram of a block attention point cloud segmentation based on curve features provided in an embodiment of the present invention.

[0077] Figure 2 A flowchart of a block attention point cloud segmentation method based on curve features provided in an embodiment of the present invention.

[0078] Figure 3 Figure 2 shows some experimental results on the ShapeNet dataset. DETAILED DESCRIPTION

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0080] like Figure 1 Shown is the architecture of block attention point cloud segmentation based on curve features provided by the present invention. Figure 1In , the input layer is an input embedding module, which is used to input the coordinate data of the point cloud to be segmented, and to extract and aggregate the local neighborhood features in the coordinate data of the point cloud to be segmented. The attention layer is a module based on the offset attention mechanism, which is used to perform point-by-point operations on the point cloud to be segmented, and calculate the offset difference between the self-attention feature and the neighborhood embedding output feature. The CIC module is a curve intervention convolution module, which is used to extract and aggregate curve features. The PAT layer is a block attention module, which is used to extract global context information from the curve output features and the offset attention output features. The horizontal output is the point cloud segmentation output, and the vertical output is the point cloud classification output. In the horizontal output, the global features are then decoded through the LBRD layer, the LBR layer, and the linear layer in sequence to output the point cloud segmentation result; in the vertical output, the global features are then decoded through two LBR layers and the linear layer in sequence to output the point cloud classification result. The LBR layer is a feedforward neural network composed of a linear layer L (N in the figure represents a linear layer), a batch normalization layer BN (BatchNormalization), and a ReLU activation function layer. The LBRD layer performs a Dropout operation after the feedforward neural network of the LBR layer. As shown Figure 2 As shown, the specific block attention point cloud segmentation method based on curve features may include the following steps:

[0081] Step 1: Obtain the point cloud coordinate data of the point cloud to be segmented;

[0082] Step 2: Extract and aggregate the local neighbor features of each point in the point cloud coordinate data based on sampling grouping to obtain the neighborhood embedding output features;

[0083] Step 3: Based on the offset attention mechanism, perform point-by-point operations on the segmented point cloud, calculate the offset difference between the self-attention feature and the neighborhood embedding output feature, and obtain the offset attention output feature;

[0084] Step 4: Connect the point cloud coordinate data into curves according to certain rules, and fuse the obtained curve features with the curve point features that constitute the curve to obtain the curve output features;

[0085] Step 5: Extract global context information from the curve output features and offset attention output features based on the block attention mechanism to obtain point features;

[0086] Step 6: Pool the point features to obtain pooled features;

[0087] Step 7: Fuse the pooled features and point features to obtain global features;

[0088] In this embodiment, when performing a segmentation operation on a point cloud to be segmented, the point cloud coordinate data of the point cloud to be segmented is first obtained, and then the local domain features of each point in the point cloud coordinate data are extracted and aggregated based on the sampling grouping to obtain a neighborhood embedding output feature. This fully considers the importance of local domain information in distinguishing objects of different shapes and understanding scene structures, thereby greatly improving the accuracy of point cloud segmentation; further, the offset difference between the self-attention feature and the neighborhood embedding output feature is calculated through the offset attention mechanism to obtain the offset attention output feature. This can reduce the influence of surrounding environment nodes on the results when segmenting the object point cloud by sharpening the attention weight, thereby improving the accuracy of segmentation; then the point cloud data is connected by curves, and The obtained curve features and the curve point features that constitute the curve are fused to obtain the curve output features. Compared with the traditional local feature aggregation method, the curve feature extraction and aggregation can enhance the geometric information representation of the point cloud and capture the longer-range geometric relationship in the point cloud, thereby extracting and aggregating more accurate feature information in the point cloud segmentation; further, the global context information is extracted based on the block attention mechanism to obtain point features, which can capture long-range dependencies, and measure the correlation between each feature to obtain the global context information of the point cloud, so as to dynamically pay attention to and learn important features; finally, the pooled features are obtained by pooling the point features, and the segmentation process is performed after fusion with the pooled features to obtain the segmentation result of the point cloud to be segmented. It can be seen that this scheme can greatly improve the accuracy of point cloud segmentation.

[0089] For step 1, obtain the point cloud coordinate data of the point cloud to be segmented;

[0090] The data of the point cloud to be segmented obtained in this step may include the three-dimensional coordinate data of the point cloud. For example, the point cloud coordinate data may be obtained by laser ranging method, three-dimensional scanner method, structured light method, depth camera method, etc.

[0091] For step 2, local neighbor features of each point in the point cloud coordinate data are extracted and aggregated based on sampling grouping to obtain neighborhood embedding output features;

[0092] Since the existing global feature extraction network based on the point embedding layer ignores the equally important local neighborhood information in point cloud learning, this step is to extract and aggregate the local neighborhood features to enhance the local feature extraction capability of the network and fully consider the impact of local neighborhood features on point cloud segmentation. Step 2 is implemented as follows when extracting and aggregating the local neighborhood features of each point in the point cloud coordinate data based on sampling grouping:

[0093] The farthest sampling algorithm is used to downsample the point cloud coordinate data to obtain sampling features;

[0094] For each sampling point in the point cloud to be segmented, calculate the feature difference between the current sampling point and its k nearest neighboring points;

[0095] The feature differences corresponding to each sampling point are spliced ​​to obtain local difference features;

[0096] The local difference feature is concatenated with the sampling feature to obtain the difference fusion feature; wherein the sampling feature is repeated k times before concatenation to match the local difference feature dimension;

[0097] The difference fusion features are passed through two LBR networks for feature extraction and transformation, and the output neighborhood is embedded into the output features through the maximum pooling operation; among them, the LBR network consists of a linear layer, a batch normalization layer and a ReLU activation function layer.

[0098] In this embodiment, it is considered to embed a neighbor embedding module in the input layer. The neighbor embedding module can be composed of three LBR layers and two SG layers, where the SG layer is a sampling grouping layer, which is used to use Euclidean distance to aggregate the local neighbor features of each point from the K nearest neighbor algorithm KNN search group during point cloud sampling, that is, the point cloud coordinates and the neighbor point seats extracted by KNN are input, and the sampled point cloud and its corresponding aggregated features are output.

[0099] Specifically, we can first use the farthest sampling algorithm to downsample the point cloud P to P s , then for P s For each sampling point p in, let k p is the nearest neighbor point set of point p, then the neighborhood embedding output feature F can be calculated by the following formula: s :

[0100]

[0101] Where F(p) is the input feature of point p, that is, the point cloud coordinate data of point p, and F s (p) is the output feature of the sampling point p, MP is the maximum pooling operator, RP(x,k) represents the operator used to repeat the vector x, k times to form a matrix, and LBR is the Linear-BatchNorm-ReLU network. In this embodiment, the relative feature information between points in the local neighborhood can be captured by calculating the feature difference between the sampling point and the neighboring points. By aggregating the sampling features and the local feature differences, the local features are fully integrated into the sampling features, thereby realizing the consideration of local features in point cloud segmentation.

[0102] For step 3, based on the offset attention mechanism, the segmented point cloud is operated point by point, the offset difference between the self-attention feature and the neighborhood embedding output feature is calculated, and the offset attention output feature is obtained;

[0103] In this step, considering that the eigenvalues ​​and eigenvectors of the Laplacian matrix can be used for spectral clustering of graphs, this graph-based data point segmentation method can identify tightly connected node groups in the graph by analyzing the eigenvectors of the Laplacian matrix, which is suitable for segmenting disordered and unstructured point cloud data features. Therefore, it is considered to calculate the offset difference between the self-attention feature and the input neighborhood embedding output feature through point-by-point operations based on the offset attention mechanism layer. Specifically, step 3 can be implemented as follows:

[0104] Calculate the attention weight matrix of each attention layer;

[0105] Based on the neighborhood embedding output features, calculate the layer output features of each attention layer;

[0106] The layer output features of each attention layer are fused to obtain the offset attention output features.

[0107] In this embodiment, when calculating the attention weight matrix of each attention layer, the following calculation formula can be used to calculate each attention weight in the attention weight matrix:

[0108]

[0109] where α i,j Used to represent the attention weight, To characterize the The attention weights obtained after scaling, It is the normalized attention score obtained by calculating the product of the query matrix Q and the transpose of the key matrix K. Compared with the traditional normalization method, this method can enhance the discrimination of the attention weights and thus improve the performance of the model.

[0110] When calculating the layer output features of each attention layer based on the neighborhood embedding output features, the layer output features can be calculated using the following formula:

[0111]

[0112] in, Represents the output of the nth attention layer, Analogous to the discrete Laplace operator, Characterize the features of the nth attention layer input, when n = 1, is the neighborhood embedding output feature, F sa is the self-attention feature, A n is the attention weight matrix of the nth attention layer. LBR is a network composed of a linear layer, a batch normalization layer, and a ReLU activation function layer. The offset attention module in this scheme consists of 4 attention layers.

[0113] The following calculation formula can be used to fuse the output features of each attention layer:

[0114]

[0115] Among them, F concat represents the offset attention output feature, N represents the total number of attention layers, and W represents the weight parameter of the linear layer.

[0116] In this embodiment, different methods are used in two dimensions to normalize the attention map in the offset attention mechanism. The softmax operator is used for normalization in the first dimension, and the norm normalization in the offset attention mechanism is different from that in the ordinary attention mechanism in the second dimension. Compared with the traditional Transformer, the weight of the first dimension is scaled and only the softmax is used for normalization in the second dimension. This can more effectively sharpen the attention weight, which helps to reduce the influence of surrounding environment nodes on the results when segmenting the object point cloud, thereby improving the accuracy of segmentation.

[0117] For step 4, the point cloud coordinate data is connected by curves according to certain rules, and the obtained curve features are fused with the curve point features constituting the curve to obtain the curve output features;

[0118] In this step, the CIC module enhances the geometric information representation of the point cloud by aggregating the curve features in the point cloud. Compared with the traditional local feature aggregation method, curve aggregation can capture the geometric relationship of the point cloud at a longer distance. The curve aggregation operation of the CIC module can mainly include two processes, namely, firstly connecting the points in the point cloud into curves according to certain rules, and then fusing these curve features with the point features to enhance the geometric representation ability of the point cloud. Specifically, step 4 can be implemented as follows:

[0119] Construct an undirected graph G = (F, E) based on the point cloud coordinate data; where the point set F is the discrete point features in the point cloud, and the edge set E is the connection relationship between the point and the adjacent points;

[0120] According to the defined random walk strategy, a random walk is performed on the undirected graph G to form a curve c; wherein the curve c is defined as a point feature sequence;

[0121] Attention pooling operations are applied to the grouped curves C to learn the inter-curve feature vectors and intra-curve feature vectors; the inter-curve feature vectors are used to characterize the mutual relationship between different curves, and the intra-curve feature vectors are used to characterize the internal relationship within each curve; the grouped curve is a set of curves c obtained by applying a random walk strategy to different starting points;

[0122] The curve point features, inter-curve feature vectors and intra-curve feature vectors are fused to obtain the curve output features.

[0123] Among them, a curve c is formed by random walking on the undirected graph G according to the defined random walk strategy, including:

[0124] The point set F is grouped by iteratively executing the following calculation formula l times:

[0125] s i+1 =π(s i ),0≤i∈Z + ≤l

[0126] The curve c is defined as c = {s1,…,s l |s∈F},s i is the point feature of the i-th point constituting the curve, π(·) is the random walk strategy, which is specifically obtained by the following calculation formula:

[0127]

[0128] in, is the state descriptor of the K nearest neighbor algorithm, consisting of point features s i and the curve descriptor r i Together, N s Used to represent the set of neighboring points related to the current state s.

[0129] In this embodiment, the purpose of the CIC module is to capture the point cloud structure, distinguish scenes with complex geometric shapes and object point clouds, and then refine and compress feature representation, remove redundant information, and improve feature discrimination. The curve represents the connection sequence of points in the point cloud, which is specifically defined as: given a point cloud P, a feature F, and an undirected graph G = (F, E), where the connectivity E is calculated on P by the K nearest neighbor algorithm KNN. In the feature space, a curve of length l is generated as a point feature sequence in F, such that c = {s1,…,s l |s∈F}. In order to group the curves, consider the walking strategy π(·) defined on the undirected graph G. The walking strategy starts from the starting point s1 and after l steps, finally obtains an l-dimensional feature vector. The walking strategy π(·) determines the way the curve is generated, that is, it restricts the walking direction, length, etc., to ensure that the generated curve can better describe the geometric shape of the point cloud.

[0130] Specifically, a sigmoid gated MLP can be used to learn the selection score of each point feature in F, with the n points with the highest scores as the starting points. After constructing the starting point set, walk W from a starting point in the starting point set and walk exactly l steps. The points passed by W are grouped to form a curve c. By executing the walking strategy with different starting points, the grouped curve C = {c1,…,c n}. The intermediate state s of a given curve i , when the curves are grouped in the feature space, s i Equal to f in value i After walking i steps, the next step is to find a suitable curve walking strategy π(·). For the predefined π(·), the curve c={s1,…,s l |s∈F} can iteratively execute s i+1 =π(s i ),0≤i∈Z + ≤ll times to finally group.

[0131] In order to achieve high-quality π(·), a learnable π(·) is proposed. For state s, the state descriptor h s ∈R 2||s|| Use MLP to determine the next state descriptor, which is composed of point features s i and the curve descriptor r i Together, they provide a complete description of the state. Input α to softmax, assign 0 to 1 based on the domain point, and the maximum point is the output of π(·), where π(·) and α can be expressed as:

[0132]

[0133] In the formula, is the descriptor of KNN. In the forward conduction, π(s) determines the next state of α. In order to avoid the occurrence of loops in the curve and thus the redundant and repeated scale features, it is necessary to consider the dynamic encoding of the state descriptor of the current curve progress. Keep the curve descriptor r i ∈R ||s|| It encodes the prefix of the curve in step i. Key point s i The state descriptor of each neighboring point For i and r i The prefix r of the curve can be updated according to the momentum paradigm so that the momentum coefficient β becomes a dynamic variable for better encoding. The specific method can be:

[0134]

[0135] After forming the curve, we further consider extracting curve features and fusing curve features. Specifically, the feature channel output by the Top-k structure can be defined as C, the number of points can be defined as P, and the basic attention pooling operator is recorded as AP. In AP, the point features s constituting the curve are scaled as input features in a self-attention manner and summed along the R dimension. Given a grouped curve C = {c1,…,c n}, in order to aggregate the features in the curves and consider the interrelationships between curves and the internal relationships within each curve. First, the feature vector f between curves is learned by applying AP along different axes on C. intra and the eigenvector f inside the curve inter , then fuse the feature vectors between curves, the feature vectors within the curves and the features of the curve points, and finally output the curve output features. Specifically, when fusing the feature vectors between curves, the feature vectors within the curves and the features of the curve points, it can be achieved in the following ways:

[0136] Inputting the curve point feature, the inter-curve feature vector and the intra-curve feature vector into the first multi-layer perceptron, the second multi-layer perceptron and the third multi-layer perceptron respectively, and outputting the first output feature, the second output feature and the third output feature respectively;

[0137] Input the second output feature into a fourth multi-layer perceptron to obtain a fourth output feature;

[0138] Input the third output feature into the fifth multi-layer perceptron to obtain a fifth output feature;

[0139] The first output feature is combined with the second output feature to obtain a sixth output feature;

[0140] The sixth output feature is combined with the fourth output feature to obtain a seventh output feature;

[0141] The first output feature is combined with the third output feature to obtain an eighth output feature;

[0142] The eighth output feature is combined with the fifth output feature to obtain a ninth output feature;

[0143] The seventh output feature is fused with the ninth output feature and outputted through the sixth multi-layer perceptron to obtain a curve output feature.

[0144] In this embodiment, the point feature F and the feature vector f between the curves intra and the eigenvector f inside the curve interThe input is fed into three separate MLPs together to reduce the feature size. Matrix multiplication is applied to learn the mapping of each curve point separately. The mapping is converted into a score using a softmax function. In another branch, two additional MLPs are used to further transform the simplified f intra and f inter , and then the two additional MLPs are fused with the calculated mapping scores by matrix multiplication respectively. The resulting two feature vectors are concatenated along the feature axis and fed into the final MLP.

[0145] Finally, the curve output features of the CIC module and the offset attention output features of the offset attention module are concatenated and input into the subsequent block attention module.

[0146] For step 5, the global context information is extracted from the curve output features and the offset attention output features based on the block attention mechanism to obtain the point features;

[0147] In this step, the aim is to capture the global context information of the point cloud by calculating the correlation between the query vector and the smaller basis, with flexibility in size and gear. Specifically, it includes technical estimation and data re-estimation to ensure the accuracy and robustness of multi-scale feature estimation. Step 5 can be implemented as follows when extracting global context information:

[0148] The point cloud to be segmented is divided into a set of M instance bases {S1, S2, ..., S M};

[0149] The combined features in each instance basis are fused to obtain an estimated basis; wherein the combined features are composed of the curve output features and the offset attention output features;

[0150] Re-estimate the combined features using the estimation basis to obtain the re-estimated features;

[0151] The re-estimated features and the combined features are differentiated and output as point features.

[0152] Among them, the combined features in each instance basis are fused, including calculating the estimated basis using the following calculation formula:

[0153]

[0154] Among them, b m is the instance-based feature obtained by fusing the combined features in the mth instance-based, f i For point p i The combined characteristics of i is the normalization factor, B is the estimation basis, and the transformation function is a multi-layer perceptron containing a linear layer and a ReLU activation function;

[0155] Re - estimate the combined features using the estimated basis, including calculating the re - estimated features using the following set of calculation formulas:

[0156]

[0157] Among them, A is used to represent the attention map network structure constructed by the estimated basis, Q is used to represent the important features that A needs to focus on, D′ represents the dimension of B, and N represents the total number of points in the point cloud to be segmented;

[0158] Perform a difference operation on the re - estimated features and the combined features, including calculating the point features using the following calculation formula:

[0159]

[0160] Among them, F output is used to represent the output point features, is used to standardly represent the combined features, and φ(·) is a multi - layer perceptron composed of a linear layer and a ReLU layer.

[0161] In this embodiment, first, a set of compact bases B ∈ R M×D can be estimated, where M is the number of bases. For each point cloud P in the dataset, it is divided into M blocks (M << N), thus forming M blocks called instance bases, and the set of these instance bases is used to approximately represent the global shape of the entire point cloud. Using the K - means algorithm, the point cloud P is segmented into a set of M instance bases {S1, S2, …, S M}. Aggregate the features of all points in each instance base to obtain the instance base features, and form a set to obtain the estimated basis B.

[0162] After estimating the basis, use the key matrix in the basis replacement formula to obtain the attention map network structure constructed by the geometry of the basis Then use the basis B and the attention map A to re - estimate the input F. Calculate the difference by element - by - element subtraction of the estimated features and the input features, and finally input the difference into the multi - layer perceptron for processing. Through the strategy of residual connection, information can be effectively transmitted to deeper layers of the network.

[0163] Step 6, perform pooling processing on the point features to obtain the pooled features;

[0164] In this step, when performing pooling processing on point features, the point features are firstly subjected to maximum pooling processing to obtain maximum pooling features, and then the point features are subjected to average pooling processing to obtain average pooling features, and finally the maximum pooling features and the average pooling features are fused to obtain pooling features. In this way, through the maximum pooling operation and the average pooling operation, multiple information representations of regional average features and significant features can be extracted from the point-level features, thereby enhancing the network's representation and generalization capabilities for data.

[0165] Step 7: Use global features to perform segmentation processing to obtain the segmentation result of the point cloud to be segmented.

[0166] This step is used to use global features for segmentation processing. That is, at the decoding layer end, a partial label is first predicted for each point, and then the global features obtained after the pooling layer are spliced ​​with the point-by-point features, and the one-hot object category vector is encoded into 64 dimensions, which is spliced ​​with the global features. This multi-scale feature containing curve features is particularly important for distinguishing scene point clouds and object point clouds with complex set shapes. The decoder of the segmentation network includes a feedforward neural network LBR, and then performs Dropout with a drop probability of 0.5. The LBRD and LBR layers are cascaded and output to the linear layer. In the end, the score of each point in the point cloud corresponding to each part of the point cloud will be obtained, and the partial label with the highest score is determined as the partial category of the point in the entire point cloud.

[0167] Among them, the loss function in the learning task adopts the cross entropy loss function.

[0168] Furthermore, after the network construction and training are completed, the performance of the model can be evaluated and analyzed by the mean intersection over Union (mIoU) and mean accuracy (mAcc).

[0169] The effect of this scheme is further explained below in conjunction with specific experiments.

[0170] (1) SG layer determination experiment

[0171] The network trained by this scheme is experimentally verified on the ShapeNet dataset to evaluate its performance on the indoor object part segmentation task. The dataset contains a total of 16,880 3D models, of which 14,006 are used for training and 2,874 are used for testing. It covers 16 different object categories and is labeled with 50 parts, with each model containing at least two parts. All models are downsampled to 2048 points, and the point-by-point part annotations are retained.

[0172] During the training phase, the input data was augmented by applying random translations in the range [-0.2, 0.2] and random anisotropic scaling in the range [0.67, 1.5]. During the testing phase, a multi-scale testing strategy was adopted, with the scale range set between [0.7, 1.4] and a step size of 0.1. For all models, the batch size for training was set to 4, the training epochs were 250, the learning rate was set to 0.001, and the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 was used for training.

[0173] In the experiment, the SG layers with 1 to 3 domain embeddings in the model constructed by this scheme (named CP-PCT model) were trained and tested, and their performance was evaluated on the ShapeNet dataset. As shown in Table 1, different numbers of embedding layers in the sampling grouping layer will affect the performance of the model, and the model with two SG layers performs best. Adding too many SG layers does not necessarily improve the performance of the model, because too many SG layers may make the extracted local features too redundant, interfering with the extraction of key information, thereby reducing the accuracy of the model. In addition, redundant features will increase the computational burden of the network, resulting in an increase in the time and space complexity of the algorithm.

[0174] Table 1

[0175]

[0176] (2) Comparison with existing mainstream segmentation networks

[0177] The network CP-PCT designed by the present invention was compared with the mainstream segmentation networks PointNet++, PCNN, PointConv, PointCNN, PointASNL, KPConv, PAConv, RS-CNN, and PCT on the ShapeNet dataset. The experimental effect was evaluated by two dimensions: mIoU and mAcc. Both mIoU and mAcc are the average results of a total of 16 categories of data. Only some of the data of some categories are listed in the table, as shown in Table 2 and Table 3. The visualization results of some categories of experiments are shown in Figure 3 shown.

[0178] Table 2

[0179]

[0180]

[0181] Table 3

[0182]

[0183] As can be seen from Tables 2 and 3, compared with PointNet++, the CP-PCT proposed in the present invention has achieved higher performance in each category, with mIoU increased by 14.3% and mAc increased by 14.1%. This is because PointNet++ divides the point cloud into different local areas and uses the farthest point sampling to maintain the imbalanced characteristics of the input data, but the network must determine whether the point cloud density reaches the threshold every time it passes through an area. If it does not reach it, it will resample. Therefore, the segmentation effect is not good in complex indoor scenes, that is, when the point cloud density of different objects is greatly different. The network proposed in the present invention structures the input point cloud with uneven density through curve feature operations, so that different point clouds have curve features of different paths, effectively utilizing the sparsity of point cloud data. With the help of the encoding and decoding structure of the block attention mechanism, multi-scale feature learning is performed for point clouds of different shapes, and the detailed features are well captured, so the accuracy in all categories is higher than PointNet++.

[0184] The CP-PCT proposed in this paper is 6% higher than PointConv in the evaluation index mIoU. PointConv uses the learning of weight functions, multi-layer perceptron networks and density functions for given points to perform kernel density estimation, and designs efficient convolution, which can solve the problem of inefficient segmentation of uneven point clouds to a certain extent. This approach enables PointConv to achieve the best segmentation performance on the chair class. However, compared with the proposed method, the CP-PCT algorithm still lacks deep semantic feature learning of point cloud data, such as Figure 3 As shown in the figure, the segmentation accuracy of PointConv in the wing and tail parts is low. The block attention and offset attention network proposed in the present invention fuses low-dimensional spatial geometric features with deep semantic features of the same resolution through the encoding structure, and enhances the learning of the effective features extracted by the encoder through mechanisms such as the LBR layer, which can better mine the deep semantic features.

[0185] Compared to KPConv, CP-PCT improves mIoU by 5.6%. KPConv implements flexible convolution operations by locating kernel points in Euclidean space and applying convolution weights to input points near the kernel points. Therefore, KPConv can efficiently and adaptively learn local geometric structures to extract data features for specific categories, such as bags, and KPConv achieves an average accuracy of 90.94% in this category. However, for easily confused categories, such as rockets, and complex-shaped areas, such as target areas with large surface curvature such as airplanes, cars, and desks, KPConv's higher network complexity ignores local feature similarities. Figure 3 As shown in the figure, the segmentation accuracy of KPConv is not high at the tail of the wing, the tail of the rocket, and the cap.

[0186] From Table 2 and Table 3, we can see that compared with other networks in the table, CP-PCT has higher IoU and Acc values ​​in most categories, especially in indoor object categories such as airplane, rocket and table, its performance is significantly better than other models, such as Figure 3 As shown. This shows that the CP-PCT model has a high accuracy in recognizing these shapes. And the overall average intersection-over-union ratio and average accuracy rate in the partial segmentation task of 16 types of objects are both the highest. This proves that the point cloud segmentation network constructed by CP-PCT has high accuracy and effectiveness in the segmentation task of various objects in indoor scenes in the artificial dataset ShapeNet. This experiment has guiding significance for the partial segmentation of objects in indoor scenes.

[0187] It can be seen that the existing disordered and unstructured indoor scene point cloud segmentation methods have the problem that the object morphological feature information is ignored in the feature extraction process due to the use of conventional three-dimensional convolution, and the object point cloud and the scene point cloud are easily confused. The present invention uses the local feature aggregation of curve features and the point cloud segmentation algorithm CP-PCT based on the block attention mechanism to better solve this problem. First, the CIC module is introduced to introduce the curve shape features in the point cloud into the encoding layer to solve the problem of low feature distinction between the foreground point cloud and the background point cloud. Then the block attention mechanism is used to dynamically pay attention to and learn important features, and effectively extract the high-dimensional features of each indoor object. Finally, considering that the attention mechanism has limited function in capturing global features and may ignore local geometric information, the neighborhood embedding layer strategy is introduced to improve the point embedding layer. The present invention considers more local features and morphological features when paying attention to global features, and solves the problem that there may be overlapping or similar information between the background and the object in the high-dimensional features and the high-dimensional features with different object shapes are insufficient in expression ability.

[0188] The present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute a method in any one of the embodiments in the specification.

[0189] The present specification also provides a computing device, including a memory and a processor, wherein executable codes are stored in the memory, and when the processor executes the executable codes, a method in any embodiment of the present specification is implemented.

[0190] Since the device embodiments provided by the present invention are based on the same inventive concept as the method embodiments of this specification, the specific contents can be found in the description of the method embodiments of this specification and will not be repeated here.

[0191] The modules or units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. The above disclosure is only the preferred embodiment of the present invention, and of course it cannot be used to limit the scope of the rights of the present invention. Those skilled in the art can understand that all or part of the processes of the above embodiment are implemented, and the equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A block attention point cloud segmentation method based on curve features, characterized in that: include: Obtain the point cloud coordinate data of the point cloud to be segmented; Extracting and aggregating local neighbor features of each point in the point cloud coordinate data based on sampling grouping to obtain neighborhood embedding output features; Based on the offset attention mechanism, the point cloud to be segmented is operated point by point, and the offset difference between the self-attention feature and the neighborhood embedding output feature is calculated to obtain the offset attention output feature; Connecting the point cloud coordinate data by curves according to certain rules, and fusing the obtained curve features with the curve point features constituting the curve to obtain curve output features; Extracting global context information from the curve output feature and the offset attention output feature based on a block attention mechanism to obtain a point feature; Performing pooling processing on the point features to obtain pooled features; The pooling feature and the point feature are fused to obtain a global feature; The global features are used to perform segmentation processing to obtain a segmentation result of the point cloud to be segmented.

2. The block attention point cloud segmentation method based on curve features according to claim 1 is characterized in that: The extracting and aggregating the local neighbor features of each point in the point cloud coordinate data based on sampling grouping includes: Down-sampling the point cloud coordinate data using a farthest sampling algorithm to obtain sampling features; For each sampling point in the point cloud to be segmented, calculate the feature difference between the current sampling point and its k nearest neighboring points; The feature differences corresponding to each sampling point are spliced ​​to obtain local difference features; The local difference feature is concatenated with the sampling feature to obtain a difference fusion feature; wherein the sampling feature is repeated k times before concatenation to match the local difference feature dimension; The difference fusion features are subjected to feature extraction and transformation by two LBR networks, and the neighborhood embedding output features are output through a maximum pooling operation; wherein the LBR network is composed of a linear layer, a batch normalization layer, and a ReLU activation function layer.

3. The block attention point cloud segmentation method based on curve features according to claim 1 is characterized in that: The step of performing point-by-point operations on the point cloud to be segmented based on the offset attention mechanism and calculating the offset difference between the self-attention feature and the neighborhood embedding output feature includes: Calculate the attention weight matrix of each attention layer; Based on the neighborhood embedding output features, calculating the layer output features of each attention layer; The layer output features of each attention layer are fused to obtain the offset attention output features.

4. The block attention point cloud segmentation method based on curve features according to claim 3 is characterized in that: When calculating the attention weight matrix of each attention layer, the following calculation formula is used to calculate each attention weight in the attention weight matrix: where α i,j Used to represent the attention weight, To characterize the The attention weights obtained after scaling, is the normalized attention score obtained by calculating the product of the query matrix Q and the transpose of the key matrix K; and / or, When calculating the layer output features of each attention layer based on the neighborhood embedding output features, the layer output features are calculated using the following calculation formula: in, Represents the output of the nth attention layer, Characterize the features of the nth attention layer input, when n = 1, Embed the output features of the neighborhood, F sa is the self-attention feature, A n is the attention weight matrix of the nth attention layer, and LBR is a network consisting of a linear layer, a batch normalization layer, and a ReLU activation function layer; and / or, The fusion of the layer output features of each attention layer includes using the following calculation formula: Among them, F concat represents the offset attention output feature, N represents the total number of attention layers, and W represents the weight parameter of the linear layer.

5. The block attention point cloud segmentation method based on curve features according to claim 1 is characterized in that: The step of connecting the point cloud coordinate data by curves according to certain rules and fusing the obtained curve features with the curve point features constituting the curve includes: Constructing an undirected graph G=(F, E) based on the point cloud coordinate data; wherein the point set F is the discrete point features in the point cloud, and the edge set E is the connection relationship between the point and the adjacent points; According to the defined random walk strategy, a random walk is performed on the undirected graph G to form a curve c; wherein the curve c is defined as a point feature sequence; Applying attention pooling operation to grouped curves C respectively to learn inter-curve feature vectors and intra-curve feature vectors; wherein the inter-curve feature vectors are used to characterize the mutual relationship between different curves, and the intra-curve feature vectors are used to characterize the internal relationship within each curve; wherein the grouped curves are a set of curves c obtained by applying a random walk strategy to different starting points; The curve point features, the inter-curve feature vectors and the intra-curve feature vectors are fused to obtain the curve output features.

6. The block attention point cloud segmentation method based on curve features according to claim 5 is characterized in that: The random walk on the undirected graph G according to the defined random walk strategy forms a curve c, including: The point set F is grouped by iteratively executing the following calculation formula l times: s i+1 =π(s i ),0≤i∈Z + ≤l The curve c is defined as c = {s1,…,s l s∈F},s i is the point feature of the i-th point constituting the curve, and π(·) is the random walk strategy, which is specifically obtained by the following calculation formula: in, is the state descriptor of the K nearest neighbor algorithm, consisting of point features s i and the curve descriptor r i Together, N s Used to represent the set of neighboring points related to the current state s.

7. The block attention point cloud segmentation method based on curve features according to claim 5 is characterized in that: The fusing the curve point features, the inter-curve feature vectors and the intra-curve feature vectors comprises: Input the curve point feature, the inter-curve feature vector and the intra-curve feature vector into a first multi-layer perceptron, a second multi-layer perceptron and a third multi-layer perceptron respectively, and output a first output feature, a second output feature and a third output feature respectively; Inputting the second output feature into a fourth multi-layer perceptron to obtain a fourth output feature; Inputting the third output feature into a fifth multi-layer perceptron to obtain a fifth output feature; Fusion the first output feature with the second output feature to obtain a sixth output feature; Fusion the sixth output feature with the fourth output feature to obtain a seventh output feature; Fusion the first output feature with the third output feature to obtain an eighth output feature; Merging the eighth output feature with the fifth output feature to obtain a ninth output feature; The seventh output feature and the ninth output feature are fused and output through a sixth multi-layer perceptron to obtain the curve output feature.

8. The block attention point cloud segmentation method based on curve features according to claim 1 is characterized in that: The extracting of global context information from the curve output feature and the offset attention output feature based on the block attention mechanism includes: The point cloud to be segmented is divided into a set of M instance bases {S1, S2, ..., S M }; The combined features in each instance basis are fused to obtain an estimated basis; wherein the combined features are composed of the curve output features and the offset attention output features; Re-estimating the combined feature using the estimation basis to obtain a re-estimated feature; The re-estimated feature and the combined feature are subjected to difference processing, and the point feature is outputted to obtain the point feature.

9. The block attention point cloud segmentation method based on curve features according to claim 8 is characterized in that: The fusing of the combined features in each instance basis includes calculating the estimated basis using the following calculation formula: Among them, b m is the instance-based feature obtained by fusing the combined features in the mth instance-based, f i For point p i The combined characteristics of i is the normalization factor, B is the estimation basis, and the transformation function is a multi-layer perceptron containing a linear layer and a ReLU activation function; The re-estimating the combined feature by using the estimation basis includes calculating the re-estimated feature by using the following calculation formula group: Among them, A is used to represent the attention graph network structure constructed by the estimated basis, Q is used to represent the important features that A should focus on, D′ represents the dimension of B, and N represents the total number of points in the point cloud to be segmented; The performing difference processing on the re-estimated feature and the combined feature includes calculating the point feature using the following calculation formula: Among them, F output Point features used to characterize the output, For the standard characterization of the combined features, φ(·) is a multilayer perceptron consisting of a linear layer and a ReLU layer.

10. The block attention point cloud segmentation method based on curve features according to claim 1, characterized in that: The performing pooling processing on the point features comprises: Performing maximum pooling processing on the point features to obtain maximum pooling features; Performing average pooling processing on the point features to obtain average pooling features; The maximum pooling feature and the average pooling feature are fused to obtain the pooling feature.

Images