Point cloud semantic segmentation method based on joint transformer and sparse convolution

Abstract

The application discloses a point cloud semantic segmentation method based on joint Transformer and sparse convolution, first adopts the core attention mechanism in the Transformer to extract global features of original point cloud data point by point, then voxelizes the original point cloud data, extracts local features of the voxelized point cloud data through sparse sub-manifold convolution, and finally fuses the extracted global features and local features to obtain a final semantic segmentation result. The application solves the problem of insufficient local feature extraction of the existing method, and improves the precision and efficiency of indoor scene semantic segmentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of indoor scene point cloud semantic segmentation, and relates to a point cloud semantic segmentation method based on combined Transformer and sparse convolution. BACKGROUND

[0002] With the development of the intelligent era, people's demand for indoor scene spatial information acquisition and updating is also growing. Indoor scene understanding is an important module in the field of artificial intelligence, and is also an important part of scene understanding and analysis in computer vision. Three-dimensional point cloud semantic segmentation is the basis of indoor scene understanding and analysis, and has broad application prospects in indoor robot navigation, target detection and recognition, three-dimensional reconstruction, indoor public place positioning and navigation, etc.

[0003] Indoor scene semantic segmentation refers to distinguishing different categories of objects in a complex indoor scene, assigning different labels to each point according to point coordinates, color and normal information, and marking them as different semantic categories. In recent years, with the improvement of computer hardware performance and the emergence of large-scale data sets, deep learning has gradually become a research hotspot in the field of point cloud segmentation. PointNet successfully applied point-based deep learning to point cloud feature extraction in the early stage. This method uses a T-net network to process the model for rotation invariance, and uses a multi-layer perceptron to extract the global features of the point cloud. However, due to the difficulty of shared weight multi-layer perceptron mapping to fully express the spatial distribution of point clouds, the model lacks the ability to model geometric structures and cannot effectively extract local features. SUMMARY

[0004] The purpose of the present application is to provide a point cloud semantic segmentation method based on combined Transformer and sparse convolution (TSPconv-Net), which solves the problem of insufficient local feature extraction of existing methods and improves the accuracy and efficiency of indoor scene semantic segmentation.

[0005] The technical solution adopted by the present application is a point cloud semantic segmentation method based on combined Transformer and sparse convolution. First, the core attention mechanism in Transformer is used to extract global features of the original point cloud data point by point. Then, the original point cloud data is voxelized, and the voxelized point cloud data is subjected to local feature extraction by sparse sub-manifold convolution. Finally, the extracted global features and local features are fused to obtain the final semantic segmentation result.

[0006] The present application is characterized in that,

[0007] The present application is characterized in that,

[0008] Step 1, Feature Embedding: Input point cloud data, and use a shared neural network consisting of two cascaded LBRs to merge the original location encoding and input features into a coordinate-based encoding module;

[0009] Step 2, Calculate offset attention: Input the embedded features into the attention mechanism module for further feature extraction. Specifically, first, the point cloud is mapped to a low-dimensional feature space through linear transformation. Then, the offset vector between each point and its surrounding points is calculated. The offset vector is used to guide the calculation of attention.

[0010] Step 3, construct the residual module: add the input and output of the offset attention to complete the construction of the residual module;

[0011] Step 4, Global Feature Extraction: First, four residual modules are linearly connected to form a complete attention mechanism module. Then, the output of the attention mechanism is linearly transformed to obtain point cloud features. Finally, max pooling and average pooling operations are performed on the point cloud features. The obtained max pooling feature maps and average pooling feature maps are merged to obtain global features.

[0012] Step 5, voxelization of point cloud data: normalize the coordinates of the original input point cloud, determine the voxel number of each point according to the voxel resolution, and use the average feature of each point in the voxel as the current voxel feature.

[0013] Step 6, Local Feature Extraction: Perform sparse submanifold convolution on the voxelization results to extract local features of the point cloud;

[0014] Step 7, Voxelization: The local features extracted in Step 6 are devoxed using the trilinear interpolation upsampling method. The features of each voxel are mapped to the corresponding point cloud to obtain the local features.

[0015] Step 8, Feature Fusion: The global and local features obtained in Steps 4 and 7 are concatenated and fused, and then processed by multiple multilayer perceptrons to obtain the final semantic segmentation result.

[0016] Step 1 is as follows:

[0017] Step 1.1, Input the point cloud data of the 3D object P = {p1, p2, ..., p i ,…,p N}, where p i This represents the position information of the i-th point, and N is the number of points in the input point cloud, including the three-dimensional coordinate information of the point cloud.

[0018] Step 1.2, the input point cloud is first linearly transformed by a Linear layer, which usually includes a weight matrix and a bias term, and the calculation formula is y = wx + b, where w is the weight matrix, x is the input data, b is the bias term, and y is the output data;

[0019] Step 1.3, a Batch-Normalization layer is added after the Linear layer;

[0020] Step 1.4, a ReLU activation function is added after the Batch-Normalization layer to perform a non-linear mapping on the input data, setting all negative values to 0 and keeping all positive values unchanged;

[0021] Step 1.5, the output of step 1.4 is again processed by steps 1.2-1.4 to obtain the embedding feature d pe The embedding feature contains a feature dimension for each point.

[0022] Step 2 is as follows:

[0023] Step 2.1, the output F in of the embedding feature module is linearly transformed by a Linear layer to obtain the query Q, key K, and value V matrices, and the corresponding calculation formula is shown in equation (1):

[0024]

[0025] In equation (1):

[0026] W q , W k , and W v represent the weight matrices of the shared learnable linear transformation layer;

[0027] d a represents the dimensions of Q and K;

[0028] Step 2.2, after obtaining the Q, K, and V matrices, the attention weight is calculated using the query matrix and the key matrix through matrix dot product,

[0029]

[0030] Step 2.3, the attention weight is normalized by the SoftMax operator and l1Norm to obtain the attention graph, and the normalized result is represented as A = α i,j , and the specific calculation formula is shown in equation (3).

[0031]

[0032] Step 2.4, the obtained normalized result and value vector are weighted and summed to obtain the output feature F of the offset attention sa , and the specific calculation formula is shown in formula (4),

[0033] F sa =A·V (4);

[0034] Step 2.5, the output feature F of the offset attention sa and the initially input embedding feature F in are added to obtain the final output feature F end , and the specific calculation formula is shown in formula (5),

[0035] F end =LBR(F in -F sa )+F in (5)。

[0036] Step 5 is specifically as follows:

[0037] Step 5.1, the input point cloud data is subjected to coordinate normalization processing, and all points are translated to a local coordinate system with the barycenter as the origin, and all points are divided by the maximum value max||p i ||2 of the coordinate two-norm to standardize these points to a unit sphere, and the coordinates are scaled to [-1, 1];

[0038] Step 5.2, then the coordinate line of [-1, 1] is linearly mapped to [0, 1], the coordinate normalization is completed, and the normalized coordinate is represented as wherein After obtaining the normalized coordinate information, the coordinate and the voxel resolution r are calculated in succession to obtain the voxel number corresponding to each point;

[0039] Step 5.3, all features f k corresponding to the point coordinates falling into the voxel are subjected to average pooling operation, and the average feature is taken as the current voxel feature, and the normalized point cloud is converted into a voxel grid, and thus the voxelization operation of the point cloud data is completed, and the specific calculation formula is shown in formula (6),

[0040]

[0041] In formula (6),

[0042] r represents the voxel resolution;

[0043] II[·] represents a binary indicator of whether the coordinate belongs to the voxel grid (u, v, w);

[0044] f k,c —Representative and The relevant c-th channel feature;

[0045] N u,v,w — Represents the normalization factor, which is the number of points that fall within this voxel grid.

[0046] Step 6 is as follows:

[0047] Step 6.1: Build an input hash table (Hash) based on the voxelized data. in Data p in the voxel grid in Store in a hash table in In the table, v in Indicates the sequence number, key in Represents the coordinates of the corresponding data;

[0048] Step 6.2, Build the output hash table Hash out For the input hash table Hash in A set of data H i (i = 1, 2, 3, ..., in) will be able to include data points H i The coordinates p of the output tensor obtained by the convolution operation out Store in Hash out In the table, v out Hash in The data contained in the key out Represents coordinates;

[0049] Step 6.3, p out Convert to GetOffSet(p in ,P), save p out The position of the convolution kernel weights in the corresponding convolution operation is obtained from GetOffSet(p in In p), repeat step 6.3 until p out All values ​​in the table are mapped to GetOffSet;

[0050] Step 6.4, GetOffSet(p in The P) is converted to the Rule Book, which records the corresponding input point positions and the output positions of the convolution results. The first column of the Rule Book records the weight results of GetOffSet, count records the number of each convolution weight, v_in records the input sequence number of the corresponding point, and v_out records the output sequence number corresponding to the convolution result.

[0051] Step 6.5, using the Rule Book obtained by the above steps, first find the convolution kernel weight through the weight result, secondly, according to the input data point sequence number, find the corresponding tensor vector in the input hash table, and then multiply the corresponding tensor vector according to the convolution kernel weight to obtain the final output sparse tensor, and the local feature obtained by sparse convolution.

[0052] Step 7 is specifically as follows:

[0053] Step 7.1: for the points P in the local feature that need to be interpolated i , find the nearest 8 points a~g to the interpolation point in three dimensions, for each dimension, calculate the proportion of the distance of the interpolation point C from the nearest two points (x0, y0, z0) and (x1, y1, z1) in this dimension to the total distance, and the specific calculation is shown in formula (7),

[0054]

[0055] Step 7.2: after obtaining the weight of each dimension, interpolation is performed along the direction of the three coordinate axes, first interpolation along the x-axis direction, and the specific calculation is shown in formula (8),

[0056]

[0057] Then interpolate along the y-axis direction, and the specific calculation is shown in formula (9),

[0058]

[0059] Finally, interpolate along the z-axis direction, and the specific calculation is shown in formula (10),

[0060] C=C0(1-z d )+C1z d (10)

[0061] Step 7.3: repeat the above steps 7.1-7.2 for all points that need to be interpolated, map the local feature extracted after voxelization back to the original point cloud space, and complete the de-voxelization step.

[0062] The beneficial effects of the present application are:

[0063] The application is based on a point cloud semantic segmentation method combining Transformer and sparse convolution. Based on a point-based method, the application realizes effective extraction of local features by introducing a sparse convolution module, and adopts an attention mechanism more suitable for point cloud processing to extract global features, so as to reduce unnecessary processing procedures and solve the problem of insufficient extraction of local features in existing methods. While ensuring the accuracy of point-by-point global feature extraction, the application compensates for the problem that the point-based method cannot effectively extract local features by a voxel-based method, and introduces sparse sub-manifold convolution to better improve the efficiency of voxelized convolution, so that the resulting result has higher accuracy. The application can improve the accuracy and efficiency of indoor scene semantic segmentation and ensure the efficiency of data processing. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is a network architecture diagram of the point cloud semantic segmentation method combining Transformer and sparse convolution of the application;

[0065] Fig. 2 is a qualitative result diagram of indoor scene semantic segmentation in the S3DIS dataset, wherein Fig. 2(a) is a qualitative result diagram of a true value, and Fig. 2(b) is a qualitative result diagram of the application. DETAILED DESCRIPTION

[0066] The application will be described in detail below in combination with the drawings and specific embodiments.

[0067] Embodiment 1

[0068] The embodiment provides a point cloud semantic segmentation method based on a combination of Transformer and sparse convolution. First, the core attention mechanism in Transformer is used to extract global features of original point cloud data point by point. Then, the original point cloud data is voxelized, and sparse sub-manifold convolution is used to extract local features of the voxelized point cloud data. Finally, the extracted global features and local features are fused to obtain the final semantic segmentation result.

[0069] Embodiment 2

[0070] The embodiment provides a point cloud semantic segmentation method based on a combination of Transformer and sparse convolution. Based on embodiment 1, the method is implemented according to the following steps.

[0071] Step 1, feature embedding: input point cloud data, and use a shared neural network composed of two cascaded LBRs to combine original position encoding and input features into a coordinate-based encoding module;

[0072] Step 2, calculate offset attention: input the embedded features into the attention mechanism module for further feature extraction, specifically, first map the point cloud to a low-dimensional feature space through linear transformation, then calculate the offset vector between each point and its surrounding points, and use the offset vector to guide the calculation of attention;

[0073] Step 3, construct residual module: add the input and output of the offset attention, thereby completing the construction of the residual module;

[0074] Step 4, global feature extraction: first linearly connect the four residual modules to form a complete attention mechanism module, then linearly transform the output of the attention mechanism to obtain the point cloud features, and finally perform maximum pooling and average pooling operations on the point cloud features, combine the maximum pooled feature mapping and the average pooled feature mapping, and obtain the global features;

[0075] Step 5, point cloud voxelization: normalize the coordinates of the original input point cloud, determine the voxel number of each point according to the voxel resolution, and take the average feature of each point in the voxel as the current voxel feature;

[0076] Step 6, local feature extraction: perform sparse sub-manifold convolution operation on the voxelization result to extract local features of the point cloud;

[0077] Step 7, de-voxelization processing: use trilinear interpolation upsampling method to de-voxelize the local features extracted in step 6, map the features of each voxel to the corresponding point cloud, and obtain the local features;

[0078] Step 8, feature fusion: concatenate and fuse the global features and local features obtained in steps 4 and 7, and process them through multiple multi-layer perceptrons to obtain the final semantic segmentation result.

[0079] Embodiment 3

[0080] The embodiment provides a point cloud semantic segmentation method based on joint Transformer and sparse convolution, as shown in Figure 1 Fig. 2, based on embodiment 2, the implementation is specifically as follows:

[0081] Step 1, feature embedding: input the point cloud data, and use a shared neural network composed of two cascaded LBRs to combine the original position encoding and the input features into a coordinate-based encoding module;

[0082] Step 1.1, input the point cloud data P = {p1, p2, …, pN} of the three-dimensional object, where pi represents the position information of the i-th point, and N is the number of points in the input point cloud, which contains three-dimensional coordinate information of the point cloud; i N i , …, pN}, where pi represents the position information of the i-th point, and N is the number of points in the input point cloud, which contains three-dimensional coordinate information of the point cloud;​​

[0083] Step 1.2, the input point cloud is first linearly transformed by a Linear layer, which usually includes a weight matrix and a bias term, and the calculation formula is y = wx + b, where w is the weight matrix, x is the input data, b is the bias term, and y is the output data;

[0084] Step 1.3, a Batch-Normalization layer is added after the Linear layer, which normalizes the data in each mini-batch to have a mean of 0 and a variance of 1, which can improve the expression ability of the network;

[0085] Step 1.4, a ReLU activation function is added after the Batch-Normalization layer, which maps the input data nonlinearly, sets all negative values to 0 and keeps all positive values unchanged, which can make the network more expressive and improve its fitting ability for nonlinear data;

[0086] Step 1.5, the output of step 1.4 is processed again by steps 1.2-1.4 to obtain the embedding feature d pe is the feature dimension contained in each point of the embedding feature, d pe is 128.

[0087] Step 2, calculate the offset attention: input the embedding feature into the attention mechanism module for further feature extraction, specifically, first map the point cloud to a low-dimensional feature space through linear transformation, then calculate the offset vector between each point and its surrounding points, and use the offset vector to guide the calculation of attention;

[0088] Step 2.1, the output F in of the embedding feature module is linearly transformed by a linear layer to obtain the query Q, key K and value V matrices, and the corresponding calculation formula is shown in equation (1):

[0089]

[0090] In equation (1):

[0091] W q , W k and W v represent the weight matrix of the shared learnable linear transformation layer;

[0092] d a represents the dimensions of Q and K;

[0093] Step 2.2, after obtaining the Q, K and V matrices, the query matrix and the key matrix are used to calculate the attention weight The specific calculation formula is shown in formula (2),

[0094]

[0095] Step 2.3, attention weight The attention map is obtained by normalization processing through the SoftMax operator and the l1Norm, and the normalized result is represented as A = a i,j , and the specific calculation formula is shown in formula (3),

[0096]

[0097] Step 2.4, the obtained normalized result and value vector are weighted and summed to obtain the output feature F of the offset attention sa , and the specific calculation formula is shown in formula (4),

[0098] F sa = A·V (4);

[0099] Step 2.5, the output feature F of the offset attention sa and the initially input embedding feature F in are added to obtain the final output feature F end , and the specific calculation formula is shown in formula (5),

[0100] F end = LBR(F in -F sa )+F in (5)。

[0101] Step 3, constructing a residual module: adding the input and output of the offset attention, thereby completing the construction of the residual module;

[0102] Step 4, global feature extraction: first, linearly connecting four residual modules to form a complete attention mechanism module, then performing linear transformation on the output of the attention mechanism to obtain a point cloud feature, and finally performing maximum pooling operation and average pooling operation on the point cloud feature, combining the obtained maximum pooling feature mapping and average pooling feature mapping to obtain a global feature;

[0103] Step 5, point cloud data voxelization: performing coordinate normalization on the original input point cloud, determining the voxel number of each point according to the voxel resolution, and taking the average feature of each point in the voxel as the current voxel feature;

[0104] Step 5.1, coordinate normalization processing is performed on the input point cloud data, and all points are translated to a local coordinate system with the center of gravity as the origin, and all points are divided by the maximum value max||p i||2to normalize these points into a unit sphere, the coordinates are scaled to [-1, 1];

[0105] Step 5.2, then linearly map (scaling and translation) the coordinates of [-1, 1] to [0, 1], complete the coordinate normalization, and express the normalized coordinates as where After obtaining the normalized coordinate information, the coordinates and the voxel resolution r are calculated to obtain the voxel number corresponding to each point;

[0106] Step 5.3, average pooling operation is performed on all features f k corresponding to the point coordinates falling into the voxel to take the average feature as the current voxel feature, and the normalized point cloud is converted into a voxel grid, thus completing the voxelization operation of the point cloud data. The specific calculation formula is shown in equation (6),

[0107]

[0108] In equation (6):

[0109] r represents the voxel resolution;

[0110] II[·] represents the coordinates whether it belongs to the voxel grid (u, v, w);

[0111] f k,c represents the cth channel feature related to ;

[0112] N u,v,w represents the normalization factor, i.e., the number of points falling in the voxel grid.

[0113] Step 6, local feature extraction: sparse submanifold convolution operation is performed on the voxelization result to extract point cloud local features;

[0114] Step 6.1, establish an input hash table Hash in according to the voxelization data, store the data p in in the voxel grid in the hash table Hash in , where v in represents the serial number, and key in represents the coordinates of the corresponding data;

[0115] Step 6.2, establish an output hash table Hash out , for a group of data H in (i = 1, 2, 3……, in) in the input hash table Hash i , the data points H iThe coordinates p of the output tensor obtained by the convolution operation out Store in Hash out In the table, v out Hash in The data contained in the key out Represents coordinates;

[0116] Step 6.3, p out Convert to GetOffSet(p in ,P), save p out The position of the convolution kernel weights in the corresponding convolution operation is obtained from GetOffSet(p in In p), repeat step 6.3 until p out All values ​​in the table are mapped to GetOffSet;

[0117] Step 6.4, GetOffSet(p in The P) is converted to the Rule Book, which records the corresponding input point positions and the output positions of the convolution results. The first column of the Rule Book records the weight results of GetOffSet, count records the number of each convolution weight, v_in records the input sequence number of the corresponding point, and v_out records the output sequence number corresponding to the convolution result.

[0118] Step 6.5: Using the Rule Book obtained in the above steps, firstly, find the convolution kernel weights through the weight results. Secondly, based on the input data point index, look up the corresponding tensor vector in the input hash table. Then, multiply the convolution kernel weights by the corresponding tensor vector to obtain the final output sparse tensor, and obtain the local features through sparse convolution.

[0119] Step 7, Voxelization: The local features extracted in Step 6 are devoxed using the trilinear interpolation upsampling method. The features of each voxel are mapped to the corresponding point cloud to obtain the local features.

[0120] Step 7.1: Find the 8 points a to g closest to the interpolation point C in the three dimensions. For each dimension, find the two points p0(x0,y0,z0) and p1(x1,y1,z1) closest to the interpolation point C. Calculate the ratio of the distance between C and p0 to the total distance between p0 and p1. The specific calculation is shown in Equation (7).

[0121]

[0122] Step 7.2: After obtaining the weights of each dimension, interpolation is performed along the three coordinate axes. First, interpolation is performed along the x-axis, and the specific calculation is shown in equation (8).

[0123]

[0124]

[0125] Then interpolate along the y-axis direction, and the specific calculation is as shown in formula (9),

[0126]

[0127] Finally, interpolate along the z-axis direction, and the specific calculation is as shown in formula (10),

[0128] C=C0(1-z d )+C1z d (10)

[0129] Step 7.3: Repeat steps 7.1-7.2 above for all points that need to be interpolated, and map the local features extracted after voxelization back to the original point cloud space to complete the de-voxelization step.

[0130] Step 8, feature fusion: splice and fuse the global features and local features obtained in steps 4 and 7, and obtain the final semantic segmentation result through multiple multi-layer perception processing.

[0131] As can be seen from the above, the offset attention mechanism can efficiently extract point cloud global features without preprocessing, and the voxel-based sparse convolution method can effectively extract local features. In view of the problem that the point-based method cannot effectively extract local features, the present application uses the core attention mechanism in the Transformer to extract global features from the original point cloud data point by point, and introduces a sparse sub-manifold convolution to extract local features. By fusing global features and local features, the indoor scene point cloud semantic segmentation task is completed. The problem of insufficient local feature extraction of the point-based method is solved.

[0132] The method of the present application is tested on the S3DIS dataset, and FIG. 2(a) and FIG. 2(b) are the visualization results of the true value and the segmentation method of the present application respectively. By comparison, the first two scenes in the two figures are large stair classrooms under different angles. For such data-intensive indoor point cloud scenes, the method of the present application can better extract the points of the chair part and obtain a result closer to the true value, proving that the network framework of the present application can extract more accurate point features compared to the PointNet series of methods, and thus obtain better visualization effect. And for the last two indoor scenes in the two figures, which are relatively sparse in data density, the present application can also obtain good experimental results.

[0133] The experimental results of FIG. 2(a) show that the method reaches 68.7% and 78.9% in the two indexes of average intersection-over-union and average accuracy, respectively. It is proved that the method of combining the global feature and the local feature extraction and merging proposed by the application is good for realizing the feature transmission between points, and more accurately extracts each object. Table 1 is the quantitative results of each semantic category of the method on the S3DIS dataset. Through analysis, it is found that the method has good segmentation effect on the local details in the three-dimensional point cloud object, which also shows that the local feature extraction module of the application can better extract the local features of the input point cloud.

[0134] Table 1

[0135]

Claims

1. A point cloud semantic segmentation method based on joint Transformer and sparse convolution, characterized in that, First, the core attention mechanism in the Transformer is used to extract global features of the original point cloud data point by point, then the original point cloud data is voxelized, and the local features of the voxelized point cloud data are extracted by sparse sub-manifold convolution, and finally the extracted global features and local features are fused to obtain the final semantic segmentation result; The following steps are implemented: Step 1, feature embedding: input the point cloud data, and use a shared neural network composed of two cascaded LBRs to combine the original position encoding and the input features into a coordinate-based encoding module; Step 1.1, Input the point cloud data of the 3D object. ,in Indicates the first Location information of each point, The input is the number of points in the point cloud, including the three-dimensional coordinate information of the point cloud; Step 1.2, the input point cloud is first linearly transformed by a Linear layer, which usually includes a weight matrix and a bias term, and the calculation formula is y = wx + b , wherein w is the weight matrix, x is the input data, b is the bias term, y is the output data; Step 1.3, add a Batch-Normalization layer after the Linear layer; Step 1.4, add a ReLU activation function to the Batch-Normalization layer, which performs a nonlinear mapping on the input data, sets all negative values to 0, and keeps all positive values unchanged; Step 1.5, the output of Step 1.4 is processed again by Step 1.2-Step 1.4 to obtain the embedding features , is the feature dimension contained by each point of the embedding features; Step 2, calculate the offset attention: input the embedded features into the attention mechanism module for further feature extraction, specifically, first map the point cloud to a low-dimensional feature space through linear transformation, then calculate the offset vector between each point and its surrounding points, and use the offset vector to guide the calculation of attention; Step 2.1, outputting the embedding feature module linear transformation through a linear layer to obtain a query Q , a keyword K and a value V matrix, and the corresponding calculation formula is shown in equation (1): In formula (1): , and - represents the weight matrix of a shared learnable linear transformation layer; - represents and a dimension; Step 2.2, after obtaining , and matrices, the attention weights are calculated by matrix multiplication using the query matrix and the key matrix , the specific calculation formula is shown in equation (2), Step 2.3, attention weight by SoftMax operator and L 1Norm, and the normalized result is denoted as , and the specific calculation formula is shown in equation (3). Step 2.

4. Weighting and summing the obtained normalized results and value vectors to obtain the output features of the offset attention The specific calculation formula is shown in equation (4). Step 2.5, output features of shifted attention and the initially input embedding features are added to obtain final output features The specific calculation formula is shown in equation (5). Step 3, construct the residual module: add the input and output of the offset attention to complete the construction of the residual module; Step 4, global feature extraction: first linearly connect four residual modules to form a complete attention mechanism module, then linearly transform the output of the attention mechanism to obtain point cloud features, and finally perform maximum pooling and average pooling operations on the point cloud features, combine the maximum pooled feature map and the average pooled feature map, and obtain the global features; Step 5, voxelization of point cloud data: normalize the coordinates of the original input point cloud, determine the voxel number of each point according to the voxel resolution, and take the average feature of each point in the voxel as the current voxel feature; Step 6, local feature extraction: perform sparse sub-manifold convolution on the voxelization result to extract local features of the point cloud; Step 6.

1. Building an input hash table from the voxelized data Hash in The data in the voxel grid p in is stored in a hash table Hash in in which the v in order numbers key in represent the coordinates of the corresponding data; Step 6.2, Establishing the output hash table Hash out For the input hash table Hash in A set of data H i ( i =1,2,3……, in ), will be able to contain to the data point H i The coordinates of the output tensor obtained by the convolution operation p out are stored in Hash out , and v out represents p out the corresponding serial number, key out indicates the coordinates; Step 6.

3. Set p out transformed to GetOffSet ( p in ,P ), save p out the position of the kernel weight in the corresponding convolution operation to GetOffSet ( p in ,P ) and repeat step 6.3 until all values in p out correspond to GetOffSet ; Step 6.4, convert the GetOffSet ( p in ,P ) to Rule Book, record the corresponding input point position and the output position of the convolution result, the first column of the Rule Book table records the weight result of GetOffSet , count records the number statistics of each convolution weight, v_in records the input serial number of the corresponding point, v_out records the output serial number corresponding to the convolution result; Step 6.5, using the Rule Book obtained in the above steps, first find the convolution kernel weight through the weight result, second, according to the input data point sequence number, find the corresponding tensor vector in the input hash table, then multiply the corresponding tensor vector by the convolution kernel weight to obtain the final output sparse tensor, and obtain the local features through sparse convolution; Step 7, de-voxelization processing: use the trilinear interpolation upsampling method to de-voxelize the local features extracted in step 6, map the features of each voxel to the corresponding point cloud, and obtain the local features; Step 8, feature fusion: concatenate and fuse the global features obtained in step 4 and the local features obtained in step 7, and process them through multiple multilayer perceptrons to obtain the final semantic segmentation result.

2. The joint Transformer and sparse convolution based point cloud semantic segmentation method according to claim 1, characterized in that, The step 5 is specifically as follows: Step 5.

1. Coordinate normalization is performed on the input point cloud data, translating all points to a local coordinate system with the centroid as the origin, and dividing all points by the maximum coordinate two-norm to normalize these points into a unit sphere, the coordinates are scaled to [-1, 1]; Step 5.2, then linearly map the coordinates of [-1, 1] to [0, 1], complete the coordinate normalization, and express the normalized coordinates as wherein After obtaining the normalized coordinate information, the coordinates and the voxel resolution r are calculated to obtain the voxel number corresponding to each point. Step 5.

3. All features corresponding to the point coordinate falling into the voxel f k An average pooling operation is performed to average the features as the current voxel features, and the normalized point cloud is converted into a voxel grid. Thus, the voxelization operation of the point cloud data is completed, and the specific calculation formula is shown in equation (6). In formula (6): r represents the voxel resolution; II[·] - represents a coordinate a binary indicator of whether it belongs to the voxel grid of voxels; - represents the cth channel feature associated with the cth channel feature; - represents a normalization factor, i.e. the number of points falling in the voxel grid.

3. The joint Transformer and sparse convolution based point cloud semantic segmentation method according to claim 1, characterized in that, The step 7 is specifically as follows: Step 7.1: For points in the local features that require interpolation Find the 8 points a~g closest to the interpolation point in three dimensions. For each dimension, calculate the two points (a~g) closest to the interpolation point C in that dimension. x 0, y 0, z 0) and ( x 1, y 1, z The proportions of the distances 1) and 2) to the total distance are calculated as shown in equation (7). Step 7.2: After obtaining the weight of each dimension, interpolation is carried out along the direction of the three coordinate axes, first interpolation along the x-axis direction, and the specific calculation is as formula (8), Then interpolation along the y-axis direction, and the specific calculation is as formula (9), Finally, interpolation along the z-axis direction, and the specific calculation is as formula (10), Step 7.3: Repeat the above steps 7.1-7.2 for all points that need to be interpolated, and map the local features extracted after voxelization back to the original point cloud space, to complete the de-voxelization step.

Images