Transform-based channel-by-channel point cloud semantic segmentation method

Abstract

The invention provides a channel-by-channel point cloud semantic segmentation method based on Transform, and relates to the technical field of three-dimensional point cloud semantic segmentation. The method comprises the following steps: designing a position self-adaptive coding module and a space context coding module in a depth separable edge convolution postposition module according to a Transform feature extraction method; according to the method, multiple feature relationships are introduced into a low-level space, so that local geometric learning of point clouds is enhanced; according to the three-dimensional point cloud data segmentation method, a spatial context relation is coded in an advanced feature space by calculating spatial similarity measurement, and finally, a self-adaptive fusion strategy is provided for adjusting a weight coefficient to realize depth separable edge convolution and an output result of a spatial context coding module, so that accurate segmentation of the three-dimensional point cloud data is realized, and the segmentation precision of the three-dimensional point cloud data is improved. Compared with an existing method, the defect that a convolution kernel in common convolution indiscriminately fuses all channels is overcome, the calculation complexity of the network is simplified, and feature extraction of complex point cloud semantic information is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud semantic segmentation, and in particular relates to a channel-by-channel point cloud semantic segmentation method based on Transformer. Background Art

[0002] Point cloud semantic segmentation technology has made great progress in recent years, but its accuracy is still an urgent problem to be solved. Three-dimensional point cloud is a data structure that represents the surface shape of geometric objects and is widely used in fields such as autonomous driving, medical imaging, and robotics. Unlike two-dimensional images composed of regular dense pixels, point clouds are disordered and unstructured, so obtaining semantic information of point clouds is a challenging task.

[0003] The traditional deep learning-based method is to structure the unstructured point cloud data. One of the traditional methods is to voxelize the point cloud so that feature learning can be performed using three-dimensional discrete convolution. However, voxelization treats all points equally, ignoring the sparsity of point sets in different regions, and as the voxel resolution increases, the computational and memory costs of three-dimensional discrete convolutions increase exponentially. Therefore, the computational complexity will be significantly increased when semantic segmentation or processing complex scenes. The multi-view-based method is to project the point cloud from multiple perspectives onto a two-dimensional plane, and then apply a convolutional neural network for processing. However, point cloud information will be lost during the projection process.

[0004] In recent years, the research trend has shifted to directly processing point cloud data without introducing any intermediate representation. Modern methods tend to process point cloud data directly, using MLP (Multi-layer Perceptron, MLP) to extract features from unordered point clouds, and using maximum pooling to ensure the permutation invariance of point clouds. However, this method does not consider the geometric structure information of the point cloud neighborhood, resulting in insufficient feature extraction. In response to the problem that all point features are treated equally when MLP is used directly to analyze point clouds, a new point convolution operator is proposed in the convolution layer of the neural network. Different kernel point convolution weights are designed through linear relationships to distinguish the distance between the core point and the center point in different regions. The above improved methods mainly achieve adaptive feature extraction by improving the design of the convolution kernel, but do not deeply explore the geometric relationship of points in the region, and there is still the problem of insufficient local feature extraction.

[0005] Constructing effective neighborhood geometric relationships is the key to extracting neighborhood features of point clouds. In point cloud segmentation tasks, since point cloud features only contain three-dimensional coordinates, it is difficult to directly use ordinary convolution to extract implicit geometric structure information. Graph structures have attracted much attention from researchers because they can effectively represent non-Euclidean data. In recent years, graph convolution is used to construct structural information within the neighborhood of point clouds. The edge convolution operator EdgeConv (Edge Convolution, EdgeConv) is used to linearly aggregate central point features and edge features to capture local context information. This graph structure helps to make implicit features in point clouds explicit. Inspired by this, hierarchical features connecting different dynamic graphs are extended to achieve information transmission and fusion of features at different levels. Or by calculating the feature differences of points in the neighborhood, weighted processing of directed edges is performed to achieve feature aggregation within the region. Although the above methods have made significant progress in utilizing graph structures, densely connected graph structures will generate a lot of calculations and additional redundant information. In addition, ordinary convolution cannot distinguish different channels in the graph structure, which also brings certain limitations to feature learning.

[0006] In the field of natural language processing, the Transformer series of models have been widely used. The Transformer model dynamically adjusts the attention weights through the attention mechanism to capture the correlation between different positions in the sequence. In point cloud data, the correlation between different neighborhood points and the central point is also different. Based on this feature, the self-attention mechanism can also be effectively applied to feature learning of point clouds in local areas. In modern segmentation tasks, a novel point converter is constructed using the Transformer model attention mechanism to achieve feature extraction with point pairs as attention units. Or, a mask modeling strategy is introduced based on the attention mechanism to achieve feature training of point clouds under weak supervision. However, previous work used a single encoding method to process low-level coordinates and high-level features, resulting in insufficient information extraction. Summary of the invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the prior art and provide a channel-by-channel point cloud semantic segmentation method based on Transformer. The method realizes channel-by-channel feature extraction by designing a deeply separable edge convolution, which solves the deficiency of the convolution kernel in ordinary convolution that indiscriminately fuses all channels, simplifies the computational complexity of the model, and is conducive to feature extraction of complex point cloud semantic information.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: the present invention provides a channel-by-channel point cloud semantic segmentation method based on Transformer, comprising the following steps:

[0009] Step 1: Collect and process point cloud data: Collect real-world point cloud data and use LiDAR scanning imaging as 3D point cloud dataset samples, and divide the 3D point cloud dataset samples into training set, validation set and test set;

[0010] The point cloud data is composed of a large number of points, and the point cloud data contains the xyz axis coordinate information and RGB color information of the points; before the point cloud data is input into the deep convolutional network of deep learning, the point cloud data is preprocessed, and the points in the point cloud data that exceed the area of ​​interest are filtered out through denoising bounding box filtering, and only the points in the target area are retained; outliers and noise points are removed through statistical filtering; the processed point cloud data is used as the original input to provide a basis for the subsequent segmentation network;

[0011] Step 2: Design a segmentation network model based on Transformer: Take the point cloud data coordinate information as the original input, design and use the low-level geometric construction module to expand the dimension of the point cloud data coordinate information input, use the deep separable edge convolution and position adaptive encoding module to capture the geometric structure relationship of the point cloud data in the low-level space, and realize the encoding of the low-level point cloud data coordinate information. As the dimension of the point cloud data coordinate information increases, the high-level feature space of the point cloud data is obtained. The encoding in the high-level feature space of the point cloud data is realized by three modules: deep separable edge convolution, spatial context encoding module and fusion strategy, thereby forming a more comprehensive and more discriminative feature representation of deep learning convolution extraction. After the point cloud data coordinate information is feature encoded twice, it is spliced ​​with the category vector, including the following steps:

[0012] Step 2.1: Design low-level geometric construction modules to expand the dimension of point cloud data coordinate information input:

[0013] Input point cloud dataset P, the specific formula is:

[0014]

[0015] Among them, p i is the spatial coordinate vector of the i-th point, and N is the number of points. The Euclidean distance neighborhood search algorithm is used to construct a local area about the center point on the original point cloud data set. The specific formula is:

[0016] N i =(p i,j |j=1,2,...,k)

[0017] Among them, p i,j For p i The coordinates of the neighboring points, k is the number of neighboring points, N i is the set of points selected by the neighborhood search algorithm using Euclidean distance;

[0018] The two points closest to the center point form a plane, and the normal vector of the plane is used as the normal vector feature l of the center point. i , the features of the two edges and the normal vector features are concatenated as the feature p′ output by the low-level geometric construction module i , the specific formula is:

[0019]

[0020] p′ i =[e i,1 ,e i,2 ,l i ]

[0021] Among them, e i,1 Represents the directed edge between the center point and the nearest neighboring point, e i,2 represents the directed edge between the center point and the second closest neighboring point, [] represents the splicing operation, and p′ i Provide low-level geometric information of point cloud data for subsequent encoding;

[0022] Step 2.2: Design a depth-wise separable edge convolution to achieve information fusion between different channels and obtain the output feature map of the depth-wise separable edge convolution;

[0023] The depth-wise separable convolution consists of two-step convolution, namely depth-wise convolution and point-wise convolution;

[0024] In the depth convolution, different convolution kernels act on different input channels respectively to generate feature maps of the corresponding channels, and then these feature maps are spliced. The specific formula is:

[0025] g i,j =[p′ i -p′ i,j ,p′ i ]

[0026] y′ i,j,c′ =∑g i,j,c′ ω′ i,j,c′

[0027] Among them, [] represents the concatenation operation, i and j represent the indexes of the center point and the neighborhood point respectively, g is the input feature of the depthwise separable convolution, c′ is the channel index of the depthwise convolution, ω′ is the weight of the convolution kernel in the depthwise convolution, and y′ is the output feature after the depthwise separable convolution;

[0028] The point-by-point convolution combines the feature maps obtained by the depth convolution to achieve information fusion between different channels to obtain the output feature map of the depth-separable edge convolution. The specific formula for obtaining the output feature map Y of the depth-separable edge convolution is:

[0029]

[0030] Y=(y i )=(max(y i,j ))

[0031] Among them, c is the channel index of the point-by-point convolution kernel, ω is the weight of the point-by-point convolution kernel, Y is the feature map output after the depth-wise separable convolution and Y∈R N×F , F is the number of feature channels, y i,j is the characteristic value of the jth neighboring point after the Euclidean distance neighborhood search corresponding to the i-th sampling center point;

[0032] Step 2.3: Based on Transformer, a position adaptive encoding module and a spatial context encoding module are designed as post-modules of the depthwise separable edge convolution to further extract features, including the following steps;

[0033] Step 2.3.1: Design a position adaptive encoding module to calculate the attention weight of the original data: Use the position adaptive encoding module to embed the original point cloud data. In the position adaptive encoding module, the Q matrix and the K matrix represent the output feature map obtained after linear operations on the vertex coordinates and directed edges in the feature map, and the V matrix represents the output feature map after depth-separable convolution. The specific formula is:

[0034]

[0035] Among them, Linear is a linear operation, W q ,W k and W v is the weight matrix of different features, MLP is for feature extraction of unordered point cloud data, and E is e i The point set of

[0036] In the calculation of attention weight, the difference features between the Q matrix and the K matrix are first calculated, and then the V matrix containing the original features is combined to calculate the attention weight W T The specific formula is:

[0037] W T =Softmax(MLP(Q-K+V))

[0038] Among them, W T is the attention weight, Softmax is the normalized exponential function;

[0039] The K matrix is ​​composed of directed edge features and contains part of the information of the Q matrix composed of absolute positions. Therefore, the K matrix and the V matrix are selected as attention objects to extract the geometric structure information in the low-level space of the point cloud data. The output feature F of the position adaptive encoding module T, the specific formula is:

[0040] F T =(K+V)·W T

[0041] Step 2.3.2: Design the spatial context encoding module: Use the idea of ​​self-attention mechanism to encode the spatial context of point cloud data features; generate Q′ matrix, K′ matrix and V′ matrix by linearly transforming the output feature map Y of the depth-separable edge convolution. The specific formula is:

[0042] (Q′,K′,V′)=Y·(W′ q ,W′ k ,W′ v )

[0043] Among them, W′ q , W′ k , W′ v is the weight matrix of the learnable linear transformation;

[0044] The calculation of attention weight W′ T The specific formula is:

[0045]

[0046] Among them, d a is the characteristic dimension of Q′ matrix, K′ matrix and V′ matrix, W′ T is the attention weight matrix of the high-level features of the point cloud data, representing the spatial similarity of the point cloud data;

[0047] The position adaptive coding module adopts a feature extraction strategy of first fusion and then weighting, but the spatial context coding module adopts a feature extraction strategy of first weighting and then fusion. The output feature F′ of the spatial context coding module T The specific formula is:

[0048] F′ T =(V'·W' T )⊙α+Y

[0049] Among them, α is a learnable parameter, ⊙ is the Hadamard multiplication;

[0050] Step 2.4: Design an adaptive fusion module to enhance the interaction between different features and adjust the training weights through adaptive fusion strategies;

[0051] The features extracted by the spatial context encoding module and the depth-separable edge convolution are added element by element, and then the fusion weight Ω is calculated through a linear layer and normalization operation. The specific formula is:

[0052] Ω=Sigmoid(ψ(FT ′+Y))

[0053] Among them, ψ is a linear function and Sigmoid is an activation function;

[0054] The specific formula for the output of the adaptive fusion module is:

[0055] F out =Ω⊙Y+(1-Ω)⊙F′ T

[0056] Among them, F out is the output feature of the adaptive fusion module;

[0057] Step 3: Use the cross-validation method to obtain the experimental evaluation index, which is the average category intersection-union ratio. Introduce the trained weights into the divided test set to test the designed segmentation network model to obtain the segmentation result of the point cloud data.

[0058] The beneficial effect of adopting the above technical solution is that: the present invention provides a Transformer-based channel-by-channel point cloud semantic segmentation method, which designs deep separable edge convolution based on Transformer on the basis of graph structure, which is more flexible than traditional convolution and can treat information from different channels differently. At the same time, the position adaptive encoding module and spatial context encoding module in the deep separable edge convolution can better extract features, and then learn fusion parameters through the adaptive fusion module to enhance the interaction between different features; the effectiveness and robustness of this method have been proved through a large number of experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 An overall network diagram of a Transformer-based channel-by-channel point cloud semantic segmentation method provided in an embodiment of the present invention;

[0060] Figure 2 A diagram of a depth-separable edge convolution structure provided by an embodiment of the present invention;

[0061] Figure 3 The structural diagram of the position adaptive coding module and the spatial context coding module provided by the embodiment of the present invention, wherein (a) represents the position adaptive coding module, and (b) represents the spatial context coding module;

[0062] Figure 4 A structural diagram of an adaptive fusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0064] In this embodiment, a channel-by-channel point cloud semantic segmentation method based on Transformer is described. Figure 1 As shown, the following steps are included:

[0065] This embodiment collects 16,881 samples of real-world point cloud data, divides them into training sets, validation sets, and test sets as original input, designs the network structure and corresponding parameters, designs deep separable edge convolution to replace the traditional convolution layer, and designs a position adaptive encoding module and a spatial context encoding module based on Transformer as the post-module of the deep separable edge convolution to further extract features. Then, an adaptive fusion strategy is used to strengthen the interaction between different features and adjust the training weights. Finally, the trained weights are introduced into the divided test set to achieve accurate segmentation of the point cloud data;

[0066] Step 1: Collect and process point cloud data: Collect real-world point cloud data and use LiDAR scanning imaging as 3D point cloud dataset samples, and divide the 3D point cloud dataset samples into training set, validation set and test set;

[0067] The point cloud data is composed of a large number of points, and the point cloud data contains the xyz axis coordinate information and RGB color information of the points; before the point cloud data is input into the deep convolutional network of deep learning, the point cloud data is preprocessed, and the points in the point cloud data that exceed the area of ​​interest are filtered out through denoising bounding box filtering, and only the points in the target area are retained; through statistical filtering, based on statistical methods, obvious outliers and noise points are removed; the processed point cloud data is used as the original input to provide a basis for the subsequent segmentation network;

[0068] The data sets of this embodiment all use real-world point cloud data sets, which are imaged by laser radar scanning, with a total of 16,881 samples and 16 categories. Each point cloud data is composed of a large number of points, and the data contains the xyz axis coordinate information and RGB color information of the points. Before the data is input into the network, the data is preprocessed, denoising bounding box filtering is performed to filter out the points in the point cloud data that exceed the area of ​​interest, and only the data in the target area is retained. Statistical filtering is based on statistical methods to remove obvious outliers and noise points; conditional filtering is to set conditions such as height, density, etc., to filter out points that do not meet the conditions, and the processed data is used as the original input to obtain the results through the network model;

[0069] Step 2: Design a segmentation network model based on Transformer: Take the point cloud data coordinate information as the original input, design and use the low-level geometric construction module to expand the dimension of the point cloud data coordinate information input, use the deep separable edge convolution and position adaptive encoding module to capture the geometric structure relationship of the point cloud data in the low-level space, and realize the encoding of the low-level point cloud data coordinate information. As the dimension of the point cloud data coordinate information increases, the high-level feature space of the point cloud data is obtained. The encoding in the high-level feature space of the point cloud data is realized by three modules: deep separable edge convolution, spatial context encoding module and fusion strategy, thereby forming a more comprehensive and more discriminative feature representation of deep learning convolution extraction. After the point cloud data coordinate information is feature encoded twice, it is spliced ​​with the category vector. The category vector refers to the One-hot encoding result of the object category label, which can provide the category information of each point. Finally, the segmentation scores of 50 object categories are output through a fully connected layer with a dimension of (256,128,50). It includes the following steps:

[0070] Step 2.1: Design low-level geometric construction modules to expand the dimension of point cloud data coordinate information input:

[0071] Input point cloud dataset P, the specific formula is:

[0072]

[0073] Among them, p i is the spatial coordinate vector of the i-th point, and N is the number of points; the purpose is to enhance the low-level geometric relationship representation;

[0074] Use the Euclidean distance neighborhood search algorithm (k-NearestNeighbor, kNN) to construct a local area about the center point on the original point cloud dataset. The specific formula is:

[0075] N i =(p i,j |j=1,2,...,k)

[0076] Among them, p i,j For p i The coordinates of the neighboring points, k is the number of neighboring points, N i is the set of points selected by the neighborhood search algorithm using Euclidean distance;

[0077] The two points closest to the center point form a plane, and the normal vector of the plane is used as the normal vector feature l of the center point. i , the features of the two edges and the normal vector features are concatenated as the feature p′ output by the low-level geometric construction module i , the specific formula is:

[0078]

[0079] p′ i =[e i,1 ,e i,2 ,l i ]

[0080] Among them, e i,1 Represents the directed edge between the center point and the nearest neighboring point, e i,2 represents the directed edge between the center point and the second closest neighboring point, [] represents the splicing operation, and p′ i Provide low-level geometric information of point cloud data for subsequent encoding;

[0081] Step 2.2: Design a deep separable edge convolution to achieve information fusion between different channels and obtain the output feature map of the deep separable edge convolution, as shown in the attached figure. Figure 2 As shown;

[0082] The depth-wise separable convolution consists of two-step convolution, namely depth-wise convolution and point-wise convolution;

[0083] In the depth convolution, different convolution kernels act on different input channels respectively to generate feature maps of the corresponding channels, and then these feature maps are spliced. The specific formula is:

[0084] g i,j =[p′ i -p′ i,j ,p′ i ]

[0085] y′ i,j,c′ =∑g i,j,c′ ω′ i,j,c′

[0086] Among them, [] represents the concatenation operation, i and j represent the indexes of the center point and the neighborhood point respectively, g is the input feature of the depthwise separable convolution, c′ is the channel index of the depthwise convolution, ω′ is the weight of the convolution kernel in the depthwise convolution, and y′ is the output feature after the depthwise separable convolution;

[0087] The point-by-point convolution combines the feature maps obtained by the depth convolution to achieve information fusion between different channels to obtain the output feature map of the depth-separable edge convolution. The specific formula for obtaining the output feature map Y of the depth-separable edge convolution is:

[0088]

[0089] Y=(y i )=(max(y i,j ))

[0090] Among them, c is the channel index of the point-by-point convolution kernel, ω is the weight of the point-by-point convolution kernel, Y is the feature map output after the depth-wise separable convolution and Y∈R N×F , F is the number of feature channels, y i,j is the characteristic value of the jth neighboring point after the Euclidean distance neighborhood search corresponding to the i-th sampling center point;

[0091] Compared with ordinary edge convolution, depth-wise separable edge convolution can reduce the number of parameters and thus reduce computational complexity;

[0092] Step 2.3: Based on Transformer, a position adaptive encoding module and a spatial context encoding module are designed as post-modules of the depthwise separable edge convolution to further extract features, including the following steps;

[0093] The post-module of the depth-separable edge convolution is divided into a position adaptive encoding module and a spatial context encoding module; as shown in the attached Figure 3 As shown;

[0094] Step 2.3.1: Design a position adaptive encoding module to calculate the attention weight of the original data: In order to enhance the geometric relationship in the point cloud data and enrich the neighborhood features, the position adaptive encoding module is used to embed the original point cloud data. In the position adaptive encoding module, the (query) Q matrix and the (key) K matrix represent the output feature map obtained by linear operations on the vertex coordinates and directed edges in the feature map, respectively, which are called query matrix and key matrix, while the (value) V matrix represents the output feature map after the convolution layer, which is called value matrix. Since the position adaptive encoding module contains the original point cloud data, position encoding can be realized. The specific formula is:

[0095]

[0096] Among them, Linear is a linear operation, W q ,W k and W v is the weight matrix of different features, and MLP is used to extract features from unordered point cloud data;

[0097] In the calculation of attention weights, the difference features between the Q matrix and the K matrix are first calculated, and then the V matrix containing the original features is combined to enhance the learning ability of the errors in the embedding layer, so as to limit the propagation of redundant information in the low-level space and realize the forward error correction function of the network during the training process. Calculate the attention weight W T The specific formula is:

[0098] W T =Softmax(MLP(Q-K+V))

[0099] Among them, W T is the attention weight, Softmax is the normalized exponential function;

[0100] The K matrix is ​​composed of directed edge features, which is essentially a Laplacian matrix, and contains part of the information of the Q matrix composed of absolute positions. Therefore, the K matrix and the V matrix are selected as attention objects to extract the geometric structure information in the low-level space of the point cloud data. The output feature F of the position adaptive encoding module T , the specific formula is:

[0101] F T =(K+V)·W T

[0102] Step 2.3.2: Design spatial context encoding module: In high-level space, point cloud data features contain local and global information. In order to achieve effective information fusion and limit redundant information, the idea of ​​self-attention mechanism is used to encode the spatial context of point cloud data features; by linearly transforming the output feature map Y of the depth-separable edge convolution, the Q′ matrix, K′ matrix and V′ matrix are generated. The specific formula is:

[0103] (Q',K',V')=Y·(W' q ,W′ k ,W′ v )

[0104] Among them, W′ q , W′ k , W′ v is the weight matrix of the learnable linear transformation;

[0105] The calculation of attention weight W′ T The specific formula is:

[0106]

[0107] Among them, d a is the characteristic dimension of Q′ matrix, K′ matrix and V′ matrix, W′ T is the attention weight matrix of the high-level features of the point cloud data, representing the spatial similarity of the point cloud data;

[0108] The position adaptive coding module adopts a feature extraction strategy of first fusion and then weighting, but the spatial context coding module adopts a feature extraction strategy of first weighting and then fusion. The output feature F′ of the spatial context coding module T The specific formula is:

[0109] F′ T =(V′·W′ T )⊙α+Y

[0110] Among them, α is a learnable parameter used for adaptive feature learning, and ⊙ is the Hadamard multiplication;

[0111] Step 2.4: Design an adaptive fusion module to enhance the interaction between different features and adjust the training weights through adaptive fusion strategies;

[0112] The features extracted by the spatial context encoding module and the depth-separable edge convolution are added element by element, and then the weight Ω is calculated through a linear layer and normalization operation, as shown in Figure 4 As shown, the specific formula is:

[0113] Ω=Sigmoid(ψ(F′ T +Y))

[0114] Among them, ψ is a linear function and Sigmoid is an activation function;

[0115] The specific formula for the output of the adaptive fusion module is:

[0116] F out =Ω⊙Y+(1-Ω)⊙F′ T

[0117] Among them, F out is the output feature of the adaptive fusion module;

[0118] Step 3: Use the cross-validation method to obtain the experimental evaluation index, which is the mean intersection over Union (mIoU). Introduce the trained weights into the divided test set to test the designed segmentation network model and obtain the segmentation result of the point cloud data. The higher the experimental evaluation index, the better the point cloud segmentation result.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A channel-by-channel point cloud semantic segmentation method based on Transformer, characterized by: The following steps are involved: Step 1: Collect and process point cloud data: Collect real-world point cloud data and use LiDAR scanning imaging as 3D point cloud dataset samples, and divide the 3D point cloud dataset samples into training set, validation set and test set; Step 2: Design a segmentation network model based on Transformer: Take the point cloud data coordinate information as the original input, design and use the low-level geometric construction module to expand the dimension of the point cloud data coordinate information input, use the deep separable edge convolution and position adaptive encoding modules to capture the geometric structure relationship of the point cloud data in the low-level space, and realize the encoding of the low-level point cloud data coordinate information. As the dimension of the point cloud data coordinate information increases, the high-level feature space of the point cloud data is obtained. The encoding in the high-level feature space of the point cloud data is realized by three modules: deep separable edge convolution, spatial context encoding module and fusion strategy, thereby forming a more comprehensive and more discriminative feature representation of deep learning convolution extraction. After the point cloud data coordinate information is feature encoded twice, it is spliced ​​with the category vector; Step 3: Use the cross-validation method to obtain the experimental evaluation index, which is the average category intersection-union ratio. Introduce the trained weights into the divided test set to test the designed segmentation network model to obtain the segmentation result of the point cloud data.

2. The method for channel-by-channel point cloud semantic segmentation based on Transformer according to claim 1, characterized in that: The point cloud data is composed of a large number of points, and the point cloud data contains the xyz axis coordinate information and RGB color information of the points; before the point cloud data is input into the deep convolutional network of deep learning, the point cloud data is preprocessed, and the points in the point cloud data that exceed the area of ​​interest are filtered out through denoising bounding box filtering, and only the points in the target area are retained; outliers and noise points are removed through statistical filtering; the processed point cloud data is used as the original input to provide a basis for the subsequent segmentation network.

3. The method for channel-by-channel point cloud semantic segmentation based on Transformer according to claim 1, characterized in that: The step 2 designs a segmentation network model based on Transformer, including the following steps: Step 2.1: Design low-level geometric construction modules to expand the dimension of point cloud data coordinate information input: Input point cloud dataset P, the specific formula is: Among them, p i is the spatial coordinate vector of the i-th point, and N is the number of points. The Euclidean distance neighborhood search algorithm is used to construct a local area about the center point on the original point cloud data set. The specific formula is: N i (p i,j |j=1,2,…,k) Among them, p i,j For p i The coordinates of the neighboring points, k is the number of neighboring points, N i is the set of points selected by the neighborhood search algorithm using Euclidean distance; The two points closest to the center point form a plane, and the normal vector of the plane is used as the normal vector feature l of the center point. i , the features of the two edges and the normal vector features are concatenated as the feature p' output by the low-level geometric construction module i , the specific formula is: p' i =[and i,1 ,And i,2 ,the i ] Among them, e i,1 Represents the directed edge between the center point and the nearest neighboring point, e i,2 represents the directed edge between the center point and the second closest neighboring point, [] represents the splicing operation, p i 'Provide low-level geometric information of point cloud data for subsequent encoding; Step 2.2: Design a depth-wise separable edge convolution to achieve information fusion between different channels and obtain the output feature map of the depth-wise separable edge convolution; Step 2.3: Design a position adaptive encoding module and a spatial context encoding module based on Transformer as post-modules of the depthwise separable edge convolution to further extract features; Step 2.4: Design an adaptive fusion module to enhance the interaction between different features and adjust the training weights through adaptive fusion strategies; The features extracted by the spatial context encoding module and the depth-separable edge convolution are added element by element, and then the fusion weight Ω is calculated through a linear layer and normalization operation. The specific formula is: Ω=Sigmoid(ψ(F T ′+Y)) Among them, ψ is a linear function and Sigmoid is an activation function; The specific formula for the output of the adaptive fusion module is: F out =Ω⊙Y+(1-Ω)⊙F′ T Among them, F out is the output feature of the adaptive fusion module.

4. The method for channel-by-channel point cloud semantic segmentation based on Transformer according to claim 3, characterized in that: The depth-wise separable convolution consists of two-step convolution, namely depth-wise convolution and point-wise convolution; In the depth convolution, different convolution kernels act on different input channels respectively to generate feature maps of the corresponding channels, and then these feature maps are spliced. The specific formula is: g i,j =[p′ i -p′ i,j ,p′ i ] y′ i,j,c′ =∑g i,j,c′ ·oh′ i,j,c′ Among them, [] represents the concatenation operation, i and j represent the indexes of the center point and the neighborhood point respectively, g is the input feature of the depthwise separable convolution, c′ is the channel index of the depthwise convolution, ω′ is the weight of the convolution kernel in the depthwise convolution, and y′ is the output feature after the depthwise separable convolution; The point-by-point convolution combines the feature maps obtained by the depth convolution to achieve information fusion between different channels to obtain the output feature map of the depth-separable edge convolution. The specific formula for obtaining the output feature map Y of the depth-separable edge convolution is: And=(and i )=(max(y i,j )) Among them, c is the channel index of the point-by-point convolution kernel, ω is the weight of the point-by-point convolution kernel, Y is the feature map output after the depth-wise separable convolution and Y∈R N×F , F is the number of feature channels, y i,j It is the characteristic value of the jth neighborhood point after the Euclidean distance neighborhood search corresponding to the i-th sampling center point.

5. The method for channel-by-channel point cloud semantic segmentation based on Transformer according to claim 4, characterized in that: The step 2.3 designs a position adaptive encoding module and a spatial context encoding module based on Transformer as a post-module of the depth-separable edge convolution to further extract features, including the following steps: Step 2.3.1: Design a position adaptive encoding module to calculate the attention weight of the original data: Use the position adaptive encoding module to embed the original point cloud data. In the position adaptive encoding module, the Q matrix and the K matrix represent the output feature map obtained after linear operations on the vertex coordinates and directed edges in the feature map, and the V matrix represents the output feature map after depth-separable convolution. The specific formula is: Among them, Linear is a linear operation, W q ,W k and W v is the weight matrix of different features, MLP is for feature extraction of unordered point cloud data, and E is e i The point set of In the calculation of attention weight, the difference features between the Q matrix and the K matrix are first calculated, and then the V matrix containing the original features is combined to calculate the attention weight W T The specific formula is: W T =Softmax(MLP(Q-K+V)) Among them, W T is the attention weight, Softmax is the normalized exponential function; The K matrix is ​​composed of directed edge features and contains part of the information of the Q matrix composed of absolute positions. Therefore, the K matrix and the V matrix are selected as attention objects to extract the geometric structure information in the low-level space of the point cloud data. The output feature F of the position adaptive encoding module T , the specific formula is: F T =(K+V)·W T Step 2.3.2: Design the spatial context encoding module: Use the idea of ​​self-attention mechanism to encode the spatial context of point cloud data features; generate Q′ matrix, K′ matrix and V′ matrix by linearly transforming the output feature map Y of the depth-separable edge convolution. The specific formula is: (Q',K',V')=Y·(W q ',W k ',W v ') Among them, W q ', W k ', W v ' is the weight matrix of the learnable linear transformation; The calculation of attention weight W T The specific formula of ′ is: Among them, d a is the characteristic dimension of Q′ matrix, K′ matrix and V′ matrix, W T ′ is the attention weight matrix of the high-level features of point cloud data, representing the spatial similarity of point cloud data; The position adaptive coding module adopts a feature extraction strategy of first fusion and then weighting, but the spatial context coding module adopts a feature extraction strategy of first weighting and then fusion. The output feature F of the spatial context coding module T The specific formula of ′ is: F T ′=(V'·W T ′)⊙α+Y Among them, α is a learnable parameter and ⊙ is the Hadamard multiplication.

Images