A method and device for semantic segmentation of three-dimensional point cloud with multi-domain attention enhancement

Abstract

The present invention discloses a method and device for three-dimensional point cloud semantic segmentation with multi-domain attention enhancement, and relates to the field of computer vision technology. It includes: obtaining three-dimensional point cloud data to be segmented; inputting the three-dimensional point cloud data into a trained three-dimensional point cloud semantic segmentation model with multi-domain attention enhancement; wherein the three-dimensional point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module, and a segmentation head module; according to the three-dimensional point cloud data and the three-dimensional point cloud semantic segmentation model, a three-dimensional point cloud semantic segmentation result is obtained. The present invention designs a point cloud semantic segmentation method, which uses multi-domain attention to enhance the network segmentation performance while controlling the computational cost, and forms a custom configuration framework to facilitate users to control model training and other processes, and quickly deploy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for three-dimensional point cloud semantic segmentation with multi-domain attention enhancement. Background Art

[0002] 3D point cloud semantic segmentation is a basic task for advanced artificial intelligence tasks such as autonomous driving navigation and industrial automatic control, and is also a research hotspot in 3D computer vision and deep learning. It requires outputting the category value of each point in the input point cloud based on the 3D spatial coordinates, RGB values ​​and other data of the input point cloud.

[0003] Currently, fully supervised deep learning networks are mainly used to complete the semantic segmentation of 3D point clouds. According to different network architectures, existing work can be divided into four categories: convolutional neural network-based, local attention network-based, global attention network-based, and other network-based methods. The ease of use, operating costs (hardware conditions, training time, inference time), and evaluation index values ​​(OAcc overall accuracy, mOAcc average accuracy, mIoU average intersection-over-union ratio) of different methods vary greatly. For example, although the PointTransformer point cloud converter network has a higher evaluation index value, it requires a better GPU, long-term training, and low universality.

[0004] Methods based on CNN (Convolutional Neural Networks): CNN is often used to extract image features. Due to the disordered arrangement and uneven spatial distribution of point cloud data, convolution based on spatial distribution is not suitable for direct processing of point clouds. Therefore, existing work mainly uses CNN in two ways. One is to divide the point cloud into voxels according to spatial distribution, and design three-dimensional voxel convolution that can act on voxels, such as SurroundOcc, which uses three-dimensional voxel convolution to extract voxel features layer by layer. However, since voxel division consumes a lot of computing resources, it is not commonly used. The other is to combine convolution with pooling function, so that convolution directly acts on the point cloud encoding features, and then use pooling function to extract key point features as the features of the entire point cloud.

[0005] Qi et al. used PointNet (Graphic Processing Unit) to directly apply one-dimensional convolutional networks to point clouds for the first time. PointNet outputs key points with the largest eigenvalues ​​from an unordered point cloud through a function that is independent of the order of the point cloud, the maximum convergence function, and uses the features of the key points as the features of all points in the point cloud. At the same time, PointNet also introduces an STN (Spatial Transformer Network) to standardize the input point cloud, and then inputs the data into PointNet to improve the accuracy of network segmentation. PointNet++ uses FPS (Farest Point Sampling) to construct a neighborhood point cloud for each point, and then uses two-dimensional convolution to act on the neighborhood point cloud, and uses the learned neighborhood point features as the features of the center point. However, because PointNet++ uses a two-dimensional convolution with a kernel size of 1, it assigns the same weight to different points in the neighborhood, and the extracted features do not distinguish between different neighborhood points, and the network performance is limited.

[0006] In order to assign different weights to different neighboring points and ensure that the result of point cloud convolution is independent of the order of input points, Li et al. designed χ-convolution and proposed PoitCNN. χ-convolution first uses a multi-layer perceptron to encode the relative position between the neighborhood points and the center point, and then uses another multi-layer perceptron to generate a χ-transformation matrix based on the relative position encoding and the input point features. The transformation matrix is ​​used to sort the points in the neighborhood and assign two-dimensional convolution weights at the corresponding positions, so that the weights are related to the center point and the entire input point cloud, but not to the order of the input points in the point cloud. PoitCNN is the earliest point cloud convolution variant network based on two-dimensional convolution, which lays the foundation for subsequent research to propose various point cloud convolution variants.

[0007] Although PoitCNN can identify different neighborhood points, it is difficult to adapt to input point clouds with different point densities. Thomas et al. proposed kernel point convolution, which first defines a set of virtual kernel points with a fixed spatial position. Each kernel point is weighted by a special function to determine its mapped feature space. By finding the nearest kernel point around the input point, the encoding features of the input point are obtained using the weight of the kernel point and the relative position of the input point and the kernel point. The network based on KPConv (Kernel Point Convolution) can adapt to changes in the density of neighborhood point clouds of different input points. In addition, Thomas et al. extended variable kernel convolution based on kernel point convolution, which can change the spatial position of the kernel point according to the input point, so as to better adapt to point clouds with different geometric structures.

[0008] KPConv can extract local geometric features of point clouds very well, but it needs to generate kernel points in advance and the required video memory for calculation is large. Some works propose to construct point clouds into graph structures. Compared with kernel point convolution, they can better extract point cloud features of different geometric structures under the premise of controlling computational complexity. Wang et al. proposed DGCNN (Dynamic Graph Convolution Neural Network) to use the relative positions of input point positions and neighborhood points to form a graph relationship, generate graph convolution weights for each dimensional position channel of the neighborhood points, and obtain the center point features. Since the graph convolution weights are shared and have nothing to do with the order of the points in the point cloud, the graph convolution neural network has permutation invariance. However, as the number of network layers increases, the amount of graph calculation increases dramatically, so it is difficult to construct a deep graph convolution neural network to extract more fine-grained point cloud features.

[0009] This type of method can effectively extract local features in point clouds, but it has two disadvantages: first, it does not fully utilize the information of different neighboring points in the local area. Second, it is difficult to model the dependency relationship between distant points.

[0010] Method based on local attention network: To solve the problem that CNN is difficult to adapt to changes in point cloud density and geometric structure, the local attention network learns the attention scores of different neighborhood points in the entire neighborhood, assigns weights to the neighborhood points accordingly, and finally obtains the features of the center point by weighting.

[0011] Wang et al. proposed a graph attention convolutional network. It simultaneously learns the attention weights and attention weight distribution of neighborhood points, and filters unimportant neighborhood points according to the learned distribution, so that the model can learn the most relevant neighborhood features. Xie et al. proposed an attention graph convolutional network, which is based on a graph convolutional network. It not only uses attention to learn the relationship between local structures, but also attaches a global point graph to learn the correlation between single points, so as to better aggregate local information. Liang et al. defined K nearest neighbors based on attention on a graph convolutional network, and automatically selected and aggregated the most important neighbor point features by assigning different weights to each neighbor point. Zhang et al. proposed an attention adversarial learning network, so that the network pays more attention to different neighborhood information. The dynamic graph attention network further differentiates each edge of the constructed local graph by integrating an extended graph attention module implemented by an offset attention mechanism, so as to better learn edge features. Hu et al. proposed RandLA-Net, which is based on a simple and efficient network structure that randomly aggregates sampling and local features. It uses a hierarchical structure similar to Poi ntNet++. In the feature extraction of each layer, a local feature aggregation module with attention pooling is used to learn complex local features, which can adapt to point clouds of different densities and geometric structures to a certain extent.

[0012] This type of method can effectively learn the features of different neighborhood points through local attention, but it also has two disadvantages: first, there is feature redundancy in the features extracted by different local attentions; second, the segmentation performance of the network on outdoor datasets is worse than that on indoor datasets, and insufficient information is extracted between global points.

[0013] Methods based on global attention networks: To solve the problem that local attention networks are insufficient in extracting global information, global attention networks mainly use global channel attention or Transformer converter networks.

[0014] Feng et al. used the point-by-point spatial attention module to generate a global point correlation matrix based on extracting local features with local attention edge convolution, and used this matrix to learn more accurate long-distance spatial features. In order to obtain global information of point clouds, Ma et al. proposed a point global context reasoning method. The model uses channel attention to learn a channel graph, where the graph nodes are the features of the input points on a certain channel, and the graph edges are the correlation between any two channel features. It also learns the spatial correlation of point clouds and the correlation of different channel features.

[0015] Transformer uses self-attention, which is the development of local attention and offset attention. It is naturally permutation-invariant and rigid-body-invariant to the input point cloud, and is good at modeling long-distance global point relationships. Therefore, a lot of work in recent years has been done to improve network performance based on Transformer. Guo et al. designed a method to enhance the model's ability to learn local features by adding a vector mapping structure of neighborhood points and four consecutive offset attentions to the encoder.

[0016] To further increase the receptive field of self-attention, Zhang et al. proposed using Transformer to enhance feature extraction acting on points and large-volume voxels respectively, and combined a local attention algorithm with linear complexity to increase local features. Zhao et al. designed Point Transformer, which uses global self-attention to enable the network to directly capture the features of the entire point cloud.

[0017] Since global self-attention has high spatial and temporal complexity on large-scale point cloud datasets, Wu et al. improved Point Transformer and proposed grouped vector attention, which effectively reduced the number of model parameters. Park et al. proposed a fast and lightweight Transformer, which effectively improved the computational efficiency of the network by encoding point cloud coordinates and voxel hashes mapped by coordinates. Zhang et al. proposed a point cloud Transformer network based on block attention, which adaptively learned the features of smaller point sets and reduced the computational complexity of global self-attention.

[0018] This type of method uses global attention and can learn long-distance context information well. However, it also has two disadvantages: first, the computational complexity and memory usage of global attention are high, making it difficult to support fast training on a single GPU; second, global attention prioritizes modeling global features, which will spontaneously generate coarse-grained features and make it difficult to extract complex semantic information.

[0019] Therefore, it is an urgent problem to study and design a 3D point cloud semantic segmentation framework that can be quickly deployed and used, has a reasonable operating cost, and has high evaluation index values ​​(OAcc, mOAcc, mIoU). Summary of the invention

[0020] In order to solve the problems that the existing methods are poor in generalization, the segmentation accuracy of the existing methods has a large gap between indoor and outdoor point cloud data sets; the existing methods cannot balance the operating cost and segmentation performance well, and the methods with better performance generally require very high operating costs; the existing methods are poor in usability, and when users modify the data processing, model parameters, optimization methods, etc. in the existing methods, they need to modify the source code, which often leads to the technical problems that the methods cannot run, the embodiments of the present invention provide a multi-domain attention enhanced 3D point cloud semantic segmentation method and device. The technical solution is as follows:

[0021] On the one hand, a method for three-dimensional point cloud semantic segmentation with multi-domain attention enhancement is provided, the method is implemented by a three-dimensional point cloud semantic segmentation device, and the method includes:

[0022] S1. Obtain three-dimensional point cloud data to be segmented.

[0023] S2. Input the 3D point cloud data into the trained multi-domain attention enhanced 3D point cloud semantic segmentation model.

[0024] Among them, the 3D point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module.

[0025] S3. Obtain a 3D point cloud semantic segmentation result based on the 3D point cloud data and the 3D point cloud semantic segmentation model.

[0026] Optionally, the three-dimensional point cloud semantic segmentation result is obtained in S3 according to the three-dimensional point cloud data and the three-dimensional point cloud semantic segmentation model, including:

[0027] S31. Input the three-dimensional point cloud data into the convolutional attention encoder module to obtain the key point features of the encoder module.

[0028] S32. Input the key point features of the encoder module into the Transformer encoder module to obtain the key point features that integrate the long-distance dependency features.

[0029] S33. Input the key point features integrated with the long-distance dependency features into the convolutional attention decoder module to obtain the convolutional attention decoder output.

[0030] S34. Input the convolutional attention decoder output into the segmentation head module to obtain the 3D point cloud semantic segmentation result.

[0031] Optionally, the convolutional attention encoder module includes a point embedding stage module, a first local feature extraction stage module, a second local feature extraction stage module, and a third local feature extraction stage module.

[0032] The first local feature extraction stage module, the second local feature extraction stage module and the third local feature extraction stage module respectively include a dense local feature aggregation module DLFA and a sparse local feature expansion module SLFE.

[0033] In S31, the three-dimensional point cloud data is input into the convolutional attention encoder module to obtain the key point features of the encoder module, including:

[0034] S311. Input the three-dimensional point cloud data into the point embedding stage module, encode the three-dimensional point cloud data, and obtain point cloud features.

[0035] S312, input the point cloud features into the DLFA of the first local feature extraction stage module to obtain the key point features of the first local feature extraction stage module, input the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module to obtain the output features of the first local feature extraction stage module.

[0036] S313. Input the output features of the first local feature extraction stage module into the DLFA of the second local feature extraction stage module to obtain the key point features of the second local feature extraction stage module; input the key point features of the second local feature extraction stage module into the SLFE of the second local feature extraction stage module to obtain the output features of the second local feature extraction stage module.

[0037] S314. Input the output features of the second local feature extraction stage module into the DLFA of the third local feature extraction stage module to obtain the key point features of the third local feature extraction stage module, and input the key point features of the third local feature extraction stage module into the SLFE of the third local feature extraction stage module to obtain the key point features of the encoder module.

[0038] Optionally, the step of inputting the point cloud features into the DLFA of the first local feature extraction stage module in S312 to obtain the key point features of the first local feature extraction stage module includes:

[0039] The farthest point sampling algorithm is used to downsample the point cloud features to obtain the key points of the DLFA of the first local feature extraction stage module.

[0040] The key point of the DLFA of any first local feature extraction stage module is taken as the center point, and the neighboring points of the center point are calculated using the K nearest neighbor algorithm.

[0041] According to the coordinates of the center point and the coordinates of the neighboring points, the neighborhood geometry related features are constructed.

[0042] The first convolutional block is used to encode the neighborhood geometry-related features to obtain implicit position encoding.

[0043] The implicit position code is concatenated with the original features of the neighboring points, and the concatenated features are input into the second convolutional block to obtain the updated neighboring point features of the DLFA of the first local feature extraction stage module.

[0044] The updated neighbor point features of the DLFA of the first local feature extraction stage module are input into the attention convergence layer of the DLFA of the first local feature extraction stage module to obtain the attention convergence layer output of the DLFA of the first local feature extraction stage module.

[0045] The output of the attention aggregation layer of the DLFA of the first local feature extraction stage module is input into the bias embedding block for encoding to obtain the display position encoding.

[0046] The attention convergence layer output of the DLFA of the first local feature extraction stage module and the display position encoding are added to obtain the key point features of the first local feature extraction stage module.

[0047] Optionally, the SLFE of the first local feature extraction stage module includes multiple layers of SLFE.

[0048] Inputting the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module in S312 to obtain the output features of the first local feature extraction stage module includes:

[0049] According to the key point features of the first local feature extraction stage module and the K nearest neighbor algorithm, the neighboring points of the key points of the SLFE of the first local feature extraction stage module are obtained.

[0050] According to the neighboring points of the key points of the SLFE of the first local feature extraction stage module, the updated neighboring point features of the SLFE of the first local feature extraction stage module are obtained.

[0051] The updated neighbor point features of the SLFE of the first local feature extraction stage module are input into the attention convergence layer of the SLFE of the first local feature extraction stage module to obtain the attention convergence layer output of the SLFE of the first local feature extraction stage module.

[0052] The output of the attention pooling layer of the SLFE of the first local feature extraction stage module is input into the expanded multi-layer perceptron layer to obtain the key point features with added semantic information.

[0053] The key point features of the first local feature extraction stage module and the key point features after adding semantic information are added, and the added features are input into the batch normalization layer and the nonlinear activation layer to obtain the output features of the first local feature extraction stage module.

[0054] Optionally, the Transformer encoder module includes multiple layers of Transformers.

[0055] In S32, the key point features of the encoder module are input into the Transformer encoder module to obtain the key point features that integrate the long-distance dependency features, including:

[0056] S321. Input the key point features of the encoder module into the first layer Transformer, divide the coordinate space of the key point features of the encoder module according to the window sampling algorithm, obtain non-overlapping cubic windows, and obtain the point set in each cubic window.

[0057] S322. According to the point set, the relative positions between the points in each cubic window are calculated, and according to the relative positions, the joint window multi-head self-attention features of the fused neighborhood relative position encoding are calculated.

[0058] S323. Add the joint window multi-head self-attention features and the encoder module key point features to obtain the key point features that integrate the long-distance dependency features.

[0059] Optionally, the training process of the 3D point cloud semantic segmentation model in S2 includes:

[0060] S21. Obtain a point cloud data set, perform data sampling on each point cloud file in the point cloud data set, and obtain a sub-sampling point set.

[0061] S22. Perform data enhancement and data encapsulation on the sub-sampling point set to obtain a training data set.

[0062] S23. According to the training data set, the learning rate scheduler, the cross entropy loss and the optimizer, the three-dimensional point cloud semantic segmentation model is trained and optimized to obtain a trained three-dimensional point cloud semantic segmentation model.

[0063] Among them, the learning rate scheduler is used to preheat the learning to the maximum learning rate first, and then decay to the minimum learning rate using the cosine function.

[0064] The cross entropy loss includes class weights and label smoothing.

[0065] On the other hand, a multi-domain attention enhanced 3D point cloud semantic segmentation device is provided, and the device is applied to the multi-domain attention enhanced 3D point cloud semantic segmentation method, and the device comprises:

[0066] The acquisition module is used to acquire the three-dimensional point cloud data to be segmented.

[0067] The input module is used to input 3D point cloud data into the trained multi-domain attention enhanced 3D point cloud semantic segmentation model.

[0068] Among them, the 3D point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module.

[0069] The output module is used to obtain the 3D point cloud semantic segmentation result based on the 3D point cloud data and the 3D point cloud semantic segmentation model.

[0070] On the other hand, a three-dimensional point cloud semantic segmentation device is provided, which includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the three-dimensional point cloud semantic segmentation methods with multi-domain attention enhancement as described above is implemented.

[0071] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any of the above-mentioned three-dimensional point cloud semantic segmentation methods with multi-domain attention enhancement.

[0072] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0073] In the embodiment of the present invention, the designed network uses multi-domain attention to effectively extract features of different domains of the point cloud. Among them, the DLFA in the convolutional attention encoder uses attention to extract features of dense local areas in the point cloud, and the SLFE uses attention to extract features of sparse local areas in the point cloud. The conversion encoder uses self-attention to extract features of a large window area of ​​the point cloud.

[0074] The network designed by the present invention uses the architecture of convolutional attention encoder-transformation encoder-convolutional attention decoder, which minimizes the computational complexity and running memory of the model while ensuring a certain segmentation accuracy. First, a convolutional attention encoder is used to extract a small number of key point features in the point cloud, which greatly reduces the number of point clouds input to the transformation encoder, while increasing the semantic information of the point features input to the transformation encoder. The transformation encoder is then used to extract the dependency features between distant points in the point cloud, making up for the disadvantage that the convolutional attention encoder is difficult to model long-distance relationships. Compared with the conversion block using global self-attention, the present invention uses a conversion block using UWSA with less computational complexity.

[0075] The general model training process designed by the present invention formalizes the process from sample preparation, model training, and model optimization. In the sample preparation stage, the original data is used to the maximum extent, the sample diversity is increased, and the sample quality is improved. In the model training stage, a special learning rate scheduler is used to stabilize the model learning process, a special cross entropy loss is used to alleviate the impact of the imbalance of the sample midpoint categories, and the most advanced optimizer is used to help the model update parameters faster and better.

[0076] The 3D semantic segmentation framework constructed by the present invention supports operating environment inspection, enables distributed multi-machine and multi-card training, realizes automated data set preprocessing, and provides configuration file-based model and training management, so that users can easily train models. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a flow chart of a method for semantic segmentation of three-dimensional point cloud with multi-domain attention enhancement provided by an embodiment of the present invention;

[0079] Figure 2 is a network structure diagram of multi-domain attention enhancement provided by an embodiment of the present invention;

[0080] Figure 3 is a dense local feature aggregation module diagram provided by an embodiment of the present invention;

[0081] Figure 4 is a sparse local feature expansion module diagram provided by an embodiment of the present invention;

[0082] Figure 5 is a diagram of the FPS and K nearest neighbors process provided by an embodiment of the present invention;

[0083] Figure 6 is a neighborhood geometry correlation feature map provided by an embodiment of the present invention;

[0084] Figure 7 is a convolution block structure diagram provided by an embodiment of the present invention;

[0085] Figure 8 is a structural diagram of a bias embedding block provided by an embodiment of the present invention;

[0086] Fig. 9 is a structural diagram of an expanded multi-layer perceptron provided by an embodiment of the present invention;

[0087] Fig.10 is a structural diagram of a conversion encoder provided by an embodiment of the present invention;

[0088] Fig.11 is a diagram of an attention feature broadcasting module provided by an embodiment of the present invention;

[0089] Fig.12 It is a block diagram of a three-dimensional point cloud semantic segmentation device with multi-domain attention enhancement provided by an embodiment of the present invention;

[0090] Fig.13 It is a structural schematic diagram of a three-dimensional point cloud semantic segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0091] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0092] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0093] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0094] In the embodiments of the present invention, sometimes a subscript such as W1 may be expressed in a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0095] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0096] The embodiment of the present invention provides a method for 3D point cloud semantic segmentation with multi-domain attention enhancement, which can be implemented by a 3D point cloud semantic segmentation device, which can be a terminal or a server. Figure 1 The flowchart of the multi-domain attention enhanced 3D point cloud semantic segmentation method shown in the figure can include the following steps:

[0097] S1. Obtain three-dimensional point cloud data to be segmented.

[0098] S2. Input the 3D point cloud data into the trained multi-domain attention enhanced 3D point cloud semantic segmentation model.

[0099] Among them, Figure 2 As shown in the figure, the 3D point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module.

[0100] S3. Obtain a 3D point cloud semantic segmentation result based on the 3D point cloud data and the 3D point cloud semantic segmentation model.

[0101] Optionally, the above step S3 may include the following steps S31-S34:

[0102] S31. Input the three-dimensional point cloud data into the convolutional attention encoder module, and use the convolutional attention encoder to learn the key point features in the point cloud.

[0103] Optionally, the convolutional attention encoder module includes a point embedding stage, a first local feature extraction stage, a second local feature extraction stage, and a third local feature extraction stage.

[0104] Among them, each local feature extraction stage includes DLFA (Dense Local Feature Aggregation, dense local feature aggregation module) and SLFE (Sparse Local Feature Expanding, sparse local feature expansion module).

[0105] In a feasible implementation, the convolutional attention encoder has a total of 4 stages, the first stage is the point embedding stage, and the 2nd to 4th stages are the local feature extraction stages. According to specific requirements, the 2nd to 4th stages can be increased or decreased, which is not limited here.

[0106] The above step S31 may include the following steps S311-S314, including:

[0107] S311. Input the three-dimensional point cloud data into the point embedding stage, encode the three-dimensional point cloud data, and obtain point cloud features.

[0108] In one possible implementation, the convolutional attention encoder is fed with input having coordinate features and non-coordinate features The input point cloud {(p, f)|, where represents the real number domain, N is the number of initial points, 3 represents the (x, y, z) coordinate dimension, Represents the initial non-coordinate dimension.

[0109] Furthermore, the point embedding stage uses convolutional blocks to directly encode the input point cloud features f as Where N m represents the number of key points in the mth stage, m represents the number of stages, Represents the non-coordinate dimension of the output features of the mth stage.

[0110] S312, input the point cloud features into the DLFA of the first local feature extraction stage to obtain the key point features of the first local feature extraction stage, and input the key point features of the first local feature extraction stage into the SLFE of the first local feature extraction stage to obtain the output features of the first local feature extraction stage.

[0111] In a feasible implementation, the local feature extraction stage mainly uses a dense local feature aggregation module (such as Figure 3 ) and sparse local feature expansion modules (such as Figure 4 ), update key point features.

[0112] Optionally, the steps of stages 2-4 in the local feature extraction stage are similar. The first stage is taken as an example for explanation. The point cloud first passes through the DLFA module:

[0113] Inputting the point cloud features into the DLFA of the first local feature extraction stage in S312 to obtain the key point features of the first local feature extraction stage may include the following steps A1-A8:

[0114] A1. Use the farthest point sampling algorithm to downsample the point cloud features and obtain the key points of DLFA in the first local feature extraction stage.

[0115] In one possible implementation, DLFA processes the input point cloud Use FPS (Farthest Point Sampling) to specify the downsampling rate r s Downsample the key points in the input point cloud

[0116] A2. Taking any key point of the DLFA in the first local feature extraction stage as the center point, the K nearest neighbor algorithm is used to calculate the neighboring points of the center point.

[0117] In a feasible implementation, the i-th key point As the center, use the K nearest neighbor algorithm to calculate its K nearest neighbor points The specific process is as follows Figure 5 .

[0118] A3. Construct neighborhood geometry-related features based on the coordinates of the center point and the coordinates of the neighboring points.

[0119] In a feasible implementation, the neighborhood geometry-related features are constructed based on the coordinates of the center point and the neighboring points. like Figure 6 .in represents the relative position between the i-th key point and the j-th neighbor point, represents the L2 norm of the relative position, Indicates the elevation angle of the neighboring point relative to the key point, Indicates the x-axis azimuth of the neighboring point relative to the key point.

[0120] A4. Use the first convolutional block to encode the neighborhood geometry-related features to obtain implicit position encoding.

[0121] In one possible implementation, a convolutional block is used to encode Get implicit position encoding

[0122] A5. Concatenate the implicit position code with the original features of the neighboring points, and input the concatenated features into the second convolution block to obtain the updated neighboring point features of the DLFA in the first local feature extraction stage.

[0123] In a feasible implementation, The original features of the neighboring points After concatenation, input a convolution block to obtain the updated i-th neighbor point feature The convolutional block structure is as follows Figure 7 .

[0124] A6. Input the updated neighbor point features of the DLFA in the first local feature extraction stage to the attention convergence layer of the DLFA in the first local feature extraction stage to obtain the attention convergence layer output of the DLFA in the first local feature extraction stage.

[0125] In a feasible implementation, Input attention pooling layer attention is obtained

[0126] A7. Input the attention aggregation layer output of the DLFA in the first local feature extraction stage into the bias embedding block for encoding to obtain the display position encoding.

[0127] In one possible implementation, a bias embedding block (such as Figure 8 ) Encode key point coordinates To display the position code

[0128] A8. Add the attention convergence layer output of the DLFA in the first local feature extraction stage and the display position code to obtain the key point features of the first local feature extraction stage.

[0129] In one feasible implementation, and Add up the i key point features output by the DLFA module

[0130] In the same local feature extraction stage, after the DLFA module, the point cloud passes through multiple SLFE modules to update the key point features. The execution steps of multiple SLFE modules are similar. Now take the first SLFE module as an example to illustrate:

[0131] Optionally, the SLFE of the first local feature extraction stage includes multiple layers of SLFE.

[0132] Inputting the key point features of the first local feature extraction stage into the SLFE of the first local feature extraction stage in S312 to obtain the output features of the first local feature extraction stage may include the following steps B1-B5:

[0133] B1. Obtain neighboring points of the key points of the SLFE in the first local feature extraction stage according to the key point features in the first local feature extraction stage and the K nearest neighbor algorithm.

[0134] B2. Obtain updated neighbor point features of the SLFE in the first local feature extraction stage according to the neighbor points of the key points of the SLFE in the first local feature extraction stage.

[0135] In a feasible implementation, the SLFE module directly uses the K nearest neighbor algorithm to find the nearest neighbor points for the input sparse key points, and then uses a similar process to DLFA to obtain the updated i-th nearest neighbor point feature

[0136] B3. Input the updated neighbor point features of the SLFE in the first local feature extraction stage to the attention convergence layer of the SLFE in the first local feature extraction stage to obtain the attention convergence layer output of the SLFE in the first local feature extraction stage.

[0137] In one feasible implementation, the key point features output by the attention pooling layer are used

[0138] B4. Input the output of the attention pooling layer of the SLFE in the first local feature extraction stage into the expanded multi-layer perceptron layer to obtain the key point features with added semantic information.

[0139] In one possible implementation, an extended multilayer perceptron (e.g. Fig. 9 )Add semantic information of key point features.

[0140] B5. Add the key point features of the first local feature extraction stage and the key point features after adding semantic information, and input the added features into the batch normalization layer and the nonlinear activation layer to obtain the output features of the first local feature extraction stage.

[0141] In one possible implementation, the input feature of SLFE is And the output of the dilated multilayer perceptron Add, and then pass through the batch normalization layer and nonlinear activation layer to get Enter the next SLFE.

[0142] After N SLFE SLFE, get the i-th key point feature output in the m-th stage

[0143] S313, input the output features of the first local feature extraction stage into the DLFA of the second local feature extraction stage to obtain the key point features of the second local feature extraction stage, and input the key point features of the second local feature extraction stage into the SLFE of the second local feature extraction stage to obtain the output features of the second local feature extraction stage.

[0144] S314. Input the output features of the second local feature extraction stage into the DLFA of the third local feature extraction stage to obtain the key point features of the third local feature extraction stage, and input the key point features of the third local feature extraction stage into the SLFE of the third local feature extraction stage to obtain the key point features of the encoder module.

[0145] In one feasible implementation, after a total of 4 stages, the output key point features are obtained

[0146] S32. Input the key point features of the encoder module into the Transformer encoder module to obtain the key point features that integrate the long-distance dependency features.

[0147] Optionally, the Transformer encoder module consists of multiple Transformer blocks connected in series, such as Fig.10 shown.

[0148] Taking the first layer Transformer block as an example, the above step S32 may include the following steps S321-S323:

[0149] S321. Input the key point features of the encoder module into the first layer Transformer, divide the coordinate space of the key point features of the encoder module according to the window sampling algorithm, obtain non-overlapping cubic windows, and obtain the point set in each cubic window.

[0150] In one feasible implementation, the key point features output by the convolutional attention encoder are Input the conversion block and use the window sampling algorithm to divide the coordinate space of the input point cloud into non-overlapping cubic windows. Assume that the side length of the small window is s sw , the side length of the large window is s lw =2s sw , map the coordinates of each point to a small window index and a large window index. For each window, if it is a small window, keep all points as a point set; if it is a large window, first use FPS to specify the downsampling rate r s , downsample the key point set in the window, and then filter out the point set that is not in the same small window as the other key points from the key points.

[0151] S322. According to the point set, the relative positions between the points in each cubic window are calculated, and according to the relative positions, the joint window multi-head self-attention features of the fused neighborhood relative position encoding are calculated.

[0152] In one feasible implementation, the relative positions between points in the window are calculated The relative position Δp of the jth neighbor point of the i-th key point in the window i,j Maps an integer index id i,j , use an independent and learnable function T to transform id i,j Mapped to NRPE (Neighborhood Relative Position Encoding) The calculation formula is as follows:

[0153]

[0154]

[0155] e i，j =T(id i，j ) (3)

[0156] S323, add the joint window multi-head self-attention feature and the encoder module key point feature to obtain the key point feature y that integrates the long-distance dependency feature U-NRPE .

[0157] In a feasible implementation, the UWSA (Union Window Multi-head Self-attention) features of the NRPE fusion are calculated.

[0158] Specifically, assuming that there are N w windows, the point set in the tth window in Indicates the number of points in the window, N h represents the number of self-attention heads in the window, Indicates the feature dimension of the point to be extracted by each attention head.

[0159] make Keep the UWSA feature dimension consistent with the feature dimension of the input point, and map the key point feature f to the query tensor (p), keyword tensor (k), and value tensor (v):

[0160] q=Linear q (f)

[0161] k=Linear k (f)

[0162] v=Linear v (f)

[0163]

[0164] Take the i-th key point as the query tensor and the j-th key point as the keyword tensor, and use the corresponding neighborhood relative position encoding to calculate the attention bias in represents the NRPE corresponding to the query tensor in the hth attention head, represents the NRPE corresponding to the keyword tensor in the hth attention head, Represents the NRPE corresponding to the value tensor in the h-th attention head.

[0165] Further, the dot product self-attention is calculated and used as the weight of the value tensor to obtain the self-attention feature of the i-th key point in the independent window at the h-th attention head

[0166]

[0167]

[0168] Furthermore, the output features of all attention heads are connected in parallel, and the sum of the self-attention features of the two windows corresponding to the i-th key point is taken as the UWSA feature

[0169]

[0170] In order to avoid the loss of local features, the output of the last conversion block is added to the input features of the first conversion block, and then passed through an MLP (Multilayer Perceptron) to obtain the key point features of the final output of the conversion encoder

[0171] S33. Input the key point features integrated with the long-distance dependency features into the convolutional attention decoder module to obtain the convolutional attention decoder output.

[0172] One possible implementation is to use a convolutional attention decoder ( Figure 2 )Broadcast the key point features to each input point. The convolutional attention decoder has 4 stages. The specific process is as follows:

[0173] In the first stage of the convolutional attention decoder, the output features of the fourth stage of the convolutional attention encoder are and the output feature f of the transformation encoder TE Concatenate, and then use convolutional blocks to fuse the features into Where N m The number of points is the same as the number of key points in the mth stage of the convolutional encoder. The feature dimension of the representation point is the same as the non-coordinate dimension of the output feature of the mth stage of the convolutional encoder.

[0174] In the decoder stage 5-m (m=3, 2, 1), the attention feature broadcast module ( Fig.11 ), the output features of the decoder stage 4-m Broadcast to the output of the mth stage of the convolutional attention encoder N m At the key point, Then use the convolution block to and f m，itp Fusion After 4 stages of decoding, the decoder output is finally obtained.

[0175] S34. Input the convolutional attention decoder output into the segmentation head module to obtain the 3D point cloud semantic segmentation result.

[0176] In one possible implementation, a segmentation head is used to map the output features of the convolutional attention decoder to the class label predicted by the network for each point.

[0177] The present invention designs a method for multi-domain attention enhancement of deep neural network to achieve efficient semantic segmentation of three-dimensional point cloud. Attention is used in the convolutional attention encoder to extract local features of different densities in the point cloud. Among them, DLFA makes the most of the local neighborhood information of the point to learn the point cloud features, and SLFE enriches the semantic information in the point cloud features to the greatest extent. The conversion encoder uses joint window multi-head self-attention to extract features of a large range of windows in the point cloud, enhancing the ability of the encoder to encode key point features in the point cloud. In the CNN decoder, attention is used to broadcast key point features to neighborhood points, enhancing the ability of the decoder to decode all point features.

[0178] Furthermore, the network architecture of convolutional attention encoder-transformation encoder-convolutional attention decoder is designed, which significantly reduces the amount of computation and video memory required to run the network. The convolutional attention encoder first extracts key point features and adds semantic information to the features. Then the transformation encoder is used to add long-distance dependency information to the key point features. Finally, the convolutional attention decoder is used to gradually decode the key point information.

[0179] Optionally, the present invention designs a general model training process, including:

[0180] S21. Obtain a point cloud data set, perform data sampling on each point cloud file in the point cloud data set, and obtain a sub-sampling point set.

[0181] In a feasible implementation, data sampling is performed on each point cloud file of the data set, specifically implemented as follows:

[0182] Grid sparse sampling. First, the coordinates of the input point are converted to grid coordinates with the grid size as the unit, and the whole is translated to the first quadrant of the space. Then the coordinates of the point are mapped to hash values. For the point set with hash collision, a point is retained randomly to reduce the density of the input point cloud.

[0183] Complete spherical sampling. Specify the number of sampling points N, and randomly generate a sampling probability a for each input point; each time sampling, take the point with the minimum sampling probability p as the sampling center, calculate the distance between all points and the center and sort them, select the first N points as a sampled sub-point set, and record the index of the sampled point, then add Δa to the sampling probability a of the sampled point, and loop to sample the next sub-point set until all points have been sampled.

[0184] Sub-point cloud formatting and saving. Save all sub-sampled point sets of a point cloud file in the previous step as a .npy file.

[0185] S22. Perform data enhancement and data encapsulation on the sub-sampling point set to obtain a training data set.

[0186] In a feasible implementation, the sub-sampling point set is read, and data enhancement and data encapsulation are performed on the point set. Specifically, the following is implemented:

[0187] Random coordinate rotation: The entire point cloud is randomly rotated around the specified coordinate axis within a specified range of angles.

[0188] Coordinate random scaling: The coordinates of the entire point cloud are multiplied by the specified scaling factor s at the same time.

[0189] Coordinate or color random discard: specify to set the coordinate value of a certain axis of the entire point cloud to 0, or set all color channels to 0.

[0190] Coordinate or color random jitter: A different random value within a specified range is added to each axis coordinate of the entire point cloud at the same time, or a different random value within a specified range is added to each color channel at the same time.

[0191] Coordinate normalization: subtract the mean value from the coordinates of the entire point cloud, and then divide it by the maximum range in the three axes.

[0192] Point shuffling: If it is a training set point cloud, randomly arrange the points in the entire point cloud, otherwise skip this process.

[0193] Data encapsulation: Take a processed point set as a sample, N b The samples are packaged into one small batch of point clouds.

[0194] S23. According to the training data set, the learning rate scheduler, the cross entropy loss and the optimizer, the three-dimensional point cloud semantic segmentation model is trained and optimized to obtain a trained three-dimensional point cloud semantic segmentation model.

[0195] Among them, the learning rate scheduler is used to preheat the learning to the maximum learning rate first, and then decay to the minimum learning rate using the cosine function.

[0196] The cross entropy loss includes class weights and label smoothing.

[0197] In a feasible implementation, training the model and executing the training optimization strategy specifically includes:

[0198] Use a learning rate scheduler that first warms up to the maximum learning rate and then decays to the minimum learning rate using a cosine function.

[0199] Use cross entropy loss with class weights and label smoothing.

[0200] AdamW with a weight decay rate of 1e-03 is used as the optimizer.

[0201] S24. Record the training time, loss value and other values ​​during the training process, and intersperse the model verification process. Check the records in real time to monitor the effectiveness of model training and use them as a basis for adjusting the model and training process.

[0202] The present invention develops a universal 3D point cloud semantic segmentation training process and algorithm framework, which enables users to easily define models and training processes. By formalizing the model training process, the ideal model can be obtained faster and better. At the same time, the training log is monitored and the segmentation results are visualized to ensure the effectiveness of the trained model.

[0203] The present invention builds a 3D semantic segmentation framework, integrating modules such as operating environment inspection, data set preprocessing, model training, and segmentation result visualization. Specifically, it includes:

[0204] Run environment check: Run the check_env.sh script to check whether the host has started the hardware and software environment required to run the framework. If the hardware environment does not have a GPU, it cannot run; if it has multiple GPUs, distributed training is performed. If the software environment is not available, the required software is automatically downloaded.

[0205] Dataset preprocessing: Run the prepare_xxx.sh script to check whether the dataset has been preprocessed and convert it into a format that can be read by the framework.

[0206] Training settings: Users first edit the configuration file according to their needs, including the configuration of the corresponding data set, model parameter configuration, training strategy configuration, etc. Then run the train.sh script to retrain the model or continue to train the existing model, and monitor the training status by viewing the training log.

[0207] Visualization of segmentation results: Run the visualize.sh script, run the pre-trained model, perform semantic segmentation on the validation set, and output a comparison chart of the actual segmentation and predicted segmentation of the point cloud in the interactive interface.

[0208] In the embodiment of the present invention, the designed network uses multi-domain attention to effectively extract features of different domains of the point cloud. Among them, the DLFA in the convolutional attention encoder uses attention to extract features of dense local areas in the point cloud, and the SLFE uses attention to extract features of sparse local areas in the point cloud. The conversion encoder uses self-attention to extract features of a large window area of ​​the point cloud.

[0209] The network designed by the present invention uses the architecture of convolutional attention encoder-transformation encoder-convolutional attention decoder, which minimizes the computational complexity and running memory of the model while ensuring a certain segmentation accuracy. First, a convolutional attention encoder is used to extract a small number of key point features in the point cloud, which greatly reduces the number of point clouds input to the transformation encoder, while increasing the semantic information of the point features input to the transformation encoder. The transformation encoder is then used to extract the dependency features between distant points in the point cloud, making up for the disadvantage that the convolutional attention encoder is difficult to model long-distance relationships. Compared with the conversion block using global self-attention, the present invention uses a conversion block using UWSA with less computational complexity.

[0210] The general model training process designed by the present invention formalizes the process from sample preparation, model training, and model optimization. In the sample preparation stage, the original data is used to the maximum extent, the sample diversity is increased, and the sample quality is improved. In the model training stage, a special learning rate scheduler is used to stabilize the model learning process, a special cross entropy loss is used to alleviate the impact of the imbalance of the sample midpoint categories, and the most advanced optimizer is used to help the model update parameters faster and better.

[0211] The 3D semantic segmentation framework constructed by the present invention supports operating environment inspection, enables distributed multi-machine and multi-card training, realizes automated data set preprocessing, and provides configuration file-based model and training management, so that users can easily train models.

[0212] Fig.12 1 is a block diagram of a 3D point cloud semantic segmentation device with multi-domain attention enhancement according to an exemplary embodiment, wherein the device is used in a 3D point cloud semantic segmentation method with multi-domain attention enhancement. Fig.12 The device includes an acquisition module 310, an input module 320 and an output module 330. Among them:

[0213] The acquisition module 310 is used to acquire the three-dimensional point cloud data to be segmented.

[0214] The input module 320 is used to input the three-dimensional point cloud data into the trained multi-domain attention enhanced three-dimensional point cloud semantic segmentation model.

[0215] Among them, the 3D point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module.

[0216] The output module 330 is used to obtain a 3D point cloud semantic segmentation result based on the 3D point cloud data and the 3D point cloud semantic segmentation model.

[0217] In the embodiment of the present invention, the designed network uses multi-domain attention to effectively extract features of different domains of the point cloud. Among them, the DLFA in the convolutional attention encoder uses attention to extract features of dense local areas in the point cloud, and the SLFE uses attention to extract features of sparse local areas in the point cloud. The conversion encoder uses self-attention to extract features of a large window area of ​​the point cloud.

[0218] The network designed by the present invention uses the architecture of convolutional attention encoder-transformation encoder-convolutional attention decoder, which minimizes the computational complexity and running memory of the model while ensuring a certain segmentation accuracy. First, a convolutional attention encoder is used to extract a small number of key point features in the point cloud, which greatly reduces the number of point clouds input to the transformation encoder, while increasing the semantic information of the point features input to the transformation encoder. The transformation encoder is then used to extract the dependency features between distant points in the point cloud, making up for the disadvantage that the convolutional attention encoder is difficult to model long-distance relationships. Compared with the conversion block using global self-attention, the present invention uses a conversion block using UWSA with less computational complexity.

[0219] The general model training process designed by the present invention formalizes the process from sample preparation, model training, and model optimization. In the sample preparation stage, the original data is used to the maximum extent, the sample diversity is increased, and the sample quality is improved. In the model training stage, a special learning rate scheduler is used to stabilize the model learning process, a special cross entropy loss is used to alleviate the impact of the imbalance of the sample midpoint categories, and the most advanced optimizer is used to help the model update parameters faster and better.

[0220] The 3D semantic segmentation framework constructed by the present invention supports operating environment inspection, enables distributed multi-machine and multi-card training, realizes automated data set preprocessing, and provides configuration file-based model and training management, so that users can easily train models.

[0221] Fig.13 is a schematic diagram of the structure of a 3D point cloud semantic segmentation device provided by an embodiment of the present invention, such as Fig.13 As shown, the 3D point cloud semantic segmentation device may include the above Fig.12 The 3D point cloud semantic segmentation apparatus with multi-domain attention enhancement is shown. Optionally, the 3D point cloud semantic segmentation device 410 may include a first processor 2001 .

[0222] Optionally, the three-dimensional point cloud semantic segmentation device 410 may also include a memory 2002 and a transceiver 2003 .

[0223] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0224] Combine the following Fig.13 The components of the 3D point cloud semantic segmentation device 410 are specifically introduced as follows:

[0225] The first processor 2001 is the control center of the 3D point cloud semantic segmentation device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0226] Optionally, the first processor 2001 can perform various functions of the three-dimensional point cloud semantic segmentation device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0227] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig.13 CPU0 and CPU1 are shown in FIG.

[0228] In a specific implementation, as an embodiment, the 3D point cloud semantic segmentation device 410 may also include multiple processors, such as Fig.13 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0229] The memory 2002 is used to store the software program for executing the solution of the present invention, and the first processor 2001 controls the execution. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0230] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Fig.13 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0231] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0232] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig.13 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0233] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Fig.13 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0234] It should be noted that Fig.13 The structure of the three-dimensional point cloud semantic segmentation device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0235] In addition, the technical effects of the three-dimensional point cloud semantic segmentation device 410 can refer to the technical effects of the multi-domain attention enhanced three-dimensional point cloud semantic segmentation method of the above-mentioned method embodiment, which will not be repeated here.

[0236] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0237] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0238] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A 3D point cloud semantic segmentation method with multi-domain attention enhancement, characterized in that: The method comprises: S1. Obtain three-dimensional point cloud data to be segmented; S2. Inputting the three-dimensional point cloud data into the trained multi-domain attention enhanced three-dimensional point cloud semantic segmentation model; The three-dimensional point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module; S3. Obtaining a 3D point cloud semantic segmentation result according to the 3D point cloud data and the 3D point cloud semantic segmentation model; The step of obtaining a 3D point cloud semantic segmentation result according to the 3D point cloud data and the 3D point cloud semantic segmentation model in S3 includes: S31, inputting the three-dimensional point cloud data into the convolutional attention encoder module to obtain the key point features of the encoder module; S32, inputting the key point features of the encoder module into the Transformer encoder module to obtain key point features that incorporate long-distance dependency features; S33, inputting the key point features fused with the long-distance dependency features into the convolutional attention decoder module to obtain a convolutional attention decoder output; S34, inputting the output of the convolutional attention decoder into the segmentation head module to obtain a three-dimensional point cloud semantic segmentation result; The convolutional attention encoder module includes a point embedding stage module, a first local feature extraction stage module, a second local feature extraction stage module and a third local feature extraction stage module; The first local feature extraction stage module, the second local feature extraction stage module and the third local feature extraction stage module respectively include a dense local feature aggregation module DLFA and a sparse local feature expansion module SLFE; The step of inputting the three-dimensional point cloud data into the convolutional attention encoder module in S31 to obtain the encoder module key point features includes: S311, inputting the three-dimensional point cloud data into the point embedding stage module, encoding the three-dimensional point cloud data, and obtaining point cloud features; S312, inputting the point cloud features into the DLFA of the first local feature extraction stage module to obtain the key point features of the first local feature extraction stage module, and inputting the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module to obtain the output features of the first local feature extraction stage module; S313, inputting the output features of the first local feature extraction stage module into the DLFA of the second local feature extraction stage module to obtain the key point features of the second local feature extraction stage module, and inputting the key point features of the second local feature extraction stage module into the SLFE of the second local feature extraction stage module to obtain the output features of the second local feature extraction stage module; S314, inputting the output features of the second local feature extraction stage module into the DLFA of the third local feature extraction stage module to obtain the key point features of the third local feature extraction stage module, and inputting the key point features of the third local feature extraction stage module into the SLFE of the third local feature extraction stage module to obtain the key point features of the encoder module; The step of inputting the point cloud features into the DLFA of the first local feature extraction stage module in S312 to obtain the key point features of the first local feature extraction stage module includes: Using the farthest point sampling algorithm, down-sampling the point cloud features to obtain the key points of the DLFA of the first local feature extraction stage module; Taking the key point of the DLFA of any first local feature extraction stage module as the center point, using the K nearest neighbor algorithm to calculate the neighboring points of the center point; According to the coordinates of the center point and the coordinates of the neighboring points, the neighborhood geometry related features are constructed; Using a first convolutional block, encoding the neighborhood geometry-related features to obtain an implicit position code; Concatenating the implicit position code with the original features of the neighboring points, and inputting the concatenated features into the second convolution block to obtain updated neighboring point features of the DLFA of the first local feature extraction stage module; Inputting the neighbor point features updated by the DLFA of the first local feature extraction stage module to the attention convergence layer of the DLFA of the first local feature extraction stage module to obtain the attention convergence layer output of the DLFA of the first local feature extraction stage module; Inputting the attention convergence layer output of the DLFA of the first local feature extraction stage module into the bias embedding block for encoding to obtain a display position code; Adding the attention convergence layer output of the DLFA of the first local feature extraction stage module and the display position code to obtain the key point features of the first local feature extraction stage module; The SLFE of the first local feature extraction stage module includes multiple layers of SLFE; The step of inputting the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module in S312 to obtain the output features of the first local feature extraction stage module includes: Obtaining neighboring points of the key points of the SLFE of the first local feature extraction stage module according to the key point features of the first local feature extraction stage module and the K nearest neighbor algorithm; Obtaining updated neighbor point features of the SLFE of the first local feature extraction stage module according to the neighbor points of the key points of the SLFE of the first local feature extraction stage module; Inputting the updated neighbor point features of the SLFE of the first local feature extraction stage module to the attention convergence layer of the SLFE of the first local feature extraction stage module to obtain the attention convergence layer output of the SLFE of the first local feature extraction stage module; Input the attention convergence layer output of the SLFE of the first local feature extraction stage module into the dilated multi-layer perceptron to obtain key point features with added semantic information; The key point features of the first local feature extraction stage module and the key point features after adding semantic information are added, and the added features are input into the batch normalization layer and the nonlinear activation layer to obtain the output features of the first local feature extraction stage module.

2. The method for 3D point cloud semantic segmentation with multi-domain attention enhancement according to claim 1, characterized in that: The Transformer encoder module includes multiple layers of Transformer; The step of inputting the encoder module key point features into the Transformer encoder module in S32 to obtain key point features integrating long-distance dependency features includes: S321, inputting the encoder module key point features into the first layer Transformer, dividing the encoder module key point features in coordinate space according to the window sampling algorithm, obtaining non-overlapping cubic windows, and obtaining a point set in each cubic window; S322, calculating the relative positions between the points in each cubic window according to the point set, and calculating the joint window multi-head self-attention feature of the fused neighborhood relative position encoding according to the relative positions; S323. Add the joint window multi-head self-attention features and the encoder module key point features to obtain key point features that integrate long-distance dependency features.

3. The method for 3D point cloud semantic segmentation with multi-domain attention enhancement according to claim 1, characterized in that: The training process of the 3D point cloud semantic segmentation model in S2 includes: S21, obtaining a point cloud data set, and performing data sampling on each point cloud file in the point cloud data set to obtain a sub-sampling point set; S22, performing data enhancement and data encapsulation on the sub-sampling point set to obtain a training data set; S23, training and optimizing the three-dimensional point cloud semantic segmentation model according to the training data set, the learning rate scheduler, the cross entropy loss and the optimizer to obtain a trained three-dimensional point cloud semantic segmentation model; The learning rate scheduler is used to preheat the learning to the maximum learning rate first, and then decay to the minimum learning rate using the cosine function; The cross entropy loss includes class weights and label smoothing.

4. A three-dimensional point cloud semantic segmentation device with multi-domain attention enhancement, the three-dimensional point cloud semantic segmentation device with multi-domain attention enhancement is used to implement the three-dimensional point cloud semantic segmentation method with multi-domain attention enhancement as claimed in any one of claims 1 to 3, characterized in that: The device comprises: An acquisition module, used for acquiring three-dimensional point cloud data to be segmented; An input module, used for inputting the three-dimensional point cloud data into a trained multi-domain attention enhanced three-dimensional point cloud semantic segmentation model; The three-dimensional point cloud semantic segmentation model includes a convolutional attention encoder module, a Transformer encoder module, a convolutional attention decoder module and a segmentation head module; An output module, used to obtain a 3D point cloud semantic segmentation result according to the 3D point cloud data and the 3D point cloud semantic segmentation model; The step of obtaining a 3D point cloud semantic segmentation result based on the 3D point cloud data and the 3D point cloud semantic segmentation model includes: S31, inputting the three-dimensional point cloud data into the convolutional attention encoder module to obtain the key point features of the encoder module; S32, inputting the key point features of the encoder module into the Transformer encoder module to obtain key point features that incorporate long-distance dependency features; S33, inputting the key point features fused with the long-distance dependency features into the convolutional attention decoder module to obtain a convolutional attention decoder output; S34, inputting the output of the convolutional attention decoder into the segmentation head module to obtain a three-dimensional point cloud semantic segmentation result; The convolutional attention encoder module includes a point embedding stage module, a first local feature extraction stage module, a second local feature extraction stage module and a third local feature extraction stage module; The first local feature extraction stage module, the second local feature extraction stage module and the third local feature extraction stage module respectively include a dense local feature aggregation module DLFA and a sparse local feature expansion module SLFE; The step of inputting the three-dimensional point cloud data into the convolutional attention encoder module in S31 to obtain the encoder module key point features includes: S311, inputting the three-dimensional point cloud data into the point embedding stage module, encoding the three-dimensional point cloud data, and obtaining point cloud features; S312, inputting the point cloud features into the DLFA of the first local feature extraction stage module to obtain the key point features of the first local feature extraction stage module, and inputting the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module to obtain the output features of the first local feature extraction stage module; S313, inputting the output features of the first local feature extraction stage module into the DLFA of the second local feature extraction stage module to obtain the key point features of the second local feature extraction stage module, and inputting the key point features of the second local feature extraction stage module into the SLFE of the second local feature extraction stage module to obtain the output features of the second local feature extraction stage module; S314, inputting the output features of the second local feature extraction stage module into the DLFA of the third local feature extraction stage module to obtain the key point features of the third local feature extraction stage module, and inputting the key point features of the third local feature extraction stage module into the SLFE of the third local feature extraction stage module to obtain the key point features of the encoder module; The step of inputting the point cloud features into the DLFA of the first local feature extraction stage module in S312 to obtain the key point features of the first local feature extraction stage module includes: Using the farthest point sampling algorithm, down-sampling the point cloud features to obtain the key points of the DLFA of the first local feature extraction stage module; Taking the key point of the DLFA of any first local feature extraction stage module as the center point, using the K nearest neighbor algorithm to calculate the neighboring points of the center point; According to the coordinates of the center point and the coordinates of the neighboring points, the neighborhood geometry related features are constructed; Using a first convolutional block, encoding the neighborhood geometry-related features to obtain an implicit position code; Concatenating the implicit position code with the original features of the neighboring points, and inputting the concatenated features into the second convolution block to obtain updated neighboring point features of the DLFA of the first local feature extraction stage module; Inputting the neighbor point features updated by the DLFA of the first local feature extraction stage module to the attention convergence layer of the DLFA of the first local feature extraction stage module to obtain the attention convergence layer output of the DLFA of the first local feature extraction stage module; Inputting the attention convergence layer output of the DLFA of the first local feature extraction stage module into the bias embedding block for encoding to obtain a display position code; Adding the attention convergence layer output of the DLFA of the first local feature extraction stage module and the display position code to obtain the key point features of the first local feature extraction stage module; The SLFE of the first local feature extraction stage module includes multiple layers of SLFE; The step of inputting the key point features of the first local feature extraction stage module into the SLFE of the first local feature extraction stage module in S312 to obtain the output features of the first local feature extraction stage module includes: Obtaining neighboring points of the key points of the SLFE of the first local feature extraction stage module according to the key point features of the first local feature extraction stage module and the K nearest neighbor algorithm; Obtaining updated neighbor point features of the SLFE of the first local feature extraction stage module according to the neighbor points of the key points of the SLFE of the first local feature extraction stage module; Inputting the updated neighbor point features of the SLFE of the first local feature extraction stage module to the attention convergence layer of the SLFE of the first local feature extraction stage module to obtain the attention convergence layer output of the SLFE of the first local feature extraction stage module; Input the attention convergence layer output of the SLFE of the first local feature extraction stage module into the dilated multi-layer perceptron to obtain key point features with added semantic information; The key point features of the first local feature extraction stage module and the key point features after adding semantic information are added, and the added features are input into the batch normalization layer and the nonlinear activation layer to obtain the output features of the first local feature extraction stage module.

5. A 3D point cloud semantic segmentation device, characterized in that: The three-dimensional point cloud semantic segmentation device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images