A point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid

Abstract

The present invention discloses a point cloud semantic segmentation model based on local surface distribution and continuous codec pyramid. The present invention includes the following steps: 1. Model establishment, designing a point cloud semantic segmentation model based on local surface distribution and continuous codec pyramid to enhance the spatial structural features of local points and alleviate the accuracy drop of shallow semantic features due to insufficient prediction ability. 2. Model training, first preprocessing the data set, and then training the established model. 3. Model inference, inputting the point cloud of the test set into the trained model, inferring its semantic category prediction value, and evaluating the prediction accuracy. The present invention is a neural network model for point cloud semantic segmentation, and in particular provides a unified modeling method based on local surface distribution to extract spatial structure information and continuous codec pyramid structure to optimize shallow semantic information, thereby achieving better segmentation effect in the field of point cloud semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid. Background Art

[0002] In recent years, with the rapid development of fields such as autonomous driving and robotics, 3D point cloud processing has garnered widespread attention from both academia and industry, and semantic segmentation based on point clouds has become a key task in computer vision. Point cloud semantic segmentation is a crucial technology that classifies different types of point cloud data in real-world environments at the point level, assigning the same category identifier to points with similar semantic information within the point cloud, thereby helping machines understand their environment. For example, an autonomous sweeping robot uses LiDAR to scan indoor scenes. Using semantic segmentation of point clouds helps the robot find targets, avoid obstacles, and understand its surroundings.

[0003] Traditional point cloud segmentation methods mainly use strictly manually designed features based on geometric constraints and statistical rules to group raw 3D points into non-overlapping regions that correspond to specific structures or objects in a scene. These methods can be divided into four categories: edge-based, region growing-based, model fitting-based, and clustering-based. However, the traditional point cloud segmentation process does not require supervised prior knowledge, so the results obtained do not have strong semantic information.

[0004] Compared with traditional point cloud segmentation methods, point cloud semantic segmentation methods based on deep learning have higher accuracy and robustness, avoiding the process of manually designing features and rules in traditional methods; deep learning methods can automatically learn hidden features and rules in data by training on large amounts of data, and can better handle complex scenes and noisy data; in addition, the point-based point cloud semantic segmentation method in deep learning methods realizes end-to-end input and output, avoiding the loss of feature information caused by the conversion of point clouds into other data types, and the implementation of point-based point cloud semantic segmentation methods is mainly through the extraction of semantic features through linear layer parameters between points, and with more and more work on the effective extraction of local area features of point clouds, compared with other types of deep learning point cloud semantic segmentation methods and traditional point cloud segmentation methods, the segmentation accuracy has been improved at a lower computational cost.

[0005] The current point cloud semantic segmentation methods based on point in deep learning are as follows: (1) Most of the work only uses the coordinates of the points as the input data of the neural network. This single data source usually requires the design of a complex network structure to extract effective semantic features, which undoubtedly increases the computational cost of the neural network; (2) Although the deep semantic features in the classic semantic segmentation structure Encoder-Decoder can improve the prediction accuracy by fusing the shallow semantic features, the introduced shallow features also restrict the further improvement of the accuracy due to insufficient prediction ability.

[0006] Therefore, it is necessary to design an efficient and easy-to-implement deep network model for point cloud semantic segmentation.

[0007] References

[0008] [1]Ruder S.An overview of gradient descent optimization algorithms[J].arXiv preprint arXiv:1609.04747,2016.

[0009] [2]Landrieu L,Simonovsky M.Large-scale point cloud semanticsegmentation with superpoint graphs[C]Proceedings of the IEEE conference oncomputer vision and pattern recognition.2018:4558-4567.

[0010] [3]Ran H, Liu J, Wang C.Surface representation for point clouds[C]Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:18942-18952.

[0011] [4]Zhao H, Jiang L, Jia J, et al. Point transformer [C] Proceedings of the IEEE / CVF international conference on computer vision. 2021: 16259-16268. Summary of the Invention

[0012] The present invention provides a point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid. The structural design is concise, efficient and easy to implement.

[0013] The technical solutions of the invention are as follows:

[0014] A point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid. The complete process includes data preprocessing, model establishment, model training and verification;

[0015] Data preprocessing mainly involves the given point cloud P = {P coord ,P normal}, corresponding to the true label sequence GT, constitutes the training set, providing the data basis for model training, where P coord With P normal Represents the coordinate field and normal vector (or RGB color) field of P.

[0016] When building the model, we analyze the point cloud semantic enhancement process to design specific submodules and construct an end-to-end semantic segmentation deep learning network model;

[0017] During model training and verification, the parameters of the established model are determined and the segmentation effect of the designed model is tested;

[0018] The point cloud semantic segmentation model based on feature-enhanced position attention mechanism is developed in the following steps:

[0019] Step 1: Model building;

[0020] Step 2: Model training;

[0021] Step 3: Model inference.

[0022] In step 1, a point cloud semantic segmentation model based on local surface distribution and continuous codec pyramid is designed. The spatial structural features of local points are enhanced by the local surface distribution features constructed by local neighborhood eigenvalues ​​and eigenvectors. The two continuous codec pyramids are used to alleviate the accuracy loss caused by the insufficient prediction ability of shallow semantic features, thereby improving the accuracy of point cloud semantic segmentation.

[0023] First, a local feature analysis module is constructed to perform preliminary feature extraction on the original point cloud features to strengthen the local relationship of the point cloud, and the distribution feature F of the local point cloud is extracted through the local surface distribution submodule. l , use the Umbrella RepSurf [3] submodule to fit the local surface as the surface feature F u , F={F l , Fu , P a} as the initial extraction stage features, where P a Represents the preprocessed point cloud;

[0024] Then, the codec module is constructed, and the continuous codec pyramid submodule is established to extract features from the multi-layer continuous pyramid of feature F to obtain point cloud features E with enhanced semantic information.

[0025] Finally, the output module is established, and the point cloud features E are further fused and extracted by using the linear layer Linear to obtain the predicted features Z of the point cloud;

[0026] In step 2, the provided point cloud is first preprocessed. Let P be the point cloud data. First, voxelization is performed, and then the minimum value is normalized, random scaling, random rotation with the z axis unchanged, random Gaussian noise, random horizontal rotation, and random spatial movement are applied. If RGB information exists, the color contrast is enhanced, offset, and dithered. After the above data preprocessing, the processed point cloud data P is obtained. a ;

[0027] Then train the model established in step 1 and transform the point cloud P a The point cloud feature E is obtained through the Local Surface Distribution submodule, the Umbrella RepSurf [3] submodule, and the Continuous Codec Pyramid submodule, and the predicted value Z is obtained after passing through the output module. The predicted value and the true value GT are cross-entropy calculated to measure the error between the predicted value and the true value; the network model parameters defined in step 1 are iteratively optimized and trained through the back-propagation algorithm until the entire model converges;

[0028] In step 3, the point cloud of the test set is input into the trained model, the semantic category prediction value of the test point cloud is inferred, and the average category intersection over union (mIoU), average category accuracy (mAce), and overall accuracy (OA) of the test prediction value and the true value are calculated to evaluate the prediction accuracy.

[0029] The specific process of building the model described in step 1 is as follows:

[0030] First, construct a local feature analysis module, such as Figure 2As shown in the figure, this module consists of the Local Surface Distribution submodule, the Umbrella RepSurf submodule, and the feature concatenation operation (Concat). The model establishment and specific operation process are as follows:

[0031] (1) Establishment of the Local Surface Distribution submodule. First, a of The K-nearest neighbor (KNN) algorithm is used to search for the index sequence of the k points closest to each point in the point cloud, where P a ∈R N×6 , Indicates P a Coordinate field, N represents the number of point clouds, and P is extracted using the index sequence a The coordinate field of the feature, denoted as D, D∈R N×k×3 , the second dimension is called the grouping dimension, and then the coordinate field D is standardized. The processing process is as follows:

[0032]

[0033] in, Indicates the averaging operation on the data in the grouping dimension; perform SVD decomposition on the M matrix, and the decomposition formula is as follows:

[0034] M=USV T (2)

[0035] Among them, S is a singular matrix, which is a diagonal matrix composed of singular values, U is a left singular matrix, V is a right singular matrix, U and V are both orthogonal matrices (both modulo 1), and the superscript T represents the transpose of the matrix. Derivation of the above formula yields:

[0036] M T M=VLV T =VLV -1 (3)

[0037] Among them, M T M is the scatter matrix of the local point cloud, and L is M T The eigenvalue matrix of the M matrix, the column vector of V is M r Eigenvectors of the M matrix, L∈R N×3×3 , V∈R N×3×3 , the singular value matrix S can be expressed by L as:

[0038] L=SS (4)

[0039] Since the calculated eigenvalues ​​are generally small, singular values ​​are used instead of eigenvalues. In order to prevent the values ​​of the singular value matrix S from being too large or too small, which leads to unstable training, the absolute value and square root operations are performed on the singular values, and the features after the operation are recorded as S'; the vectors in the eigenvector matrix V can point to either positive or negative, so a random inversion operation is performed on it, and the eigenvector matrix after random inversion is recorded as V'. Then, Hadamard product is performed on S' and V' to obtain a set of basis vectors that can describe the distribution of local point clouds. Finally, the basis vectors are extracted through neural networks. The above process can be described by the following formula:

[0040]

[0041] Among them, ξ(.) represents the random reverse flipping of the eigenvector direction, δ(S′) represents the dimension increase of S′, ⊙ represents the Hadamard product processing data, and h Θ (.) is a nonlinear function with Θ as the parameter to be learned, and the output feature

[0042] (2) Establishment of the Umbrella RepSurf submodule [3]

[0043] like Figure 2 As shown, the Umbrella RepSurf submodule [3] inputs the feature P a Then extract the surface representation feature F of the point cloud u ,

[0044] (3) Feature concatenation operation (Concat)

[0045] For feature F l 、F u 、P a After concatenation in the feature dimension (second dimension), we obtain feature F. The operation process is:

[0046] F=[F l , F u , P a ]2 (9)

[0047] Where “[]2” indicates the concatenation operation in the second dimension;

[0048] Secondly, we construct the codec module, which consists of only the Continuous CodecPyramid submodule and uses PointTransformer[4] as the basic network baseline. Figure 3As shown in the figure, the structure contains four basic structures, namely the downsampling structure Transition Down, the upsampling structure Transition UP, the basic feature extraction structure PointTransformer Blcok and the global feature fusion structure Global Feature Fusion. The above basic structures are used to construct the continuous codec pyramid submodule. The specific process is as follows:

[0049] (1) Establishment of infrastructure

[0050] ①PointTransformer Blcok:such as Figure 5 As shown, the input feature of PointTransofrmer Block is recorded as feat, Among them C in Represents the number of input channels of the point cloud, posi represents the spatial coordinates of the corresponding feat, posi∈R N×3 ; PointTransofrmer Block first uses the linear layer Linear to extract further point features. The specific process is expressed as follows:

[0051]

[0052] in Indicates that the output feature dimension is C ou Linear layer Linear, output feature Then pass n PointTransformer Layer, n defaults to 2, PointTransformerLayer structure is as follows Figure 6 As shown in Figure 1, this structure is a vector self-attention mechanism structure. Its operation process is as follows: the KNN algorithm is used to select the k=16 points closest to each point to form a group, and the points in the group are enhanced by the vector self-attention of the spatial position to obtain the feature feat" after local feature enhancement. The PointTransformer Layer operation is as follows:

[0053] feat′ group =KNN(feat′, k) (11)

[0054]

[0055]

[0056]

[0057]

[0058] Y = Softmax(K-Q+posi′) (16)

[0059]

[0060] Unsqueeze means to expand the specified dimension; the calculation formula of Softmax is:

[0061]

[0062] Finally, a set of linear layers is used to extract features again for the feature feat, and then fused with the feature feat in an additive manner to form a residual form to reduce the impact of gradient disappearance. The specific process is expressed as follows:

[0063]

[0064] Its output characteristics are So far, the PointTransformer Blcok structure is completed;

[0065] ②Transition Down:such as Figure 7 As shown in (a), the input feature of the downsampling structure Transition Down is recorded as feat. posi represents the spatial coordinates of the corresponding feat, posi∈R N×3 ; First, use the Farthest Point Sampling (FPS) algorithm to find the farthest S in the space d points as sampling points to obtain the features after sampling Then use the KNN algorithm to find the k points closest to each point to form a group and obtain the features Use the linear layer Linear to extract features for the points in the group as feat″, The specific process is as follows:

[0066]

[0067] Then, for the features MaxPooling is used to select the maximum feature point in the group as the set feature. The specific process is as follows:

[0068] feat″=MaxPooling(feat″ group ) (twenty one)

[0069] in, The Transition Down sampling structure is now completed;

[0070] ③Transition UP: Figure 7 As shown in (b), the upsampling structure has two inputs. The first is the input feature from the upper Decoder structure, denoted as feat1. Where S1 represents the number of points before sampling, posi1 represents the spatial coordinates corresponding to feat1, The second is the input feature from the Encoder at the same level, recorded as feat2, Where S2 represents the number of points after sampling, posi2 represents the spatial coordinates corresponding to feat2, The upsampling structure Transition UP first uses a set of linear layers to extract the features of feat1, and then uses the trilinear interpolation algorithm to upsample. The algorithm process is: first use the linear layer Linear to extract the feature feat1, and the feature is obtained Then, use the KNN algorithm to search for the three points closest to posi1 in posi2 and group them. The coordinates of the group after the search are marked as posi′2. Get the upsampled features based on the distance between the three points as the feature weight The specific process is as follows:

[0071]

[0072]

[0073]

[0074] where dist i Represents the distance from the i-th point among the three nearest points Then use the linear layer to extract the feature feat2 and add it to feat″1. The operation process is as follows:

[0075]

[0076] in To output features, the Transition UP upsampling structure is completed.

[0077] ④Global Feature Fusion: Figure 7 As shown in (c), the input feature of the global feature fusion structure is denoted as feat, First, the global average feature is obtained in the number dimension (first dimension) of the point cloud by averaging, and then the global average feature is further extracted using the linear layer Linear. The extracted feature is recorded as The operation process is as follows:

[0078]

[0079] Then, concatenate feat′ with each point in feat in the feature dimension (second dimension), and record it as The operation process is:

[0080] feat″=[feat′, feat]2 (27)

[0081] Where "[]2" represents the concatenation operation in the second dimension; finally, a set of linear layers are used to fuse the point features with the global average features, so that each point has the ability to observe the global features:

[0082]

[0083] in Represents the output features of the Global Feature Fusion global feature fusion structure, thus completing the Global Feature Fusion global feature fusion structure;

[0084] (2) Establishment of the Continuous Codec Pyramid submodule

[0085] like Figure 3 As shown in the figure, the network is divided into grids, P0~P5 represent the levels of the submodule, Enc1 and Dec1 represent the encoding and decoding process of the first encoding and decoding pyramid, Enc2 and Dec2 represent the encoding and decoding process of the first encoding and decoding pyramid, the submodule inputs the feature F, and uses two sets of encoding and decoding pyramid structures to extract features to alleviate the adverse effects of low-level information in shallow features on prediction; in terms of structural details, the features in Dec1 are fused into Enc2 of the same layer in a jump connection manner, and after multiple feature fusions and feature extractions to deepen the prediction ability of shallow features, the features in Enc2 are fused into Dec2 of the same layer, and shallow features with certain prediction capabilities are used to assist deep features, and finally the output feature E is obtained. The following formula shows the calculation process of the continuous encoding and decoding pyramid Enc2 and Dec2 structures:

[0086] ①The first coding pyramid structure operation process is as follows:

[0087]

[0088] ②The second coding pyramid structure operation process is as follows:

[0089]

[0090] Among them, P i Represents the output feature of the i-th level of Enc (Encoder) or Dec (Decoder), Linear represents the linear layer, Tu and Td represent Transition UP and Transition Down, Gff represents the Global Feature Fusion structure, and Ptb represents the PointTransformer Blcok structure; As the output feature of the continuous encoding pyramid;

[0091] Finally, if Figure 4 As shown, two sets of linear layers are used to extract the input features E and the number of channels is adjusted to the specified number of categories n. cls The specific process is as follows:

[0092]

[0093] Where Z represents the output feature of the output module, This completes the model building operation.

[0094] The specific process of model training described in step 2 is as follows:

[0095] Input point cloud P, preprocess the point cloud data to obtain point cloud P a , extract the distribution features F of the local point cloud through the Local Surface Distribution submodule l , use the UmbrellaRepSurf[3] submodule to fit the local surface as the surface feature F u , F={F l , F u , P a The local feature analysis module outputs the feature, which is then input into the Continuous Codec Pyramid submodule in the codec module to obtain the output feature E. Finally, the output head module outputs the predicted value Z, and the cross entropy loss function is used to calculate the loss between the predicted value and the true value GT. The loss function is as follows:

[0096]

[0097] Where C represents the number of categories, and the SGD[1] optimizer is used to optimize the function Loss to minimize the loss value.

[0098] The specific process of model reasoning in step 3 is as follows:

[0099] After the model is trained in step 2, the model parameters are fixed, the point cloud of the test set is input to obtain the test prediction value, the semantic category prediction value of the test point cloud point is inferred, and the mIoU, mAcc, and OA of the test prediction value and the true value are calculated until all test sets are tested and the final prediction accuracy is given.

[0100] The beneficial effects of the present invention are as follows:

[0101] The method proposed in this paper improves the accuracy of point cloud semantic segmentation compared with other algorithms, as shown in the following aspects:

[0102] 1) Compared with some previous network models, the model proposed in this invention is easier to implement;

[0103] 2) The present invention effectively extracts the distribution characteristics of the original point cloud data, and has a strong response to segmentation edges and continuous large point cloud areas, thereby improving the prediction accuracy of edge areas and continuous large areas;

[0104] 3) This invention improves the feature structure of point clouds from a macroscopic perspective, reduces the adverse effects of shallow features on prediction, and achieves higher accuracy than previous models in point cloud semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 This is a flow chart of the model processing of the present invention.

[0106] Figure 2 This is a diagram of the local feature analysis module of the present invention.

[0107] Figure 3 This is a diagram of the encoding and decoding module of the present invention.

[0108] Figure 4 This is the output module diagram of the present invention.

[0109] Figure 5 This is the structure diagram of the PointTransformer Block of the present invention.

[0110] Figure 6 This is the PointTransformer Layer structure diagram of the present invention.

[0111] Figure 7 (a) is a diagram of the downsampling structure Transition Down of the present invention.

[0112] Figure 7 (b) is a structural diagram of the upsampling structure Transition UP of the present invention.

[0113] Figure 7 (c)Global Feature Fusion structure diagram.

[0114] Figure 8 This is a visualization result diagram of the model experiment of the present invention. DETAILED DESCRIPTION

[0115] In order to make the purpose and technical solution of the present invention more clearly understood, the application principle of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments. However, the protection scope of the present invention is not limited to the following specific embodiments.

[0116] Unless otherwise defined, all technical terms used hereinafter have the same meanings as those generally understood by those skilled in the art. The technical terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention.

[0117] Example 1:

[0118] like Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 As shown, the present invention provides a point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid. The specific steps are as follows:

[0119] The model establishment described in step 1 is as follows:

[0120] This invention uses the S3DIS (Stanford 3D Indoor Scene Dataset) [2] dataset, which contains 6 teaching and office areas, 271 rooms, and a total of 695,878,620 3D points with color information and semantic labels. Each point is annotated as one of 13 semantic categories, where areas 1 to 4 and 6 are used as training sets and area 5 is used as a test set. The complete processing flow of the model is as follows: Figure 1 As shown;

[0121] First, the point cloud is preprocessed. The original point cloud data P is voxelized with a 4 cm grid. Then, each data is normalized using the minimum value of the spatial coordinate, a random scaling with a scaling ratio of 0.2, a random rotation with the z-axis unchanged, a Gaussian noise with a standard deviation of 0.01, and a spatial random translation with a maximum value of 0.1 cm. The output point cloud P is obtained. a , P a ∈RN×6 ;

[0122] Secondly, construct a local feature analysis module, such as Figure 2 As shown in the figure, the module consists of the local surface distribution (LocalSurface Distribution) submodule, the umbrella surface representation (UmbrellaRepSurf) submodule and the feature splicing operation (Concat);

[0123] (1) First, the local surface distribution submodule performs the preprocessing of the point cloud P a Perform feature extraction and input the preprocessed point cloud P a Coordinate fields Output local surface distribution characteristics of point cloud The specific operation process of the Local Surface Distribution submodule is as follows:

[0124] ①For input features First, use the K nearest neighbor algorithm to obtain the features D∈R of the k=16 points closest to each point. N×k×3 ;

[0125] ② Standardize D by subtracting the mean of D on the x, y, and z axes. The processed feature is recorded as M∈R N ×k×3 ;

[0126] ③Use SVD decomposition on M to obtain M T The singular value matrix S∈R of the M matrix N×3×3 and the eigenvector matrix V∈R N ×3×3 , used to estimate the spatial distribution characteristics of local point clouds;

[0127] ④ Perform absolute value and square root processing on the singular value matrix S to prevent the singular value matrix S from being too large or too small, which leads to unstable training. The processed singular value matrix is ​​recorded as S′∈R N×3×3 ;

[0128] ⑤ Perform a random inversion operation on the eigenvector matrix V. Since the direction of the eigenvector matrix V can be positive or negative, the eigenvector matrix after random inversion is recorded as V′;

[0129] ⑥ Perform Hadamard product on S′ and V′ to obtain a set of basis vectors that can describe the distribution of local point cloud, and finally pass the basis vectors through the linear layer Linear output feature

[0130] (2) Then, the Umbrella RepSurf[3] submodule inputs the feature P a Then extract the surface representation feature F of the point cloud u ,

[0131] (3) For feature F l 、F u 、P a After concatenation in the feature dimension (second dimension), we obtain feature F.

[0132] Then build the codec module, such as Figure 3 As shown, the Continuous CodecPyramid submodule encodes and decodes the input feature F and outputs the predicted feature Among them, the operation process of the first encoding pyramid structure is as follows:

[0133]

[0134] The second encoding pyramid structure operation process is as follows:

[0135]

[0136] Among them, P i Represents the output feature of the i-th level of Enc (Encoder) or Dec (Decoder), Linear represents the linear layer, Tu and Td represent Transition UP and Transition Down, Gff represents the Global Feature Fusion structure, and Ptb represents the PointTransformer Blcok structure; As the output feature of the continuous encoding pyramid;

[0137] Finally, if Figure 4 As shown, two sets of linear layers are used to extract the input features E and the number of channels is adjusted to the specified number of categories n. cls The specific process is as follows:

[0138]

[0139] Where Z represents the output feature of the output module,

[0140] This completes the model building operation.

[0141] The model training described in step 2 is as follows:

[0142] Train the model established in step 1, input point cloud P, and obtain point cloud P after preprocessing the point cloud data. a , extract the distribution features F of the local point cloud through the Local Surface Distribution submodule l , use the Umbrella RepSurf [3] submodule to fit the local surface as the surface feature F u , F={F l , F u , P a The local feature analysis module outputs the feature, which is then input into the Continuous Codec Pyramid submodule in the codec module to obtain the output feature E. Finally, the output module outputs the predicted value Z, which is compared with the true label GT of the dataset. The loss value between the predicted value and the true value is calculated through the defined loss function, and then the parameter values ​​of the entire network are adjusted according to the loss value until the network converges.

[0143] This completes the model training operation.

[0144] The model reasoning described in step 3 is as follows:

[0145] After the model is trained in step 2, the model parameters are fixed, the point cloud of the test set is input to obtain the test prediction value, the semantic category prediction value of the test point cloud is inferred, the average intersection-over-union ratio of the test prediction value to the true value is calculated, and the prediction accuracy is evaluated;

[0146] This completes the model inference operation.

[0147] Table 1 shows the accuracy of the method proposed in this paper in S3DIS. PointTransformer[4] is the baseline of this model, Ours is the deep model provided by this paper, and “*” indicates the result of reproducing the original model.

[0148] / Col. / Win. / Tab. / Cha. / Boo. / Boa. / Clu. represent categories Ceiling / Floor / Beam / Column / Window respectively

[0149] / Table / Chair / Bookcase / Board / Clutter, using the average category intersection over union (mIoU), average category accuracy (mAcc), and overall accuracy (OA) to evaluate the average accuracy of point cloud semantic segmentation. As shown in Table 1, our model is 2.0% / 1.7% / 0.6% higher than the baseline in OA / mAcc / mIoU (the actual test is 2.4% / 2.4% / 0.8%). The visualization of the test is shown in the figure below. Figure 8 shown.

[0150] Table 1

[0151]

Claims

1. A point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid, characterized by The following steps are involved: Step 1: Model building; Specifically, a point cloud segmentation method based on local surface distribution and continuous encoding and decoding is designed. The spatial structural features of local points are enhanced by the local surface distribution features constructed by local neighborhood eigenvalues ​​and eigenvectors. Two continuous encoding and decoding pyramids are used to alleviate the accuracy drop caused by the insufficient prediction ability of shallow semantic features, thereby improving the accuracy of point cloud semantic segmentation. First, a local feature analysis module is constructed to perform preliminary feature extraction on the original point cloud features to strengthen the local relationship of the point cloud, and the distribution feature F of the local point cloud is extracted through the local surface distribution submodule. l , use the umbrella surface representation submodule to fit the local surface as the surface feature F u , F={F l , F u , P a } as the initial extraction stage features, where P a Represents the preprocessed point cloud; Then, the encoding and decoding module is constructed, and the continuous encoding and decoding pyramid submodule is established to extract features from the multi-layer continuous pyramid of feature F to obtain point cloud features E with enhanced semantic information; Finally, the output module is established to further fuse and extract the point cloud features E by using the linear layer parameters to obtain the predicted features Z of the point cloud; The establishment of the local surface distribution is characterized by including the following operations: First, the preprocessed point cloud P a The specific operation process of the local surface distribution submodule is as follows: First, P a Coordinates Use the K nearest neighbor algorithm to search for the index sequence of the k points closest to each point in the point cloud, where P a ∈R N×6 , Indicates P a Coordinate field, N represents the number of point clouds, and P is extracted using the index sequence a The coordinate field of the feature, denoted as D, D∈R N×k×3 , the second dimension is called the grouping dimension, and then the coordinate field D is standardized. The processing process is as follows: in, Indicates the averaging operation on the data in the grouping dimension; perform SVD decomposition on the M matrix, and the decomposition formula is as follows: M=USV T (2) Among them, S is a singular matrix, which is a diagonal matrix composed of singular values, U is a left singular matrix, V is a right singular matrix, U and V are both orthogonal matrices, and the superscript T represents the transpose of the matrix. Derivation of the above formula yields: M T M=VLV T =VLV -1 (3) Among them, M T M is the scatter matrix of the local point cloud, and L is M T The eigenvalue matrix of the M matrix, the column vector of V is M T Eigenvectors of the M matrix, L∈R N×3×3 , V∈R N×3×3 , the singular value matrix S can be expressed by L as: L=SS (4) Where S∈R N×3×3 is the singular value; since the calculated eigenvalue is generally small, the singular value is used instead of the eigenvalue. In order to prevent the value of the singular value matrix S from being too large or too small, which leads to unstable training, the absolute value and square root operations are performed on the singular value, and the features after the operation are recorded as S'; the vector direction in the eigenvector matrix V can be positive or negative, so it is randomly inverted, and the eigenvector matrix after random inversion is recorded as V', and then S' and V' are Hadamard-producted to obtain a set of basis vectors that can describe the distribution of the local point cloud. Finally, the basis vectors are extracted through the neural network. The above process can be described by the following formula: Among them, ξ(.) represents the random reverse flipping of the eigenvector direction, δ(S′) represents the dimension increase of S′, ⊙ represents the Hadamard product processing data, and h Θ (.) is a nonlinear function with Θ as the parameter to be learned, and the output feature The submodule for establishing a continuous encoding and decoding pyramid is characterized by including the following operations: PointTransformer is used as the basic network Baseline. The structure contains four basic structures, namely the downsampling structure Transition Down, the upsampling structure Transition UP, the basic feature extraction structure PointTransformer Blcok and the global feature fusion structure Global Feature Fusion. The network is divided into grids. P0~P5 represent the levels of the submodule. Enc1 and Dec1 represent the encoding and decoding process of the first encoding and decoding pyramid. Enc2 and Dec2 represent the encoding and decoding process of the first encoding and decoding pyramid. The submodule inputs the feature F and uses a jump connection to fuse the features in Dec1 into the Enc2 of the same layer. After multiple feature fusions and feature extractions to deepen the prediction ability of shallow features, the features in Enc2 are fused into the Dec2 of the same layer. Shallow features with certain prediction capabilities are used to assist deep features, and finally the output feature E is obtained. The following formula shows the calculation process of the continuous encoding and decoding pyramid Enc2 and Dec2 structures: ①The first coding pyramid operation process is as follows: ②The second coding pyramid operation process is as follows: Among them, P i Represents the output feature of the i-th level of Enc (Encoder) or Dec (Decoder), Linear represents the linear layer, Tu and Td represent Transition UP and Transition Down, Gff represents the Global Feature Fusion structure, and Ptb represents the PointTransformer Blcok structure; As the output feature of the continuous encoding pyramid; Step 2: Model training; Train the network model parameters defined in step 1 until the entire model converges; Step 3: Model inference; Input the point cloud of the test set into the trained model, infer the semantic category prediction value of the test point cloud, and evaluate the prediction accuracy.

2. The point cloud semantic segmentation model based on local surface distribution and continuous encoding and decoding pyramid according to claim 1 is characterized in that The model training described in step 2 is as follows: Input point cloud P, preprocess the point cloud data to obtain point cloud P a , extract the distribution features F of the local point cloud through the local surface distribution submodule l , use the umbrella surface representation submodule to fit the local surface as the surface feature F u , F={F l , F u , P a } is used as the output feature of the local feature analysis module, and then input into the continuous encoding and decoding pyramid submodule in the encoding and decoding module to obtain the output feature E. Finally, the predicted value Z is output through the output module, and the cross entropy loss function is used to calculate the loss between the predicted value and the true value GT. The loss function is as follows: Among them, C represents the number of categories, and the SGD optimizer is used to optimize the function Loss to minimize the loss value.

Images