Three-dimensional point cloud data classification method and device based on multiple scales, terminal equipment and storage medium

Abstract

The invention discloses a multi-scale-based three-dimensional point cloud data classification method and apparatus, a terminal device and a storage medium. The method comprises the steps of obtaining three-dimensional point cloud data of a to-be-tested transformer substation; carrying out monomer extraction on the three-dimensional point cloud data to obtain monomer point cloud data of a plurality of monomer devices; performing multi-scale feature extraction on each single point cloud data to obtain a multi-scale feature pyramid structure corresponding to each single point cloud data; performing local scale feature fusion on each multi-scale feature pyramid structure to obtain a point feature expression corresponding to each multi-scale feature pyramid structure; performing cross-scale feature fusion on the feature expression of each point to obtain multi-scale geometric features; and classifying the multi-scale geometric features corresponding to each piece of three-dimensional point cloud data to obtain a classification result corresponding to each piece of three-dimensional point cloud data. Compared with the prior art that deep learning is simply carried out based on label marking, the method improves the recognition accuracy of the types of the complicated three-dimensional point cloud equipment in the complicated scene of the transformer substation.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional processing technology, and in particular to a multi-scale based three-dimensional point cloud data classification method, device, terminal equipment and storage medium. Background Art

[0002] With the rapid development of laser scanning technology, SLAM mapping, and photogrammetry technology, 3D point cloud data has been widely used in production scenarios such as autonomous driving, digital twins, and slope monitoring. Since point cloud acquisition equipment has a limited range of environmental perception, and the quality of point clouds collected by different devices is limited by hardware and physical factors and has different forms of defects, in order to obtain a large-scale fused point cloud that meets practical requirements, it is usually necessary to align and fuse the 3D point clouds of multiple measuring stations to obtain a large-scene point cloud with geometric consistency. However, the existing technology simply performs deep learning on 3D point clouds by marking categories, and is unable to mine the deep features of 3D point clouds, resulting in low accuracy in the category recognition of complex 3D point cloud devices in complex substation scenes.

[0003] Therefore, a 3D point cloud data classification strategy is urgently needed to solve the problem of low accuracy in category recognition of complex 3D point cloud devices in complex substation scenes. Summary of the invention

[0004] The embodiments of the present invention provide a multi-scale three-dimensional point cloud data classification method, apparatus, terminal device and storage medium to solve the problem of low accuracy in category recognition of complex three-dimensional point cloud devices in complex substation scenes.

[0005] In order to solve the above problems, an embodiment of the present invention provides a multi-scale three-dimensional point cloud data classification method based on multi-scale feature expression and cross-scale cross-attention mechanism, including:

[0006] Acquire three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes a plurality of single devices;

[0007] Extracting the three-dimensional point cloud data in a monomeric manner to obtain monomer point cloud data of a plurality of monomer devices;

[0008] Performing multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data;

[0009] Performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures;

[0010] Performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions;

[0011] The multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain a classification result corresponding to each of the three-dimensional point cloud data.

[0012] As an improvement of the above solution, the multi-scale feature extraction is performed on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data, including:

[0013] Based on the farthest point sampling technology and K nearest neighbor aggregation, multi-scale downsampling is performed on each of the monomer point cloud data at a preset downsampling rate to obtain a multi-scale point cloud distribution structure corresponding to each of the monomer point cloud data; wherein the multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, and the three-dimensional point clouds in the point cloud distribution structure corresponding to each scale have the same shape and different numbers of three-dimensional point cloud points;

[0014] In each of the multi-scale point cloud distribution structures, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain local geometric features corresponding to each scale, and the local geometric features corresponding to all scales are aggregated to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data.

[0015] As an improvement of the above solution, the step of fusing local scale features on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures includes:

[0016] In each of the multi-scale feature pyramid structures, within the feature pyramid structure corresponding to each scale, a feature expression generation operation is performed on each point until all points perform the feature expression generation operation, and the local feature expression generated for each point is summarized to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures;

[0017] Among them, the feature expression generation operation is specifically as follows:

[0018] Generate query point features corresponding to the current point, and generate a query vector based on the query point features and the learnable weight matrix;

[0019] Determine neighbor points based on the coordinates of the current point;

[0020] Generate neighbor point features corresponding to the neighbor points, and generate a keyword matrix based on the neighbor point features and the learnable weight matrix;

[0021] Based on the keyword matrix and query vector, a local feature expression of the current point is generated.

[0022] As an improvement of the above solution, the local feature expression of the current point is generated based on the keyword matrix and the query vector, including:

[0023] The query vector, the keyword matrix, the first learnable weight matrix and the second learnable weight matrix are input into the first point-by-point difference formula to calculate the first attention matrix from the neighbor point to the current point; wherein the first point-by-point difference formula is specifically:

[0024]

[0025] In the formula, A ij is the weight value corresponding to an element in the first attention matrix from the neighbor point to the current point, f i is the query point feature, f j is the neighbor point feature, W query is the first learnable weight matrix, W key is the second learnable weight matrix;

[0026] The first attention matrix, the neighbor point features and the third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature expression of the current point; wherein the first matrix multiplication formula is specifically:

[0027]

[0028] In the formula, f i ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W value is the third learnable weight matrix.

[0029] As an improvement of the above solution, performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions includes:

[0030] In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale, and a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale based on the seed point to obtain neighboring points in each feature pyramid structure corresponding to the non-highest scale;

[0031] In each of the multi-scale feature pyramid structures, a cross-scale fusion feature of the seed point is calculated based on the feature of the seed point and the feature of each neighboring point;

[0032] In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global maximum pooling layer to obtain the multi-scale geometric features corresponding to the point feature expression.

[0033] As an improvement of the above solution, the cross-scale fusion feature of the seed point is calculated based on the feature of the seed point and the feature of each neighboring point, including:

[0034] Substitute the feature corresponding to the seed point and the feature of each neighboring point into the second point-by-point difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to the non-highest scale; wherein the second point-by-point difference formula is specifically:

[0035]

[0036] In the formula, A kl is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point, f k is the feature corresponding to the seed point, f l is the feature of the neighboring point, W′ query is the fourth learnable weight matrix, W′ key is the fifth learnable weight matrix;

[0037] Substitute the second attention matrix of the same scale and the features of the adjacent points into the second matrix multiplication formula, calculate the local feature expression of the seed point at each non-highest scale, and concatenate the local feature expression of the seed point at each non-highest scale to obtain the cross-scale fusion feature of the seed point; wherein, the second matrix multiplication formula is specifically:

[0038]

[0039] In the formula, f k ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W′ value is the sixth learnable weight matrix.

[0040] As an improvement of the above solution, the multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain the classification results corresponding to each of the three-dimensional point cloud data, including:

[0041] In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain a classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output, and a multi-layer perceptron layer with a maximum pooling layer is trained.

[0042] Correspondingly, an embodiment of the present invention further provides a multi-scale based three-dimensional point cloud data classification device, comprising: a data acquisition module, a monomer extraction module, a multi-scale extraction module, a feature fusion module, a cross-scale feature fusion module and a classification module;

[0043] The data acquisition module is used to acquire three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes a plurality of single devices;

[0044] The monomer extraction module is used to perform monomer extraction on the three-dimensional point cloud data to obtain monomer point cloud data of several monomer devices;

[0045] The multi-scale extraction module is used to perform multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data;

[0046] The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures;

[0047] The cross-scale feature fusion module is used to perform cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions;

[0048] The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification results corresponding to each of the three-dimensional point cloud data.

[0049] Correspondingly, an embodiment of the present invention further provides a computer terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a multi-scale based three-dimensional point cloud data classification method as described in the present invention.

[0050] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a multi-scale three-dimensional point cloud data classification method as described in the present invention.

[0051] As can be seen from the above, the present invention has the following beneficial effects:

[0052] The present invention provides a multi-scale three-dimensional point cloud data classification method based on multi-scale feature expression and cross-scale cross-attention mechanism, and obtains the three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes several monomer devices; the three-dimensional point cloud data is subjected to monomer extraction to obtain the monomer point cloud data of several monomer devices; multi-scale feature extraction is performed on each of the monomer point cloud data to obtain the multi-scale feature pyramid structure corresponding to each of the monomer point cloud data; local scale feature fusion is performed on each of the multi-scale feature pyramid structures to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures; cross-scale feature fusion is performed on each of the point feature expressions to obtain the multi-scale geometric features corresponding to each of the point feature expressions; the multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain the classification results corresponding to each of the three-dimensional point cloud data. The present invention extracts three-dimensional point clouds from the monomer devices of the substation to be tested, extracts multi-scale features from the three-dimensional point cloud data corresponding to the monomer devices, and performs local scale feature fusion to obtain multi-scale geometric features, and the present invention can extract feature information at different scales of the monomer devices. Compared with the prior art that simply performs deep learning based on label marking, the present invention can discover more feature information of the three-dimensional point cloud data of a single device, so that accurate category classification can be performed based on specific feature information during classification, thereby improving the recognition accuracy of complex three-dimensional point cloud device categories in complex substation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a multi-scale three-dimensional point cloud data classification method provided by an embodiment of the present invention;

[0054] Figure 2 is a schematic structural diagram of a multi-scale three-dimensional point cloud data classification device provided by an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a process of multi-scale feature extraction provided by an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of a process of local scale feature fusion provided by an embodiment of the present invention;

[0058] Figure 6 is a schematic diagram of a process of cross-scale feature fusion provided by an embodiment of the present invention;

[0059] Figure 7 It is a flowchart of a multi-scale three-dimensional point cloud data classification method provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Embodiment 1

[0062] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a multi-scale three-dimensional point cloud data classification method provided by an embodiment of the present invention. Figure 1 As shown, this embodiment includes steps 101 to 106, and each step is specifically as follows:

[0063] Step 101: Acquire three-dimensional point cloud data of a substation to be tested; wherein the substation to be tested includes a plurality of single devices.

[0064] In this embodiment, the three-dimensional point cloud data of the substation to be tested can be retrieved from a network three-dimensional resource library, or the three-dimensional point cloud data of the substation to be tested can be collected and generated based on a three-dimensional camera.

[0065] Step 102: extract the three-dimensional point cloud data in a monomeric manner to obtain monomer point cloud data of a plurality of monomer devices.

[0066] In this embodiment, this embodiment adopts the complex scene point cloud data monomer object automatic extraction technology to the three-dimensional point cloud data of the substation scene to be tested.

[0067] Step 103: performing multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data.

[0068] In this embodiment, the multi-scale feature extraction is performed on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data, including:

[0069] Based on the farthest point sampling technology and K nearest neighbor aggregation, multi-scale downsampling is performed on each of the monomer point cloud data at a preset downsampling rate to obtain a multi-scale point cloud distribution structure corresponding to each of the monomer point cloud data; wherein the multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, and the three-dimensional point clouds in the point cloud distribution structure corresponding to each scale have the same shape and different numbers of three-dimensional point cloud points;

[0070] In each of the multi-scale point cloud distribution structures, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain local geometric features corresponding to each scale, and the local geometric features corresponding to all scales are aggregated to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data.

[0071] In a specific embodiment, to better illustrate the operation of multi-scale feature extraction, see Figure 4 :

[0072] First, the furthest point sampling technology (FPS) is selected, the sampling rate of the spatial point cloud is set to 50%, and the K nearest neighbor aggregation (KNN) technology is used to achieve multi-scale stability downsampling of the monomer point cloud data to obtain a multi-scale point cloud distribution structure. This is because the randomness of the point cloud distribution obtained after the furthest point sampling has a small impact on the geometric feature encoding, and can proportionally reduce the number of three-dimensional points in the point cloud while keeping the shape of the three-dimensional point cloud unchanged.

[0073] Secondly, within each scale, the PointNeXt encoder is used to generate the basic feature description of the 3D points to reflect the local geometric features of the 3D points. It should be noted here that although the number of point clouds decreases with the increase in sampling scale, the feature dimensions of 3D points at different scales remain unchanged. On this basis, a multi-scale feature pyramid structure of point cloud data is formed.

[0074] Step 104: performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures.

[0075] In this embodiment, the step of fusing local scale features on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures includes:

[0076] In each of the multi-scale feature pyramid structures, within the feature pyramid structure corresponding to each scale, a feature expression generation operation is performed on each point until all points perform the feature expression generation operation, and the local feature expression generated for each point is summarized to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures;

[0077] Among them, the feature expression generation operation is specifically as follows:

[0078] Generate query point features corresponding to the current point, and generate a query vector based on the query point features and the learnable weight matrix;

[0079] Determine neighbor points based on the coordinates of the current point;

[0080] Generate neighbor point features corresponding to the neighbor points, and generate a keyword matrix based on the neighbor point features and the learnable weight matrix;

[0081] Based on the keyword matrix and query vector, a local feature expression of the current point is generated.

[0082] In this embodiment, the generating of the local feature expression of the current point based on the keyword matrix and the query vector includes:

[0083] The query vector, the keyword matrix, the first learnable weight matrix and the second learnable weight matrix are input into the first point-by-point difference formula to calculate the first attention matrix from the neighbor point to the current point; wherein the first point-by-point difference formula is specifically:

[0084]

[0085] In the formula, A ij is the weight value corresponding to an element in the first attention matrix from the neighbor point to the current point, f i is the query point feature, f j is the neighbor point feature, W query is the first learnable weight matrix, W key is the second learnable weight matrix;

[0086] The first attention matrix, the neighbor point features and the third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature expression of the current point; wherein the first matrix multiplication formula is specifically:

[0087]

[0088] In the formula, f i ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W value is the third learnable weight matrix.

[0089] In a specific embodiment, to better illustrate the operation of local scale feature fusion, see Figure 5 :

[0090] First, a certain point is selected in a feature pyramid structure of a certain scale, and the query vector is encoded using the query point feature of the current point and the learnable weight matrix;

[0091] Secondly, according to the coordinates of the current point, a ball query operation is performed to obtain neighbor points and their corresponding neighbor point features, and the encoding of the keyword matrix and the encoding of the value vector are completed using the learnable weight matrix;

[0092] Third, the query vector and the keyword matrix are subtracted point by point, and after being modulated by the linear transformation layer and the activation layer, the attention matrix from the neighbor point to the seed point is obtained. Each element value in the attention matrix represents the weight value from a neighbor point to the current query point.

[0093] Finally, the attention matrix is ​​multiplied by the modulated neighbor point features (i.e., the present invention multiplies the neighbor point features by a learnable weight matrix) at the element level to obtain a point feature expression of the current point with local feature aggregation, thereby completing the local scale self-attention feature fusion.

[0094] Step 105: performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions.

[0095] In this embodiment, performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions includes:

[0096] In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale, and a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale based on the seed point to obtain neighboring points in each feature pyramid structure corresponding to the non-highest scale;

[0097] In each of the multi-scale feature pyramid structures, a cross-scale fusion feature of the seed point is calculated based on the feature of the seed point and the feature of each neighboring point;

[0098] In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global maximum pooling layer to obtain the multi-scale geometric features corresponding to the point feature expression.

[0099] In a specific embodiment, to better illustrate the operation of cross-scale feature fusion, see Figure 6 :

[0100] First, taking the selected 3D points in the highest-scale pyramid point cloud as seeds, KNN queries are performed on the 3D points on each lower-scale pyramid point cloud in turn to obtain the corresponding neighboring points;

[0101] Secondly, the cross attention weight value between the lower-scale 3D point and the highest-scale query seed point is calculated using the second point-by-point difference formula, where the point features on the highest-scale pyramid point cloud are used as query inputs, and the adjacent point features on each lower-scale pyramid point cloud are used as keyword inputs;

[0102] Third, the second matrix multiplication formula is used to calculate the aggregate features of the seed points at each lower scale, and the aggregate features obtained at different scales are further concatenated to form a cross-scale fusion feature of the seed points with scale invariance and cross-scale reliability;

[0103] Finally, the cross-scale fusion features of these seed points are sent to the global maximum pooling layer to obtain multi-scale geometric features with a dimension size of 1024x1.

[0104] In this embodiment, the cross-scale fusion feature of the seed point is calculated based on the feature of the seed point and the feature of each neighboring point, including:

[0105] Substitute the feature corresponding to the seed point and the feature of each neighboring point into the second point-by-point difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to the non-highest scale; wherein the second point-by-point difference formula is specifically:

[0106]

[0107] In the formula, A kl is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point, f k is the feature corresponding to the seed point, f l is the feature of the neighboring point, W′ query is the fourth learnable weight matrix, W′ key is the fifth learnable weight matrix;

[0108] Substitute the second attention matrix of the same scale and the features of the adjacent points into the second matrix multiplication formula, calculate the local feature expression of the seed point at each non-highest scale, and concatenate the local feature expression of the seed point at each non-highest scale to obtain the cross-scale fusion feature of the seed point; wherein, the second matrix multiplication formula is specifically:

[0109]

[0110] In the formula, f k ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W′ value is the sixth learnable weight matrix.

[0111] Step 106: classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain a classification result corresponding to each of the three-dimensional point cloud data.

[0112] In this embodiment, classifying the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification results corresponding to each of the three-dimensional point cloud data includes:

[0113] In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain a classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output, and a multi-layer perceptron layer with a maximum pooling layer is trained.

[0114] In a specific embodiment, the 3D point cloud object classification module is mainly composed of the following steps:

[0115] First, the multi-scale geometric features corresponding to the 3D point cloud data are input into the multi-layer perceptron layer to implement linear transformation and full connection operations;

[0116] Secondly, the maximum pooling operation is used to extract the category information of the transformed multi-scale geometric features to obtain the probability vector of each type of the 3D point cloud object;

[0117] Finally, the maximum value and the corresponding category in the type probability vector are selected and used as the final predicted category of the three-dimensional point cloud object. At this point, the classification task of the three-dimensional point cloud object is completed.

[0118] For better explanation, the following experimental case is provided:

[0119] To evaluate this model, we compared the model classification results with the most advanced methods in the literature, including PointNet, PointNet++, PointNeXt, PointVector, and SPoTr. As shown in Table 1 below, this experiment lists the overall accuracy (OA) and average accuracy (MA) of this model relative to different methods when evaluated on three datasets. The calculation of OA and MA is shown in the following formula:

[0120]

[0121]

[0122] In the formula, Num correct Indicates the total number of correct object labels predicted by the model classifier, Num total Indicates the total number of samples used by the model for prediction; and Indicates the total number of samples for which the model classifier predicts the correct object label when the type label is clsid and the total number of samples for the corresponding label used for prediction, Num cls is the total number of sample label types in the test dataset.

[0123] In addition, the three data sets used in the experiment are ModelNet40, ModelNet40-C, and the substation cloud data set collected by the project.

[0124] ModelNet40 is a 3D shape classification dataset based on CAD drawing and 3D patch sampling. It has a total of 40 types and 12,318 noise-free and well-shaped point cloud samples. It is the most widely used benchmark for point cloud analysis and 3D shape classification tasks. ModelNet40-C is a challenging 3D point cloud object classification dataset. This dataset is built based on the regular shape point cloud of ModelNet40. By adding local noise, global noise, local shape defects, global shape defects, point cloud jitter, etc. to the dataset, a total of 185,000 point cloud classification data samples with noise interference are formed. These two datasets are divided into 80% training set and 20% test set to complete the training and testing of the model, among which the training set is further divided into 80% for model training and 20% for model verification.

[0125] The substation site cloud dataset includes the BIM three-dimensional model and the real-life point cloud scanned by a ground-based lidar scanner. By analyzing the label information of the BIM model, the equipment in the substation scene is classified and sorted, and a total of 177 three-dimensional equipment shapes in 11 categories are obtained. The point cloud sampled from the substation equipment model is rotated and transformed to form a total of 6,372 training samples as the training dataset of this model; the ground points are eliminated from the real-life point cloud collected by the ground-based lidar, and the point cloud is manually segmented and cropped, and the object type labels are manually annotated to form a total of 222 individual real-life equipment objects as the test dataset of this model.

[0126] For the ModelNet40 and ModelNet40-C datasets, the OA values ​​of all methods are greater than 90%, and the MA values ​​are greater than 85%, which shows that these methods are effective. However, regardless of the OA value or the MA value, this model shows a certain competitiveness, while SPoTr or PointNet performs relatively poorly. For example, on the ModelNet40 dataset, the OA value of this model is 2.05%, 0.78%, 0.38%, 0.33% and 2.81% higher than PointNet, PointNet++, PointNeXt, PointVector and SPoTr, respectively, and the MA value is 3.02%, 1.01%, 0.06% and 2.78% higher than PointNet, PointNeXt, PointVector and SPoTr, respectively, but the classification accuracy is slightly lower than that of PointNet++. Compared with other models, this model also performs well on the ModelNet40-C dataset. For example, for OA values, the accuracy of this model is improved by 2.94%, 1.06%, 0.76%, 0.41% and 2.91% respectively; and for MA values, the accuracy of this model is 3.28%, 1.37%, 0.08% and 3.39% higher than PointNet, PointNet++, PointNeXt and SPoTr respectively, but slightly lower than PointVector. This result once again proves the effectiveness and robustness of the model.

[0127] In addition, this experiment also uses our dataset to evaluate the model. The results show that among the state-of-the-art models, the model performs best, but PointNeXt performs the worst. Specifically, in terms of OA value, the model outperforms PointNet, PointNet++, PointNeXt, PointVector, and SPoTr by 5.20%, 1.49%, 9.20%, 4.56%, and 1.49%, respectively; in terms of MA value, the model outperforms PointNet, PointNeXt, and PointVector by 20.02%, 22.62%, and 9.20%, respectively, but the accuracy is slightly lower than SPoTr and PointNet++. It should be noted that the classification accuracy of PointNet++ is relatively high in the substation cloud dataset because it can well capture the hierarchical features or multi-level organization of three-dimensional objects, which is particularly applicable to electrical equipment with complex structures. In this regard, it is very similar to the model and is also competitive with the model in terms of accuracy. Therefore, these results once again confirm the effectiveness of the proposed model in classifying 3D point cloud objects of electrical equipment with complex structures by quantitative comparison with state-of-the-art methods.

[0128] Table 1. Comparison of classification accuracy results between this model and other advanced models

[0129]

[0130] In a specific embodiment, for better explanation, see Figure 7 , the steps of a multi-scale three-dimensional point cloud data classification method involved in another embodiment have been described.

[0131] See also Figure 2 , Figure 2 1 is a schematic diagram of a structure of a multi-scale three-dimensional point cloud data classification device provided by an embodiment of the present invention, comprising: a data acquisition module 201, a monomer extraction module 202, a multi-scale extraction module 203, a feature fusion module 204, a cross-scale feature fusion module 205 and a classification module 206;

[0132] The data acquisition module is used to acquire three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes a plurality of single devices;

[0133] The monomer extraction module is used to perform monomer extraction on the three-dimensional point cloud data to obtain monomer point cloud data of several monomer devices;

[0134] The multi-scale extraction module is used to perform multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data;

[0135] The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures;

[0136] The cross-scale feature fusion module is used to perform cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions;

[0137] The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification results corresponding to each of the three-dimensional point cloud data.

[0138] It can be understood that the above-mentioned system item embodiment corresponds to the method item embodiment of the present invention, which can implement the multi-scale based three-dimensional point cloud data classification method provided by any one of the above-mentioned method item embodiments of the present invention.

[0139] This embodiment obtains three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes several single devices; the three-dimensional point cloud data is subjected to single-body extraction to obtain single point cloud data of several single devices; multi-scale feature extraction is performed on each of the single point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the single point cloud data; local scale feature fusion is performed on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures; cross-scale feature fusion is performed on each of the point feature expressions to obtain a multi-scale geometric feature corresponding to each of the point feature expressions; the multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain a classification result corresponding to each of the three-dimensional point cloud data. The present invention performs three-dimensional point cloud extraction on the single devices of the substation to be tested, performs multi-scale feature extraction on the three-dimensional point cloud data corresponding to the single devices, and performs local scale feature fusion to obtain multi-scale geometric features. The present invention can extract feature information of single devices at different scales. Compared with the prior art that simply performs deep learning based on label marking, the present invention can discover more feature information of the three-dimensional point cloud data of a single device, so that accurate category classification can be performed based on specific feature information during classification, thereby improving the recognition accuracy of complex three-dimensional point cloud device categories in complex substation scenarios.

[0140] Embodiment 2

[0141] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention.

[0142] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the above-mentioned multi-scale three-dimensional point cloud data classification method in the embodiment are implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of the multi-scale three-dimensional point cloud data classification device are shown.

[0143] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the multi-scale based three-dimensional point cloud data classification method as described in any of the above embodiments.

[0144] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0145] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 301 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0146] The memory 302 can be used to store the computer program and / or module. The processor 301 implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0147] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0148] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0149] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-scale three-dimensional point cloud data classification method, characterized in that: include: Acquire three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes a plurality of single devices; Extracting the three-dimensional point cloud data in a monomeric manner to obtain monomer point cloud data of a plurality of monomer devices; Performing multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data; Performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures; Performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions; The multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain a classification result corresponding to each of the three-dimensional point cloud data.

2. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that: The step of extracting multi-scale features from each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data includes: Based on the farthest point sampling technology and K nearest neighbor aggregation, multi-scale downsampling is performed on each of the monomer point cloud data at a preset downsampling rate to obtain a multi-scale point cloud distribution structure corresponding to each of the monomer point cloud data; wherein the multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, and the three-dimensional point clouds in the point cloud distribution structure corresponding to each scale have the same shape and different numbers of three-dimensional point cloud points; In each of the multi-scale point cloud distribution structures, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain local geometric features corresponding to each scale, and the local geometric features corresponding to all scales are aggregated to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data.

3. The multi-scale three-dimensional point cloud data classification method according to claim 2, characterized in that: The performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures includes: In each of the multi-scale feature pyramid structures, within the feature pyramid structure corresponding to each scale, a feature expression generation operation is performed on each point until all points perform the feature expression generation operation, and the local feature expression generated for each point is summarized to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures; Among them, the feature expression generation operation is specifically as follows: Generate query point features corresponding to the current point, and generate a query vector based on the query point features and the learnable weight matrix; Determine neighbor points based on the coordinates of the current point; Generate neighbor point features corresponding to the neighbor points, and generate a keyword matrix based on the neighbor point features and the learnable weight matrix; Based on the keyword matrix and query vector, a local feature expression of the current point is generated.

4. The multi-scale three-dimensional point cloud data classification method according to claim 3, characterized in that: The generating of the local feature expression of the current point based on the keyword matrix and the query vector includes: The query vector, the keyword matrix, the first learnable weight matrix and the second learnable weight matrix are input into the first point-by-point difference formula to calculate the first attention matrix from the neighbor point to the current point; wherein the first point-by-point difference formula is specifically: In the formula, A ij is the weight value corresponding to an element in the first attention matrix from the neighbor point to the current point, f i is the query point feature, f j is the neighbor point feature, W query is the first learnable weight matrix, W key is the second learnable weight matrix; The first attention matrix, the neighbor point features and the third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature expression of the current point; wherein the first matrix multiplication formula is specifically: In the formula, f i ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W value is the third learnable weight matrix.

5. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that: The performing cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions includes: In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale, and a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale based on the seed point to obtain neighboring points in each feature pyramid structure corresponding to the non-highest scale; In each of the multi-scale feature pyramid structures, a cross-scale fusion feature of the seed point is calculated based on the feature of the seed point and the feature of each neighboring point; In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global maximum pooling layer to obtain the multi-scale geometric features corresponding to the point feature expression.

6. The multi-scale three-dimensional point cloud data classification method according to claim 5, characterized in that: The step of calculating the cross-scale fusion feature of the seed point based on the feature of the seed point and the feature of each neighboring point includes: Substitute the feature corresponding to the seed point and the feature of each neighboring point into the second point-by-point difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to the non-highest scale; wherein the second point-by-point difference formula is specifically: In the formula, A kl is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point, f k is the feature corresponding to the seed point, f l is the feature of the neighboring point, W′ query is the fourth learnable weight matrix, W′ key is the fifth learnable weight matrix; Substitute the second attention matrix of the same scale and the features of the adjacent points into the second matrix multiplication formula, calculate the local feature expression of the seed point at each non-highest scale, and concatenate the local feature expression of the seed point at each non-highest scale to obtain the cross-scale fusion feature of the seed point; wherein, the second matrix multiplication formula is specifically: In the formula, f k ′ is the local feature expression of the current point, ⊙ is the multiplication symbol at the matrix element level, W′ value is the sixth learnable weight matrix.

7. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that: The step of classifying the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain a classification result corresponding to each of the three-dimensional point cloud data includes: In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain a classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output, and a multi-layer perceptron layer with a maximum pooling layer is trained.

8. A multi-scale three-dimensional point cloud data classification device, characterized in that: include: Data acquisition module, monomer extraction module, multi-scale extraction module, feature fusion module, cross-scale feature fusion module and classification module; The data acquisition module is used to acquire three-dimensional point cloud data of the substation to be tested; wherein the substation to be tested includes a plurality of single devices; The monomer extraction module is used to perform monomer extraction on the three-dimensional point cloud data to obtain monomer point cloud data of several monomer devices; The multi-scale extraction module is used to perform multi-scale feature extraction on each of the monomer point cloud data to obtain a multi-scale feature pyramid structure corresponding to each of the monomer point cloud data; The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain a point feature expression corresponding to each of the multi-scale feature pyramid structures; The cross-scale feature fusion module is used to perform cross-scale feature fusion on each of the point feature expressions to obtain multi-scale geometric features corresponding to each of the point feature expressions; The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification results corresponding to each of the three-dimensional point cloud data.

9. A computer terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a multi-scale three-dimensional point cloud data classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, the computer-readable storage medium comprising a stored computer program, wherein: When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a multi-scale three-dimensional point cloud data classification method as described in any one of claims 1 to 7.

Images