A method, apparatus, terminal device, and storage medium for classifying 3D point cloud data based on multiple scales.

Abstract

This invention discloses a multi-scale 3D point cloud data classification method, apparatus, terminal device, and storage medium. The method involves acquiring 3D point cloud data of a substation under test; extracting individual points from the 3D point cloud data to obtain individual point cloud data for several individual devices; extracting multi-scale features from each individual point cloud data to obtain a multi-scale feature pyramid structure corresponding to each individual point cloud data; fusing local scale features into each multi-scale feature pyramid structure to obtain point feature representations corresponding to each multi-scale feature pyramid structure; fusing cross-scale features into each point feature representation to obtain multi-scale geometric features; and classifying the multi-scale geometric features corresponding to each 3D point cloud data to obtain a classification result for each 3D point cloud data. Compared to existing technologies that simply rely on labeling for deep learning, this invention improves the accuracy of identifying complex 3D point cloud device categories in complex substation scenarios.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional processing technology, and in particular to a method, apparatus, terminal device and storage medium for classifying three-dimensional point cloud data based on multiple scales. Background Technology

[0002] With the rapid development of laser scanning, SLAM mapping, and photogrammetry technologies, 3D point cloud data has been widely applied in production scenarios such as autonomous driving, digital twins, and slope monitoring. However, due to the limited environmental perception range of point cloud acquisition devices and the varying quality defects resulting from hardware and physical limitations, obtaining large-scale fused point clouds that meet practical requirements typically requires registration and fusion of 3D point clouds from multiple stations to achieve geometrically consistent large-scene point clouds. Currently, existing technologies for 3D point clouds simply rely on deep learning through labeling categories, failing to uncover the deep features of 3D point clouds. This leads to low accuracy in classifying complex 3D point cloud devices in the intricate environments of substations.

[0003] Therefore, there is an urgent need for a 3D point cloud data classification strategy to solve the problem of low accuracy in classifying complex 3D point cloud devices in the complex scene of substations. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal device, and storage medium for classifying 3D point cloud data based on multiple scales, in order to solve the problem of low accuracy in classifying complex 3D point cloud devices in substation scenarios.

[0005] To address the aforementioned problems, one embodiment of the present invention provides a multi-scale 3D point cloud data classification method based on multi-scale feature representation and cross-scale cross-attention mechanism, comprising:

[0006] Acquire three-dimensional point cloud data of the substation under test; wherein the substation under test includes several individual devices;

[0007] The three-dimensional point cloud data is extracted into individual units to obtain individual point cloud data of several individual devices;

[0008] Multi-scale feature extraction is performed on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data.

[0009] Local scale feature fusion is performed on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each of the multi-scale feature pyramid structures.

[0010] Cross-scale feature fusion is performed on each of the point feature representations to obtain the multi-scale geometric features corresponding to each point feature representation.

[0011] The multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain the classification result corresponding to each of the three-dimensional point cloud data.

[0012] As an improvement to the above scheme, the step of performing multi-scale feature extraction on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data includes:

[0013] Based on the farthest point sampling technique and K-nearest neighbor aggregation, each of the individual point cloud data is downsampled at a preset downsampling rate to obtain the multi-scale point cloud distribution structure corresponding to each individual point cloud data; wherein, the multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, and the three-dimensional point cloud shapes in the point cloud distribution structure corresponding to each scale are the same but the number of three-dimensional point cloud points is different.

[0014] In each of the multi-scale point cloud distribution structures, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain the local geometric features corresponding to each scale. The local geometric features corresponding to all scales are summarized to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data.

[0015] As an improvement to the above scheme, the step of performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure includes:

[0016] In each of the multi-scale feature pyramid structures, within the feature pyramid structure corresponding to each scale, a feature expression generation operation is performed on each point until all points have performed the feature expression generation operation. The local feature expressions generated at each point are then summarized to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures.

[0017] The feature representation generation operation specifically includes:

[0018] Generate the query point features corresponding to the current point, and generate a query vector based on the query point features and the learnable weight matrix;

[0019] Determine neighboring points based on the coordinates of the current point;

[0020] Generate the neighbor point features corresponding to the neighbor points, and generate a keyword matrix based on the neighbor point features and the learnable weight matrix;

[0021] Based on the keyword matrix and query vector, a local feature representation of the current point is generated.

[0022] As an improvement to the above scheme, the step of generating the local feature representation of the current point based on the keyword matrix and query vector includes:

[0023] The query vector, keyword matrix, first learnable weight matrix, and second learnable weight matrix are input into the first point-by-point subtraction formula to calculate the first attention matrix from neighboring points to the current point; wherein, the first point-by-point subtraction formula is specifically:

[0024]

[0025] In the formula, A ij f is the weight value corresponding to an element in the first attention matrix from neighboring points to the current point. i For query point features, f j For the characteristics of neighboring points, W query W is the first learnable weight matrix. key This is the second learnable weight matrix;

[0026] The first attention matrix, neighbor point features, and third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature representation of the current point; wherein, the first matrix multiplication formula is specifically as follows:

[0027]

[0028] In the formula, f i ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W... value This is the third learnable weight matrix.

[0029] As an improvement to the above scheme, the step of performing cross-scale feature fusion on each point feature representation to obtain the multi-scale geometric features corresponding to each point feature representation includes:

[0030] In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale. Based on the seed point, a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale to obtain the neighboring points in each feature pyramid structure corresponding to a non-highest scale.

[0031] In each of the multi-scale feature pyramid structures, the cross-scale fusion features of the seed point are calculated based on the features of the seed point and the features of each neighboring point.

[0032] In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global max pooling layer to obtain the multi-scale geometric features corresponding to the point feature representation.

[0033] As an improvement to the above scheme, the calculation of the cross-scale fusion feature of the seed point based on the features of the seed point and the features of each neighboring point includes:

[0034] Substitute the features corresponding to the seed point and the features of each neighboring point into the second pointwise difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to a non-highest scale; wherein, the second pointwise difference formula is specifically as follows:

[0035]

[0036] In the formula, A kl f is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point. k f represents the feature corresponding to the seed point. l As a feature of neighboring points, W′ query W′ is the fourth learnable weight matrix. key This is the fifth learnable weight matrix;

[0037] Substituting the second attention matrix of the same scale and the features of neighboring points into the second matrix multiplication formula, the local feature representation of the seed point at each non-maximum scale is calculated. These local feature representations are then concatenated to obtain the cross-scale fusion feature of the seed point. Specifically, the second matrix multiplication formula is as follows:

[0038]

[0039] In the formula, f k ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W′ value This is the sixth learnable weight matrix.

[0040] As an improvement to the above scheme, the step of classifying the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data includes:

[0041] In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain the classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output to train the multilayer perceptron layer with a max pooling layer.

[0042] Accordingly, one embodiment of the present invention also provides a multi-scale three-dimensional point cloud data classification device, including: a data acquisition module, a single-unit extraction module, a multi-scale extraction module, a feature fusion module, a cross-scale feature fusion module, and a classification module;

[0043] The data acquisition module is used to acquire three-dimensional point cloud data of the substation under test; wherein, the substation under test includes several individual devices;

[0044] The single-unit extraction module is used to extract single-unit data from the three-dimensional point cloud data to obtain single-unit point cloud data of several single devices.

[0045] The multi-scale extraction module is used to perform multi-scale feature extraction on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each of the individual point cloud data.

[0046] The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each of the multi-scale feature pyramid structures.

[0047] The cross-scale feature fusion module is used to perform cross-scale feature fusion on each point feature expression to obtain the multi-scale geometric features corresponding to each point feature expression.

[0048] The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data.

[0049] Accordingly, one embodiment of the present invention also provides a computer terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a multi-scale three-dimensional point cloud data classification method as described in the present invention.

[0050] Accordingly, one embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a multi-scale three-dimensional point cloud data classification method as described in the present invention.

[0051] As can be seen from the above, the present invention has the following beneficial effects:

[0052] This invention provides a multi-scale 3D point cloud data classification method based on multi-scale feature representation and cross-scale cross-attention mechanism to acquire 3D point cloud data of a substation under test. The substation under test includes several individual devices. The method involves: extracting individual points from the 3D point cloud data to obtain individual point cloud data for each individual device; extracting multi-scale features from each individual point cloud data to obtain a corresponding multi-scale feature pyramid structure; performing local scale feature fusion on each multi-scale feature pyramid structure to obtain a point feature representation corresponding to each multi-scale feature pyramid structure; performing cross-scale feature fusion on each point feature representation to obtain a corresponding multi-scale geometric feature; and classifying the multi-scale geometric feature corresponding to each 3D point cloud data to obtain a classification result for each 3D point cloud data. This invention extracts 3D point clouds from individual devices in the substation under test, performs multi-scale feature extraction on the corresponding 3D point cloud data, and performs local scale feature fusion to obtain multi-scale geometric features. This invention can extract feature information of individual devices at different scales. Compared to existing technologies that simply rely on labeling for deep learning, this invention can uncover more feature information from the 3D point cloud data of individual devices, enabling accurate category classification based on specific feature information during classification. This improves the accuracy of identifying complex 3D point cloud device categories in substation scenarios. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a multi-scale three-dimensional point cloud data classification method provided in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of a multi-scale three-dimensional point cloud data classification device provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a terminal device structure provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the multi-scale feature extraction process provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the local scale feature fusion process provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the cross-scale feature fusion process provided in an embodiment of the present invention;

[0059] Figure 7 This is a flowchart illustrating a multi-scale three-dimensional point cloud data classification method provided in another embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Example 1

[0062] See Figure 1 , Figure 1 This is a flowchart illustrating a multi-scale 3D point cloud data classification method according to an embodiment of the present invention, as shown below. Figure 1 As shown, this embodiment includes steps 101 to 106, and the specific steps are as follows:

[0063] Step 101: Obtain the three-dimensional point cloud data of the substation under test; wherein the substation under test includes several individual devices.

[0064] In this embodiment, the three-dimensional point cloud data of the substation under test can be retrieved from a network three-dimensional resource library, or the three-dimensional point cloud data of the substation under test can be collected and generated based on a three-dimensional camera.

[0065] Step 102: Extract individual units from the three-dimensional point cloud data to obtain individual point cloud data for several individual devices.

[0066] In this embodiment, the automatic extraction technology of single objects from point cloud data in complex scenes is used to extract the three-dimensional point cloud data of the substation scene under test.

[0067] Step 103: Perform multi-scale feature extraction on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each of the individual point cloud data.

[0068] In this embodiment, the step of performing multi-scale feature extraction on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data includes:

[0069] Based on the farthest point sampling technique and K-nearest neighbor aggregation, each of the individual point cloud data is downsampled at a preset downsampling rate to obtain the multi-scale point cloud distribution structure corresponding to each individual point cloud data; wherein, the multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, and the three-dimensional point cloud shapes in the point cloud distribution structure corresponding to each scale are the same but the number of three-dimensional point cloud points is different.

[0070] In each of the multi-scale point cloud distribution structures, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain the local geometric features corresponding to each scale. The local geometric features corresponding to all scales are summarized to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data.

[0071] In one specific embodiment, for a better illustration of the multi-scale feature extraction operation, see [link to relevant documentation]. Figure 4 :

[0072] First, the farthest point sampling (FPS) technique was selected, and the sampling rate of the spatial point cloud was set to 50%. Then, the K-nearest neighbor (KNN) aggregation technique was used to achieve multi-scale stable downsampling of the individual point cloud data, resulting in a multi-scale point cloud distribution structure. This is because the randomness of the point cloud distribution obtained after farthest point sampling has a smaller impact on geometric feature encoding, and it can proportionally reduce the number of 3D points in the point cloud while maintaining the shape of the 3D point cloud.

[0073] Secondly, within each scale, the PointNeXt encoder is used to generate basic feature descriptions of the 3D points to reflect their local geometric features. It's important to note that although the number of point clouds decreases with increasing sampling scale, the feature dimensions of the 3D points remain constant across different scales. This forms the multi-scale feature pyramid structure of the point cloud data.

[0074] Step 104: Perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each of the multi-scale feature pyramid structures.

[0075] In this embodiment, the step of performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure includes:

[0076] In each of the multi-scale feature pyramid structures, within the feature pyramid structure corresponding to each scale, a feature expression generation operation is performed on each point until all points have performed the feature expression generation operation. The local feature expressions generated at each point are then summarized to obtain the point feature expression corresponding to each of the multi-scale feature pyramid structures.

[0077] The feature representation generation operation specifically includes:

[0078] Generate the query point features corresponding to the current point, and generate a query vector based on the query point features and the learnable weight matrix;

[0079] Determine neighboring points based on the coordinates of the current point;

[0080] Generate the neighbor point features corresponding to the neighbor points, and generate a keyword matrix based on the neighbor point features and the learnable weight matrix;

[0081] Based on the keyword matrix and query vector, a local feature representation of the current point is generated.

[0082] In this embodiment, generating the local feature representation of the current point based on the keyword matrix and query vector includes:

[0083] The query vector, keyword matrix, first learnable weight matrix, and second learnable weight matrix are input into the first point-by-point subtraction formula to calculate the first attention matrix from neighboring points to the current point; wherein, the first point-by-point subtraction formula is specifically:

[0084]

[0085] In the formula, A ij f is the weight value corresponding to an element in the first attention matrix from neighboring points to the current point. i For query point features, f j For the characteristics of neighboring points, W query W is the first learnable weight matrix. key This is the second learnable weight matrix;

[0086] The first attention matrix, neighbor point features, and third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature representation of the current point; wherein, the first matrix multiplication formula is specifically as follows:

[0087]

[0088] In the formula, f i ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W... value This is the third learnable weight matrix.

[0089] In a specific embodiment, for a better illustration of the local scale feature fusion operation, please refer to [link to specific implementation]. Figure 5 :

[0090] First, in a feature pyramid structure of a certain scale, a point is selected, and the query vector is encoded using the query point features of the current point and the learnable weight matrix.

[0091] Secondly, based on the coordinates of the current point, a ball query operation is performed to obtain neighboring points and their corresponding features. The learnable weight matrix is ​​then used to complete the encoding of the keyword matrix and the value vector.

[0092] Third, the query vector is subtracted from the keyword matrix point by point, and then modulated through a linear transformation layer and an activation layer to obtain the attention matrix from the neighbor point to the seed point. Each element value in the attention matrix represents the weight value from a certain neighbor point to the current query point.

[0093] Finally, the attention matrix is ​​multiplied by the modulated neighbor point features (i.e., the neighbor point features are multiplied by the learnable weight matrix in this invention) to obtain the point feature expression of the current point with local feature aggregation, thereby completing the local scale self-attention feature fusion.

[0094] Step 105: Perform cross-scale feature fusion on each point feature representation to obtain the multi-scale geometric features corresponding to each point feature representation.

[0095] In this embodiment, the step of performing cross-scale feature fusion on each point feature representation to obtain the multi-scale geometric features corresponding to each point feature representation includes:

[0096] In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale. Based on the seed point, a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale to obtain the neighboring points in each feature pyramid structure corresponding to a non-highest scale.

[0097] In each of the multi-scale feature pyramid structures, the cross-scale fusion features of the seed point are calculated based on the features of the seed point and the features of each neighboring point.

[0098] In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global max pooling layer to obtain the multi-scale geometric features corresponding to the point feature representation.

[0099] In a specific embodiment, for a better illustration of the cross-scale feature fusion operation, please refer to [link to relevant documentation]. Figure 6 :

[0100] First, using selected 3D points in the highest-scale pyramid point cloud as seeds, KNN queries are performed sequentially on each 3D point in the lower-scale pyramid point cloud to obtain the corresponding neighboring points.

[0101] Secondly, the cross-attention weight value between the lower-scale 3D point and the highest-scale query seed point is calculated using the second point-by-point difference formula. The point features on the highest-scale pyramid point cloud are used as the query input, while the features of each neighboring point on the lower-scale pyramid point cloud are used as the keyword input.

[0102] Third, the aggregation features of seed points at each lower scale are calculated using the second matrix multiplication formula, and the aggregation features obtained at different scales are further spliced ​​together to form the cross-scale fusion features of seed points with scale invariance and cross-scale reliability.

[0103] Finally, the cross-scale fusion features of these seed points are fed into a global max pooling layer to obtain multi-scale geometric features with a dimension of 1024x1.

[0104] In this embodiment, calculating the cross-scale fusion feature of the seed point based on the features of the seed point and the features of each neighboring point includes:

[0105] Substitute the features corresponding to the seed point and the features of each neighboring point into the second pointwise difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to a non-highest scale; wherein, the second pointwise difference formula is specifically as follows:

[0106]

[0107] In the formula, A kl f is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point. k f represents the feature corresponding to the seed point. l As a feature of neighboring points, W′ query W′ is the fourth learnable weight matrix. key This is the fifth learnable weight matrix;

[0108] Substituting the second attention matrix of the same scale and the features of neighboring points into the second matrix multiplication formula, the local feature representation of the seed point at each non-maximum scale is calculated. These local feature representations are then concatenated to obtain the cross-scale fusion feature of the seed point. Specifically, the second matrix multiplication formula is as follows:

[0109]

[0110] In the formula, f k ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W′ value This is the sixth learnable weight matrix.

[0111] Step 106: Classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data.

[0112] In this embodiment, classifying the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data includes:

[0113] In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain the classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output to train the multilayer perceptron layer with a max pooling layer.

[0114] In a specific embodiment, the 3D point cloud object classification module mainly consists of the following steps:

[0115] First, the multi-scale geometric features corresponding to the 3D point cloud data are input into the multilayer perceptron layer to achieve linear transformation and fully connected operations;

[0116] Secondly, the max pooling operation is used to extract category information from the transformed multi-scale geometric features to obtain the probability vector of each type of the 3D point cloud object.

[0117] Finally, the maximum value in the type probability vector and its corresponding category are selected as the final predicted category of the 3D point cloud object. Thus, the classification task of the 3D point cloud object is completed.

[0118] To better illustrate this, the following experimental case is provided:

[0119] To evaluate this model, we compared its classification results with state-of-the-art methods in the literature, including PointNet, PointNet++, PointNeXt, PointVector, and SPoTr. Table 1 below shows the comparison of the model's overall accuracy (OA) and average accuracy (MA) relative to different methods when evaluated on three datasets. OA and MA are calculated as follows:

[0120]

[0121]

[0122] In the formula, Num correct Num represents the total number of correctly predicted object labels by the model classifier. total This represents the total number of samples used by the model for prediction; and This represents the total number of samples where the model classifier correctly predicts the object label when the type label is clsid, compared to the total number of samples used for prediction of the corresponding label. Num cls This represents the total number of sample label types in the test dataset.

[0123] In addition, the three datasets used in the experiment were ModelNet40, ModelNet40-C, and the substation cloud dataset collected by the project.

[0124] ModelNet40 is a 3D shape classification dataset based on CAD drawing and 3D patch sampling. It contains 40 categories and 12,318 noise-free, well-shaped point cloud samples, making it the most widely used benchmark for point cloud analysis and 3D shape classification tasks. ModelNet40-C is a challenging 3D point cloud object classification dataset. This dataset is built upon the regular-shaped point clouds of ModelNet40. By adding local noise, global noise, local shape defects, global shape defects, and point cloud jitter, it creates a total of 185,000 noisy point cloud classification data samples. Both datasets are divided into 80% training sets and 20% test sets for model training and testing, respectively. The training set is further divided into 80% for model training and 20% for model validation.

[0125] The substation site cloud dataset includes BIM 3D models and real-world point clouds scanned by a ground-based LiDAR scanner. By analyzing the label information of the BIM models, the equipment in the substation scene was classified and organized, resulting in a total of 177 3D equipment shapes in 11 categories. The point clouds sampled from the substation equipment models were rotated and transformed to form a total of 6372 training samples, which served as the training dataset for this model. Ground points were removed from the real-world point clouds collected by the ground-based LiDAR, and the point clouds were manually segmented, cropped, and labeled with object type labels to form a total of 222 individual real-world equipment objects, which served as the test dataset for this model.

[0126] For the ModelNet40 and ModelNet40-C datasets, all methods achieved OA values ​​greater than 90% and MA values ​​greater than 85%, indicating their effectiveness. However, our model demonstrated competitiveness in both OA and MA values, while SPoTr and PointNet performed relatively poorly. For example, on the ModelNet40 dataset, our model's OA values ​​were 2.05%, 0.78%, 0.38%, 0.33%, and 2.81% higher than PointNet, PointNet++, PointNeXt, PointVector, and SPoTr, respectively, and its MA values ​​were 3.02%, 1.01%, 0.06%, and 2.78% higher, respectively, although its classification accuracy was slightly lower than PointNet++. Compared to other models, our model also performed well on the ModelNet40-C dataset. For example, for OA values, the accuracy of this model improved by 2.94%, 1.06%, 0.76%, 0.41%, and 2.91%, respectively; while for MA values, the accuracy of this model improved by 3.28%, 1.37%, 0.08%, and 3.39% compared to PointNet, PointNet++, PointNeXt, and SPoTr, respectively, but was slightly lower than PointVector. This result further demonstrates the effectiveness and robustness of the model.

[0127] Furthermore, this experiment evaluated the model using our dataset. The results show that among state-of-the-art models, our model performs best, while PointNeXt performs worst. Specifically, in terms of OA (Optimal Aspect Ratio), our model outperforms PointNet, PointNet++, PointNeXt, PointVector, and SPoTr by 5.20%, 1.49%, 9.20%, 4.56%, and 1.49%, respectively; in terms of MA (Maximum Aspect Ratio), our model outperforms PointNet, PointNeXt, and PointVector by 20.02%, 22.62%, and 9.20%, respectively, but its accuracy is slightly lower than SPoTr and PointNet++. It should be noted that PointNet++ has relatively high classification accuracy in the substation cloud dataset because it captures the hierarchical features or multi-level organization of 3D objects well, which is particularly suitable for structurally complex electrical equipment. In this respect, it is very similar to our model and is competitive with it in terms of accuracy. Therefore, through quantitative comparison with state-of-the-art methods, these results once again confirm the effectiveness of our model in classifying 3D point cloud objects of electrical equipment with complex structures.

[0128] Table 1. Comparison of classification accuracy results between this model and other advanced models

[0129]

[0130] In one specific embodiment, for better illustration, see [link to relevant documentation]. Figure 7 The steps of a multi-scale three-dimensional point cloud data classification method involved in another embodiment have been described.

[0131] See Figure 2 , Figure 2 This is a schematic diagram of a multi-scale three-dimensional point cloud data classification device provided in an embodiment of the present invention, including: a data acquisition module 201, a single-item extraction module 202, a multi-scale extraction module 203, a feature fusion module 204, a cross-scale feature fusion module 205, and a classification module 206.

[0132] The data acquisition module is used to acquire three-dimensional point cloud data of the substation under test; wherein, the substation under test includes several individual devices;

[0133] The single-unit extraction module is used to extract single-unit data from the three-dimensional point cloud data to obtain single-unit point cloud data of several single devices.

[0134] The multi-scale extraction module is used to perform multi-scale feature extraction on each of the individual point cloud data to obtain the multi-scale feature pyramid structure corresponding to each of the individual point cloud data.

[0135] The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each of the multi-scale feature pyramid structures.

[0136] The cross-scale feature fusion module is used to perform cross-scale feature fusion on each point feature expression to obtain the multi-scale geometric features corresponding to each point feature expression.

[0137] The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data.

[0138] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the multi-scale three-dimensional point cloud data classification method provided by any of the above method embodiments of the present invention.

[0139] This embodiment acquires 3D point cloud data of a substation under test, wherein the substation under test includes several individual devices. The 3D point cloud data is extracted into individual devices to obtain individual point cloud data for each device. Multi-scale feature extraction is performed on each individual point cloud data to obtain a multi-scale feature pyramid structure corresponding to each individual point cloud data. Local-scale feature fusion is performed on each multi-scale feature pyramid structure to obtain a point feature representation corresponding to each multi-scale feature pyramid structure. Cross-scale feature fusion is performed on each point feature representation to obtain multi-scale geometric features corresponding to each point feature representation. The multi-scale geometric features corresponding to each 3D point cloud data are classified to obtain a classification result corresponding to each 3D point cloud data. This invention extracts 3D point clouds from individual devices in the substation under test, performs multi-scale feature extraction on the corresponding 3D point cloud data, and performs local-scale feature fusion to obtain multi-scale geometric features. This invention can extract feature information of individual devices at different scales. Compared to existing technologies that simply rely on labeling for deep learning, this invention can uncover more feature information from the 3D point cloud data of individual devices, enabling accurate category classification based on specific feature information during classification. This improves the accuracy of identifying complex 3D point cloud device categories in substation scenarios.

[0140] Example 2

[0141] See Figure 3 , Figure 3 This is a schematic diagram of the terminal device structure provided in an embodiment of the present invention.

[0142] One terminal device in this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the steps of the various multi-scale-based 3D point cloud data classification methods described above in this embodiment, for example... Figure 1 The steps of the multi-scale 3D point cloud data classification method shown are described. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiments, for example: Figure 2 The diagram shows all modules of a multi-scale 3D point cloud data classification device.

[0143] In addition, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the multi-scale three-dimensional point cloud data classification method as described in any of the above embodiments.

[0144] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0145] The processor 301 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor 301 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0146] The memory 302 can be used to store the computer programs and / or modules. The processor 301 implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0147] Wherein, if the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0148] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0149] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-scale three-dimensional point cloud data classification method, characterized in that, include: Acquire three-dimensional point cloud data of the substation under test; wherein the substation under test includes several individual devices; The three-dimensional point cloud data is extracted into individual units to obtain individual point cloud data of several individual devices; Multi-scale feature extraction is performed on each of the individual point cloud data to obtain a multi-scale feature pyramid structure corresponding to each individual point cloud data. This multi-scale feature extraction includes: performing multi-scale downsampling on each individual point cloud data at a preset downsampling rate based on farthest point sampling technology and K-nearest neighbor aggregation to obtain a multi-scale point cloud distribution structure corresponding to each individual point cloud data. The multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, where the 3D point cloud shapes are the same but the number of 3D point clouds differs in each scale's point cloud distribution structure. In each multi-scale point cloud distribution structure, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain local geometric features corresponding to each scale. These local geometric features are then summarized to obtain the multi-scale feature pyramid structure corresponding to each individual point cloud data. Local scale feature fusion is performed on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure. The process of performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure includes: within each multi-scale feature pyramid structure, performing a feature representation generation operation on each point within the feature pyramid structure corresponding to each scale, until all points have undergone the feature representation generation operation; summarizing the local feature representations generated for each point to obtain the point feature representation corresponding to each multi-scale feature pyramid structure. Specifically, the feature representation generation operation includes: generating query point features corresponding to the current point, and generating a query vector based on the query point features and a learnable weight matrix; determining neighboring points based on the coordinates of the current point; generating neighboring point features corresponding to the neighboring points, and generating a keyword matrix based on the neighboring point features and a learnable weight matrix; and generating a local feature representation of the current point based on the keyword matrix and the query vector. Cross-scale feature fusion is performed on each of the point feature representations to obtain the multi-scale geometric features corresponding to each point feature representation. The multi-scale geometric features corresponding to each of the three-dimensional point cloud data are classified to obtain the classification result corresponding to each of the three-dimensional point cloud data.

2. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that, The process of generating a local feature representation of the current point based on the keyword matrix and query vector includes: The query vector, keyword matrix, first learnable weight matrix, and second learnable weight matrix are input into the first point-by-point subtraction formula to calculate the first attention matrix from neighboring points to the current point; wherein, the first point-by-point subtraction formula is specifically: In the formula, A ij f is the weight value corresponding to an element in the first attention matrix from neighboring points to the current point. i For query point features, f j For the characteristics of neighboring points, W query W is the first learnable weight matrix. key This is the second learnable weight matrix; The first attention matrix, neighbor point features, and third learnable weight matrix are input into the first matrix multiplication formula to calculate the local feature representation of the current point; wherein, the first matrix multiplication formula is specifically as follows: In the formula, f i ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W... value This is the third learnable weight matrix.

3. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that, The step of performing cross-scale feature fusion on each of the point feature representations to obtain the multi-scale geometric features corresponding to each point feature representation includes: In each of the multi-scale feature pyramid structures, a point is selected as a seed point in the feature pyramid structure corresponding to the highest scale. Based on the seed point, a KNN query is performed on each feature pyramid structure corresponding to a non-highest scale to obtain the neighboring points in each feature pyramid structure corresponding to a non-highest scale. In each of the multi-scale feature pyramid structures, the cross-scale fusion features of the seed point are calculated based on the features of the seed point and the features of each neighboring point. In each of the multi-scale feature pyramid structures, the cross-scale fusion features corresponding to all seed points in the feature pyramid structure corresponding to the highest scale are input into the global max pooling layer to obtain the multi-scale geometric features corresponding to the point feature representation.

4. The multi-scale three-dimensional point cloud data classification method according to claim 3, characterized in that, The calculation of cross-scale fusion features of the seed point based on its features and the features of each neighboring point includes: Substitute the features corresponding to the seed point and the features of each neighboring point into the second pointwise difference formula to calculate the second attention matrix between the seed point and each neighboring point corresponding to a non-highest scale; wherein, the second pointwise difference formula is specifically as follows: In the formula, A kl f is the weight value corresponding to an element in the second attention matrix from a neighboring point to the current point. k f represents the feature corresponding to the seed point. l As a feature of neighboring points, W′ query W′ is the fourth learnable weight matrix. key This is the fifth learnable weight matrix; Substituting the second attention matrix of the same scale and the features of neighboring points into the second matrix multiplication formula, the local feature representation of the seed point at each non-maximum scale is calculated. These local feature representations are then concatenated to obtain the cross-scale fusion feature of the seed point. Specifically, the second matrix multiplication formula is as follows: In the formula, f k ′ This represents the local feature representation of the current point, ⊙ represents the matrix element-level multiplication symbol, and W′ value This is the sixth learnable weight matrix.

5. The multi-scale three-dimensional point cloud data classification method according to claim 1, characterized in that, The step of classifying the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data includes: In each of the three-dimensional point cloud data, the corresponding multi-scale geometric features are input into a preset three-dimensional object classification model to obtain the classification result corresponding to the current three-dimensional point cloud data; wherein, the three-dimensional point cloud samples marked with the classification results are used as input, and the three-dimensional object classification model is used as output to train the multilayer perceptron layer with a max pooling layer.

6. A multi-scale three-dimensional point cloud data classification device, characterized in that, include: The system includes a data acquisition module, a single-entity extraction module, a multi-scale extraction module, a feature fusion module, a cross-scale feature fusion module, and a classification module. The data acquisition module is used to acquire three-dimensional point cloud data of the substation under test; wherein, the substation under test includes several individual devices; The single-unit extraction module is used to extract single-unit data from the three-dimensional point cloud data to obtain single-unit point cloud data of several single devices. The multi-scale extraction module is used to perform multi-scale feature extraction on each of the individual point cloud data to obtain a multi-scale feature pyramid structure corresponding to each individual point cloud data. The step of performing multi-scale feature extraction on each of the individual point cloud data to obtain a multi-scale feature pyramid structure includes: performing multi-scale downsampling on each individual point cloud data based on the farthest point sampling technique and K-nearest neighbor aggregation at a preset downsampling rate to obtain a multi-scale point cloud distribution structure corresponding to each individual point cloud data. The multi-scale point cloud distribution structure includes point cloud distribution structures corresponding to multiple scales, where the three-dimensional point cloud shapes are the same but the number of three-dimensional point cloud points differs in each scale's point cloud distribution structure. In each multi-scale point cloud distribution structure, feature extraction is performed on the point cloud distribution structure corresponding to each scale to obtain local geometric features corresponding to each scale. The local geometric features corresponding to all scales are then summarized to obtain a multi-scale feature pyramid structure corresponding to each individual point cloud data. The feature fusion module is used to perform local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure. The step of performing local scale feature fusion on each of the multi-scale feature pyramid structures to obtain the point feature representation corresponding to each multi-scale feature pyramid structure includes: in each multi-scale feature pyramid structure, within the feature pyramid structure corresponding to each scale, performing a feature representation generation operation on each point until all points have undergone the feature representation generation operation, summarizing the local feature representations generated for each point to obtain the point feature representation corresponding to each multi-scale feature pyramid structure. Specifically, the feature representation generation operation includes: generating query point features corresponding to the current point, and generating a query vector based on the query point features and a learnable weight matrix; determining neighboring points based on the coordinates of the current point; generating neighboring point features corresponding to the neighboring points, and generating a keyword matrix based on the neighboring point features and a learnable weight matrix; and generating a local feature representation of the current point based on the keyword matrix and the query vector. The cross-scale feature fusion module is used to perform cross-scale feature fusion on each point feature expression to obtain the multi-scale geometric features corresponding to each point feature expression. The classification module is used to classify the multi-scale geometric features corresponding to each of the three-dimensional point cloud data to obtain the classification result corresponding to each of the three-dimensional point cloud data.

7. A computer terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a multi-scale three-dimensional point cloud data classification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium comprising a stored computer program, wherein, When the computer program is running, it controls the device containing the computer-readable storage medium to execute a multi-scale three-dimensional point cloud data classification method as described in any one of claims 1 to 5.

Images