3D Point Cloud Object Detection and Abnormal State Evaluation Method, System and Device

Abstract

The present invention discloses a three-dimensional point cloud target detection and abnormal state evaluation method, system and device, which collect three-dimensional point cloud data of target devices in a power scene, perform data annotation and division, make a power scene target detection data set, and divide it into a training set, a validation set and a test set; construct a three-dimensional target detection and abnormal state evaluation model, input the training set in the power scene target detection data set into the three-dimensional target detection model for training, use a multi-task loss function to optimize the parameters of the model until the model converges, and complete the model training; input the test set into the trained power scene three-dimensional target detection model, evaluate the detection performance of the model, and verify the effectiveness of the model. It provides reliable data support for the intelligent operation and maintenance of power equipment, helps to improve the operation safety and stability of the power system, reduces the cost of manual inspection at the same time, and promotes the transformation and development of the power industry towards intelligence and unmanned operation.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent inspection and three-dimensional perception in the power scenario, and particularly relates to a three-dimensional point cloud target detection and abnormal state evaluation method, system and device. Background Art

[0002] The power system is the core infrastructure for the operation of modern society, and its safety and reliability have important strategic significance for the stable development of the social economy. With the continuous growth of power demand, the types and quantities of power equipment have increased rapidly, and the complexity of its operation and maintenance has also risen accordingly. The health status of power equipment is not only related to the stability of the power system, but also directly affects the continuity of power supply and power consumption safety. Especially during the operation of key equipment such as high-voltage transmission lines and towers, their physical states and operating environments are crucial to the overall performance of the power system. However, the current inspection of power equipment mainly relies on manual methods, including on-site manual inspections, assisted analysis of drone image acquisition, etc. This method is inefficient, costly, and greatly limited by the environment and manual experience. Especially in some areas with complex terrain or harsh environments, the inspection tasks face higher safety risks and work intensities.

[0003] In recent years, with the popularization of intelligent power grids and the development of emerging technologies, the traditional manual inspection method has gradually become difficult to meet the power industry's requirements for real-time, high-precision, and full-coverage inspections. How to use modern technologies to accurately detect and evaluate the abnormal states of power equipment has become the focus of the industry. The application of lidar (LiDAR) technology provides a new solution for the intelligent inspection of power equipment. However, in the process of applying three-dimensional point cloud technology to the power scenario, many challenges still need to be solved urgently. On the one hand, the key equipment (such as wires, towers, etc.) in the power scenario has complex structures, variable sizes and postures, and significant geometric feature differences between equipment. On the other hand, abnormal states such as wire breakage, equipment tilt, and foreign object hanging may occur during the operation of power equipment. These abnormalities usually manifest as changes in the local geometric structure in the point cloud data. Therefore, precise point cloud analysis methods need to be combined to accurately identify them.

[0004] Currently, lidar is mainly used as the core sensor for equipment detection and status evaluation in the power scenario to collect high-precision point cloud data for complex scenarios. Under the algorithm framework of deep learning, through feature extraction and target classification of point cloud data, accurate detection and status recognition of power equipment on transmission lines (such as wires, towers, etc.) can be achieved, providing technical support for the abnormal state evaluation of power equipment (such as wire breakage, equipment tilt, foreign object hanging, etc.). However, the following difficulties are faced in the specific implementation process:

[0005] 1. In the power scenario, the key equipment of transmission lines (such as wires, poles, etc.) has various shapes and complex structures. There are significant changes in the size, position, and attitude among different equipment. In addition, the power scenario is usually distributed in an open outdoor environment with complex backgrounds such as vegetation and foreign object interference, which increases the difficulty of 3D object detection.

[0006] 2. During the operation of power equipment, abnormal conditions such as wire breakage, equipment tilt, and foreign object hanging may occur. These abnormalities usually manifest as local changes in the point cloud data, and it is necessary to accurately extract the structural information of the equipment from the 3D point cloud and combine it with the state evaluation algorithm for accurate identification.

[0007] 3. The 3D point cloud data volume is large, and directly processing the full amount of point cloud will result in too high computational costs. While downsampling or voxelizing the point cloud can speed up the calculation, it may cause the loss of key detailed information. Therefore, it is necessary to optimize the computational efficiency while ensuring the detection accuracy. Summary of the Invention

[0008] Aiming at the above technical problems, the present invention provides a 3D point cloud object detection and abnormal state evaluation method, system, and device, aiming to optimize the technical problems such as high complexity of point cloud data, diverse equipment categories, easy loss of feature information, and insufficient accuracy of abnormal state recognition in the process of equipment detection and state evaluation in the power scenario.

[0009] The technical solution adopted by the present invention to solve its technical problems is:

[0010] A 3D point cloud object detection and abnormal state evaluation method, the method includes the following steps:

[0011] S100: Build a 3D perception platform for the power scenario based on lidar, collect the 3D point cloud data of the target equipment in the power scenario, perform data annotation and division on the collected 3D point cloud data, make a power scenario object detection data set, and divide it into a training set, a validation set, and a test set;

[0012] S200: Construct a 3D point cloud object detection and abnormal state evaluation model, the model includes a point set extraction module, a feature refinement module based on local context, a centroid prediction and aggregation module, a detection head, and an abnormal state evaluation module;

[0013] S300: Input the training set in the power scenario object detection data set into the 3D point cloud object detection and abnormal state evaluation model, process the point cloud data after multi-sensor fusion through the point set extraction module, and extract the local structure and point-to-point relationship of the point cloud data;

[0014] S400: The feature refinement module based on local context extracts target features using extended bounding box context clues, and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block;

[0015] S500: The context centroid prediction module obtains the core position of the target, estimates the instance center offset in combination with the representative points within the extended bounding box, and further improves the ability to capture global and local features of the point cloud target using the aggregation module;

[0016] S600: The accurate prediction of the target category and status is achieved through the detection head and the abnormal status evaluation module, classifying the abnormal status and quantitatively evaluating it;

[0017] S700: The parameters of the model are optimized using the multi-task loss function, the training set, and the validation set until the model converges, completing the model training. The test set is input into the trained model to evaluate the detection performance of the model, and the data in the test set is evaluated to obtain the test results.

[0018] Preferably, S100 includes:

[0019] S110: Deploy an unmanned aerial vehicle (UAV) acquisition device in the power scenario, install a lidar on the reserved position at the bottom of the UAV, and connect the lidar to the data acquisition device through a data cable to ensure that the lidar can be normally powered and transmit point cloud data;

[0020] S120: Start the lidar to collect point cloud data of the target device in the power scenario, adjust the lidar scanning parameters to obtain high-quality three-dimensional point cloud data covering the entire scenario, and save the data as a standard format file;

[0021] S130: Use a point cloud annotation tool to annotate 50% of the collected three-dimensional point cloud data. Among them, annotate the target categories in the point cloud data, and make detailed annotations on the normal status and common abnormal status of the target, generating corresponding annotation files, denoted as the dataset , and the remaining 50% of the data is not annotated, denoted as ;

[0022] S140: Divide the annotated dataset . Among them, randomly select 70% of the data as the training set, denoted as , for model training; use 30% of the data as the validation set, denoted as , for model performance evaluation; use the unannotated dataset as the test set, denoted as , for testing the generalization and detection effect of the model.

[0023] Preferably, the point set extraction module includes multiple set abstraction layers, and each set abstraction layer includes a sampling layer, a grouping layer, and a point network layer. S300 includes:

[0024] S310: Input the point cloud data after multi-sensor fusion, and represent the point cloud data as a vector of size N×4 (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data respectively, i is the reflection intensity of the point cloud data, and N is the number of points in the point cloud;

[0025] S320: Take an N×(d + C) matrix as the input, and this matrix comes from N points with d-dimensional coordinates and C-dimensional point features. Given the input points {x1, x2, …, x n} to the sampling layer, use iterative farthest point sampling to select a subset of points {x i1 , x i2 , …, x im} such that x ij is the point farthest from the set {x i1 , x i2 , …, x ij-1};

[0026] S330: Input a point set of size N×(d + C) and the coordinates of a set of centroids of size to the grouping layer for grouping, and the output is a point set group of size , where each group corresponds to a local area. Here, N is the number of points in the input point set, d is the spatial dimension, C is the feature dimension, is the number of centroids, and K is the number of points in the centroid point neighborhood;

[0027] S340: Input the local areas with a data size of to the point network layer, and convert the coordinates of the points in the local area into a local frame relative to the centroid point: , i = 1, 2, …, K and j = 1, 2, …, d, where is the coordinate of the centroid. Use the point network as the basic building block for local pattern learning, capture the point-to-point relationship in the local area through relative coordinates and point features, and each local area in the output is abstracted by its centroid and the local features encoding the centroid neighborhood. The output data volume is , where is the local feature dimension after point network abstraction.

[0028] Preferably, a feature refinement module based on local context is added after each set abstraction layer. The feature refinement module based on local context includes a local context fusion block and a global bilinear regularization block. S400 includes:

[0029] S410: Input the output of the point cloud layer into the local context fusion block. The local context fusion block follows the ball query algorithm to find the k neighbors of a certain point among all points and, under the metric of the 3D Euclidean distance between scattered points, define a local geometric graph in 3D space by combining the edges between itself and its k neighbors . Specifically: The local geometric graph of all points P is denoted as ;

[0030] ;

[0031] In addition, encode the local geometric graph through a shared and aggregate the local geometric context by applying a max pooling function on the k neighbors:

[0032] where is the shared MLP in the first local context fusion block to encode the local feature map; when implementing the shared MLP operation, it is specifically 1×1 convolution, batch normalization, and activation layer processing;

[0033] ;

[0034] where is the shared MLP in the first local context fusion block to encode the local feature map; when implementing the shared MLP operation, it is specifically 1×1 convolution, batch normalization, and activation layer processing;

[0035] S420: Parallelly, the local feature graph in the C-dimensional space is formed as: ;

[0036] ;

[0037] where is the corresponding feature of ;

[0038] The local feature map of the feature map F is specifically:

[0039] ;

[0040] Encode the local feature map through a shared and aggregate the local feature context by applying a max pooling function on the k neighbors, specifically:

[0041] ;

[0042] where is another shared MLP to encode the local feature map;

[0043] S430: Combine the local geometric context and the local feature context Connect them as the output of the local context fusion block :

[0044] ;

[0045] S440: Input the output of the local context fusion block into the global bilinear regularization block and apply the weight matrix to reduce the channel dimension of the fused output , where r is a reduction factor; then use the ReLU function to provide the non-linearity after linear mapping and the non-negativity requirement of the low-rank global bilinear response; finally, compress the spatial information into a global channel descriptor by performing average pooling on N elements along the spatial axis : :

[0046] ;

[0047] Use another weight matrix , the ReLU function, and average pooling on elements along the channel axis to generate the global point direction descriptor :

[0048] ;

[0049] Capture the low-rank global bilinear response by taking the square root of the outer product of and :

[0050] ;

[0051] S450: After sharing and two shortcut connections, finally restore the channel dimension and generate a full-size global perception map:

[0052] ;

[0053] Subtract the local context fusion output from , and use the activation to add more non-linearity to the final output feature map to obtain the final output of the local context-based feature refinement module:

[0054] .

[0055] Preferably, S500 includes:

[0056] S510: Explicitly predict the offset of each point in the point cloud to the center of the target instance, and minimize the prediction uncertainty through regularization constraints, and its loss function is defined as:

[0057] ;

[0058] In the formula, represents the true offset of point to the center point, is an indicator function used to determine whether point is used to estimate the instance center, is the number of points used to predict the instance center, that is, the effective point set screened within the expanded true bounding box, and the centroid is predicted using the surrounding representative points in the context, represents the foreground points, that is, the points belonging to the target instance, is the predicted offset, is the average predicted value of the instance points;

[0059] S520: For the shifted representative centroid points, use the Point-Net++ module to learn the latent representation of each instance, convert the adjacent points to a local canonical coordinate system with the centroid as the origin, then extract the point-by-point features through a shared MLP, and aggregate the point features through a symmetric function.

[0060] Preferably, S600 includes:

[0061] S610: Feed the aggregated centroid point features into the proposal generation head to generate prediction candidate boxes containing the target position, size, and orientation, and simultaneously predict the category and status of the target, where the status includes normal status and abnormal status;

[0062] S620: Combine the prediction results of the abnormal status to preliminarily filter the prediction candidate boxes, apply 3D-NMS processing to the preliminarily screened candidate boxes and then filter them to generate preliminary detection results;

[0063] S630: Further analyze the bounding boxes filtered by 3D-NMS processing to obtain the final detection results; among them, the final detection results include the target category, predicted box position, size, orientation, and the status information of the target, where the status information of the target includes normal and abnormal; for the abnormal status, based on the tilt angle of the bounding box or the internal point cloud distribution pattern, calculate the tilt value of the target structure, generate a quantization result, and output the confidence score of the abnormal event.

[0064] Preferably, S700 includes:

[0065] S710: The multi-task loss is used in the model for joint optimization, and the total loss includes the downsampling strategy loss and the centroid prediction loss and the classification loss Bounding box generation loss :

[0066] ;

[0067] The bounding box generation loss is further decomposed into position, size, angle-bin, angle-res, and corner-to-corner parts:

[0068] ;

[0069] S720: Input the training set of the large-scale assembly scene point cloud dataset into the model, optimize the model parameters through the loss function. After the training is completed, use the validation set to evaluate the performance of the hyperparameters, and adjust the network hyperparameters according to the results to find the optimal local hyperparameter solution, so as to obtain the network model with the best performance under the current dataset.

[0070] Preferably, in S700, the data in the test set is evaluated, including:

[0071] S730: Test set Input the trained 3D detection model, and obtain the test results after the model inference is completed ;

[0072] S740: Download and install the open3d platform, and perform offline visualization based on the obtained test result file and the original point cloud file of the dataset on the open3d platform to verify the accuracy of the detection results and the generalization ability of the model.

[0073] 3D point cloud object detection and abnormal state evaluation system, including a dataset acquisition module, a 3D point cloud object detection and abnormal state evaluation model construction module, a point set extraction module, a local context-based feature refinement module, a centroid prediction module and an aggregation module, a detection head and an abnormal state evaluation module, a training and testing module;

[0074] The dataset acquisition module is used to build a 3D perception platform based on lidar, collect the 3D point cloud data of the target equipment in the power scene, perform data annotation and division on the collected 3D point cloud data, make a power scene object detection dataset, and divide it into a training set, a validation set, and a test set;

[0075] The 3D point cloud object detection and abnormal state evaluation model construction module is used to construct a 3D point cloud object detection and abnormal state evaluation model, and the model includes a point set extraction module, a local context-based feature refinement module, a centroid prediction and aggregation module, a detection head and an abnormal state evaluation module, a training and testing module;

[0076] The point set extraction module is used to receive the training set in the power scene target detection dataset, process the point cloud data after multi-sensor fusion, and extract the local structure and point-to-point relationship of the point cloud data;

[0077] The feature refinement module based on local context extracts target features using extended bounding box context clues and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block;

[0078] The centroid prediction module obtains the core position of the target, estimates the instance center offset in combination with the representative points within the extended bounding box, and further improves the ability to capture the global and local features of the point cloud target using the aggregation module;

[0079] The detection head and abnormal state evaluation module realizes accurate prediction of the target category and state, classifies abnormal states and quantitatively evaluates them;

[0080] The training and testing module uses a multi-task loss function, the training set and the validation set to optimize the model parameters until the model converges, completes the model training, inputs the test set into the trained model, evaluates the detection performance of the model, evaluates the data in the test set, and obtains the test results.

[0081] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the three-dimensional point cloud target detection and abnormal state evaluation method are implemented.

[0082] The above three-dimensional point cloud target detection and abnormal state evaluation method, system and device can perform high-precision three-dimensional target detection and abnormal state evaluation on target devices (such as wires, poles, etc.) in the power scene, including the category, position, direction of the target and its normal or abnormal state (such as pole tilt, wire breakage, foreign object hanging, etc.). This method provides reliable data support for the intelligent operation and maintenance of power equipment, helps to improve the operation safety and stability of the power system, reduces the cost of manual inspection at the same time, and promotes the transformation and development of the power industry towards intelligence and unmanned operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flowchart of a three-dimensional point cloud target detection and abnormal state evaluation method in an embodiment of the present invention;

[0084] Figure 2 It is a schematic diagram of a data acquisition platform and a three-dimensional point cloud target detection and abnormal state evaluation model in an embodiment of the present invention;

[0085] Figure 3 It is a schematic diagram of the principle of the feature refinement module based on local context in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0086] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0087] In one embodiment, as Figure 1 and Figure 2 shown, a three-dimensional point cloud object detection and abnormal state evaluation method, the method comprising the following steps:

[0088] S100: Build a three-dimensional perception platform for the power scene based on lidar, collect three-dimensional point cloud data of target devices (such as wires, poles, etc.) in the power scene, perform data annotation and division on the collected three-dimensional point cloud data, make a target detection data set for the power scene, and divide it into a training set, a validation set and a test set; Further, the data annotation includes the normal state and common abnormal states of wires and poles (such as broken wires, foreign object hanging, pole tilt or damage, etc.);

[0089] S200: Construct a three-dimensional point cloud object detection and abnormal state evaluation model, the model including a point set extraction module, a feature refinement module based on local context, a centroid prediction and aggregation module, a detection head and an abnormal state evaluation module;

[0090] S300: Input the training set in the power scene target detection data set into the three-dimensional point cloud object detection and abnormal state evaluation model, process the point cloud data after multi-sensor fusion through the point set extraction module, and extract the local structure and point-to-point relationship of the point cloud data;

[0091] S400: The feature refinement module based on local context extracts target features using extended bounding box context clues, and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block;

[0092] S500: Obtain the core position of the target through the context centroid prediction module, and estimate the instance center offset in combination with the representative points within the extended bounding box, and further improve the capture ability of the global and local features of the point cloud object using the aggregation module;

[0093] S600: The detection head and the abnormal state evaluation module achieve accurate prediction of the target category and state, classify the abnormal state and quantitatively evaluate it;

[0094] S700: Use the multi-task loss function, the training set and the validation set to optimize the parameters of the model until the model converges, complete the model training, input the test set into the trained model, evaluate the detection performance of the model, including the detection accuracy of different category targets, the positioning error and the abnormal state detection accuracy, evaluate the data in the test set, and obtain the test results.

[0095] Specifically, by using the built lidar three-dimensional perception platform to collect power scene point cloud data, a three-dimensional target detection and abnormal state evaluation model for power scenes is constructed. The feature refinement module based on local context enhances the feature expression ability by fusing local geometry and feature context information, capturing both the details of the point cloud and integrating global information. The detection head and abnormal state evaluation module achieve accurate prediction of target categories and states, classify abnormal states (such as wire breakage, foreign object hanging, inclination, etc.) and quantitatively evaluate them. By analyzing the inclination angle of the prediction box or the point cloud distribution, the inclination angle of the pole tower is accurately calculated, and detailed abnormal information such as "the pole tower is inclined by 15°" is output, providing support for equipment state evaluation. Through testing on the built lidar three-dimensional perception platform for power scenes, the results show that this method can achieve high-precision detection and rapid inference of target devices, and provide detailed abnormal state evaluation information, providing strong technical support for the intelligent operation and maintenance of power systems, and significantly improving the efficiency of power equipment fault troubleshooting and the overall operation and maintenance quality.

[0096] In one embodiment, S100 includes:

[0097] S110: Deploy an unmanned aerial vehicle (UAV) data collection device in the power scene, install the lidar at the reserved position at the bottom of the UAV, and connect the lidar to the data collection device through a data cable to ensure that the lidar can be normally powered and transmit point cloud data;

[0098] S120: Start the lidar to collect point cloud data of target devices in the power scene, adjust the lidar scanning parameters to obtain high-quality three-dimensional point cloud data covering the entire scene, and save the data as a standard format file (such as.pcd) for subsequent processing;

[0099] S130: Use a point cloud annotation tool (such as labelcloud or other software supporting 3D point cloud annotation) to annotate 50% of the collected three-dimensional point cloud data. Among them, annotate the target categories (such as wires, pole towers) in the point cloud data, and detail the normal state and common abnormal states (such as wire breakage, foreign object hanging, pole tower inclination or damage, etc.) of the targets to generate corresponding annotation files, denoted as the dataset , and the remaining 50% of the data is not annotated, denoted as ;

[0100] S140: Divide the annotated dataset . Among them, randomly select 70% of the data as the training set, denoted as , for model training; use 30% of the data as the validation set, denoted as , for model performance evaluation; use the unannotated dataset as the test set, denoted as , which is used to test the generalization and detection effect of the model.

[0101] Specifically, the schematic diagram of the 3D object detection and abnormal state evaluation model for the power scenario is as Figure 2 shown.

[0102] In one embodiment, the point set extraction module includes multiple set abstraction layers, and each set abstraction layer includes a sampling layer, a grouping layer, and a point network layer. S300 includes:

[0103] S310: Input the point cloud data after multi-sensor fusion, and represent the point cloud data as a vector of size N×4 (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data respectively, i is the reflection intensity of the point cloud data, and N is the number of point clouds;

[0104] S320: Take an N × (d + C) matrix as the input, which comes from N points with d-dimensional coordinates and C-dimensional point features. Given the input points {x1, x2,..., x n} to the sampling layer, use iterative farthest point sampling to select a subset of points {x i1 , x i2 ,..., x im}, such that x ij is the point farthest from the set {x i1 , x i2 ,..., x ij-1};

[0105] S330: Input a point set of size N×(d + C) and the coordinates of a set of centroids of size into the grouping layer for grouping, and the output is a point set group of size , where each group corresponds to a local area. Here, N is the number of points in the input point set, d is the spatial dimension, C is the feature dimension, is the number of centroids, and K is the number of points in the centroid point neighborhood;

[0106] S340: Input the local areas with a data size of into the point network layer, and convert the coordinates of the points in the local area into a local frame relative to the centroid point: , i = 1, 2,..., K and j = 1, 2,..., d, where is the coordinate of the centroid. Use the point network as the basic building block for local pattern learning, capture the point-to-point relationship in the local area through relative coordinates and point features, and each local area in the output is abstracted by its centroid and the local features encoding the centroid neighborhood. The output data volume is , where is the local feature dimension after point network abstraction.

[0107] In one embodiment, as Figure 3 shown, a feature refinement module based on local context is added after each set abstraction layer. The feature refinement module of local context includes a local context fusion block and a global bilinear regularization block. S400 includes:

[0108] S410: Input the output of the point net layer into the local context fusion block. The local context fusion block follows the ball query algorithm to find the k neighbors of a certain point among all points . Under the metric of the 3D Euclidean distance between scattered points, by combining the edges between itself and its k neighbors , a local geometric graph in 3D space is defined . Specifically:

[0109] ;

[0110] The local geometric graph of all points P is represented as ;

[0111] In addition, the local geometric graph is encoded by sharing and the local geometric context is aggregated by applying a max pooling function on the k neighbors:

[0112] ;

[0113] where is the shared MLP in the first local context fusion block to encode the local feature map; when implementing the shared MLP operation, it is specifically 1×1 convolution, batch normalization, and activation layer processing;

[0114] S420: Parallelly, the local feature graph in the C-dimensional space becomes:

[0115] ;

[0116] where is the corresponding feature of ;

[0117] The local feature graph of the feature map F is specifically:

[0118] ;

[0119] The local feature graph is encoded by sharing and the local feature context is aggregated by applying a max pooling function on the k neighbors. Specifically:

[0120] ;

[0121] where is another shared MLP-encoded local feature map;

[0122] S430: Concatenate the local geometric context and the local feature context as the output of the local context fusion block :

[0123] ;

[0124] S440: Input the output of the local context fusion block into the global bilinear regularization block, apply the weight matrix to reduce the channel dimension of the fused output where r is a reduction factor; then use the ReLU function to provide the non-linearity after the linear mapping and the non-negativity requirement of the low-rank global bilinear response; finally, compress the spatial information into a global channel descriptor by performing average pooling operation on N elements along the spatial axis: :

[0125] ;

[0126] Use another weight matrix , the ReLU function, and average pooling operation on elements along the channel axis to generate the global point direction descriptor :

[0127] ;

[0128] Capture the low-rank global bilinear response by taking the square root of the outer product of and :

[0129] ;

[0130] S450: After sharing and two shortcut connections, finally restore the channel dimension and generate a full-size global perception map:

[0131] ;

[0132] Subtract the local context fusion output from , and use the activation to add more non-linearity in the final output feature map to obtain the final output of the local context-based feature refinement module:

[0133] .

[0134] In one embodiment, S500 includes:

[0135] S510: Explicitly predict the offsets of each point in the point cloud to the center of the target instance , and minimize the prediction uncertainty through regularization constraints, and its loss function is defined as:

[0136] ;

[0137] In the formula, represents the true offset of point to the center point, is an indicator function used to determine whether point is used to estimate the instance center, is the number of points used to predict the instance center, that is, the effective point set screened within the expanded true bounding box, and the centroid is predicted using the surrounding representative points in the context, represents the foreground points, that is, the points belonging to the target instance, is the predicted offset, is the average predicted value of the instance points;

[0138] S520: For the shifted representative centroid points, use the Point-Net++ module to learn the latent representation of each instance, convert the adjacent points to a local canonical coordinate system with the centroid as the origin, then extract the point-by-point features through a shared MLP, and aggregate the point features through a symmetric function.

[0139] Specifically, the initial sampling points refer to the representative seed points obtained by farthest point sampling from the original point cloud, and these points serve as the reference points for subsequent instance centroid prediction. Each seed point predicts an offset pointing to the target center, and finally the instance centroid is obtained through clustering and voting points. The sampling points in this network not only include the seed points within the bounding box, but also aggregate the surrounding points by expanding the context area (for example, the neighborhood radius is expanded to 1.5 times the bounding box size) to enhance the robustness to occlusion and sparse regions.

[0140] In one embodiment, S600 includes:

[0141] S610: Feed the aggregated centroid point features into the proposal generation head to generate prediction candidate boxes including the target position, size, and orientation, and simultaneously predict the category and status of the target, where the status includes normal status and abnormal status;

[0142] S620: Combine the prediction results of the abnormal status to preliminarily filter the prediction candidate boxes, apply 3D-NMS processing to the preliminarily screened candidate boxes and then filter them to generate preliminary detection results;

[0143] S630: Further analyze the bounding boxes filtered by 3D-NMS processing to obtain the final detection results. Among them, the final detection results include the target category, the position, size, orientation of the prediction box, and the status information of the target. Among them, the status information of the target includes normal and abnormal. For abnormal status detection, by analyzing the tilt angle of the bounding box or the distribution pattern of the point cloud within the bounding box, accurately calculate the tilt angle of the pole tower and output the abnormal status information, such as "The tilt angle of the pole tower is 15°". In addition, the detection results also include the confidence scores of other abnormal statuses (such as broken wires or foreign object hanging) for users to make subsequent decision support.

[0144] In one embodiment, S700 includes:

[0145] S710: Multitask loss is used in the model for joint optimization, and the total loss includes downsampling strategy loss 、centroid prediction loss 、classification loss and box generation loss :

[0146] ;

[0147] The box generation loss is further decomposed into position, size, angle-bin, angle-res, and corner parts:

[0148] ;

[0149] S720: Input the training set of the large-scale assembly scene point cloud dataset into the model, optimize the model parameters through the loss function. After training, use the validation set to evaluate the performance of the hyperparameters, and adjust hyperparameters such as the network learning rate, batch size, and feature dimension according to the results to find the optimal local hyperparameter solution, so as to obtain the network model with the best performance under the current dataset.

[0150] In one embodiment, the evaluation of the data in the test set in S700 includes:

[0151] S730: The test set is input into the trained 3D detection model, and after the model inference is completed, the test results are obtained ;

[0152] S740: Download and install the open3d platform, and perform offline visualization based on the obtained test result file and the original point cloud file of the dataset on the open3d platform to verify the accuracy of the detection results and the generalization ability of the model.

[0153] In the power scenario, the efficiency and accuracy of equipment detection and status assessment are crucial. Currently, the power system is undergoing a transformation towards intelligence and unmanned operation, and the three-dimensional object detection ability of transmission lines and equipment directly affects the efficiency and reliability of intelligent operation and maintenance. Therefore, it is necessary to construct a high-precision three-dimensional detection and abnormal status assessment model for the power scenario. By carrying lidar to obtain the point cloud data of the power scenario, rich spatial and position information can be provided for equipment detection and status assessment. However, most current point cloud three-dimensional detection methods have limitations in long-distance detection and abnormal status recognition, and it is difficult to accurately locate and identify the status and abnormal features of special targets (such as poles), thus restricting the detection accuracy and efficiency. Therefore, the innovations of the present invention are as follows:

[0154] (1) The present invention proposes a three-dimensional object detection and abnormal assessment method for the power scenario, which can perform high-precision three-dimensional detection on targets such as poles and power lines, and evaluate the target status (such as inclination, broken wire or foreign object hanging), providing comprehensive spatial and status information for power equipment operation and maintenance, and liberating human resources; (2) Aiming at the diversity and complexity of targets in the power scenario, a method for refining the extraction of point cloud features combined with local context is proposed. The target features are extracted by using the extended bounding box context clues, and the feature expression ability is enhanced through a shared multi-layer perceptron (MLP) and a bilinear regularization block, while optimizing the training and inference efficiency; (3) Solve the problems of long-distance and detailed feature detection of power scenario targets, and accurately depict the geometric characteristics of targets through centroid prediction and aggregation modules. The core position of the target is obtained through context centroid prediction, and the instance center offset is estimated by combining the representative points within the extended bounding box. The aggregation module is used to further improve the ability to capture the global and local features of the point cloud target, thereby improving the detection accuracy of the target, especially in the evaluation of the inclination angle and abnormal status of poles; (4) Adapt to the efficient detection requirements of large-scale power scenarios, and use the feature extraction and refinement module to fully exploit the spatial and context information of the point cloud, ensuring high-precision three-dimensional detection and abnormal status assessment of multiple targets in the power scenario under lidar sensors. The model can balance efficiency while maintaining detection accuracy and is applicable to complex actual power operation and maintenance scenarios; (5) The model of the present invention is efficient, robust and scalable, and is not only applicable to power system scenarios, but also can be extended to three-dimensional detection and status assessment in other complex industrial scenarios (such as energy, transportation, etc.), with wide practical application value, contributing to the high-quality development of intelligent operation and maintenance and the industrial field.

[0155] Three-dimensional point cloud object detection and abnormal status assessment system, including a data set acquisition module, a three-dimensional point cloud object detection and abnormal status assessment model construction module, a point set extraction module, a feature refinement module based on local context, a centroid prediction module and an aggregation module, a detection head and an abnormal status assessment module, a training and testing module;

[0156] A dataset acquisition module for building a 3D perception platform for power scenarios based on lidar, collecting 3D point cloud data of target devices in power scenarios, performing data annotation and division on the collected 3D point cloud data, creating a target detection dataset for power scenarios, and dividing it into a training set, a validation set, and a test set;

[0157] A 3D point cloud target detection and abnormal state evaluation model construction module for constructing a 3D point cloud target detection and abnormal state evaluation model, which includes a point set extraction module, a feature refinement module based on local context, a centroid prediction and aggregation module, a detection head and an abnormal state evaluation module, and a training and testing module;

[0158] The point set extraction module is used to receive the training set in the power scenario target detection dataset, process the point cloud data after multi-sensor fusion, and extract the local structure and point-to-point relationship of the point cloud data;

[0159] The feature refinement module based on local context extracts target features using extended bounding box context clues and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block;

[0160] Obtain the core position of the target through the centroid prediction module, estimate the instance center offset in combination with the representative points within the extended bounding box, and use the aggregation module to further improve the ability to capture the global and local features of the point cloud target;

[0161] The detection head and abnormal state evaluation module realizes accurate prediction of target categories and states, classifies abnormal states and quantitatively evaluates them;

[0162] The training and testing module uses a multi-task loss function, the training set and the validation set to optimize the parameters of the model until the model converges, completes the model training, inputs the test set into the trained model, evaluates the detection performance of the model, and evaluates the data in the test set to obtain the test results.

[0163] For the specific limitations of the 3D point cloud target detection and abnormal state evaluation system, reference can be made to the limitations of the 3D point cloud target detection and abnormal state evaluation method in the above text, which will not be elaborated here. Each module in the above 3D point cloud target detection and abnormal state evaluation system can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0164] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it realizes the steps of the 3D point cloud target detection and abnormal state evaluation method.

[0165] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0166] The above has introduced in detail the three-dimensional point cloud target detection and abnormal state evaluation method, system, and device provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for three-dimensional point cloud object detection and abnormal state evaluation, characterized in that, The method includes the following steps: S100: Build a 3D perception platform for power scenarios based on lidar, collect 3D point cloud data of target devices in the power scenario, perform data annotation and division on the collected 3D point cloud data, make a target detection dataset for the power scenario, and divide it into a training set, a validation set, and a test set; S200: Construct a 3D point cloud target detection and abnormal state evaluation model, which includes a point set extraction module, a feature refinement module based on local context, a centroid prediction and aggregation module, a detection head, and an abnormal state evaluation module; S300: Input the training set in the power scenario target detection dataset into the 3D point cloud target detection and abnormal state evaluation model, process the point cloud data after multi-sensor fusion through the point set extraction module, and extract the local structure and point-to-point relationship of the point cloud data; the point set extraction module includes multiple set abstraction layers, and each set abstraction layer includes a sampling layer, a grouping layer, and a point network layer; S400: The feature refinement module based on local context extracts target features using extended bounding box context clues, and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block; a feature refinement module based on local context is added after each set abstraction layer, and the feature refinement module of local context includes a local context fusion block and a global bilinear regularization block; S500: Obtain the core position of the target through the context centroid prediction module, estimate the instance center offset in combination with the representative points within the extended bounding box, and use the aggregation module to further improve the ability to capture the global and local features of the point cloud target; S500 includes: S510: Explicitly predict the offset of each point in the point cloud to the target instance center, and minimize the prediction uncertainty through regularization constraints; S520: For the shifted representative centroid points, use the Point-Net++ module to learn the latent representation of each instance, convert the adjacent points to a local canonical coordinate system with the centroid as the origin, then extract point-by-point features through a shared MLP, and aggregate the point features through a symmetric function; S600: Achieve accurate prediction of the target category and state through the detection head and the abnormal state evaluation module, classify the abnormal state and quantitatively evaluate it; S700: Use a multi-task loss function, the training set, and the validation set to optimize the parameters of the model until the model converges, complete the model training, input the test set into the trained model, evaluate the detection performance of the model, evaluate the data in the test set, and obtain the test results.

2. The method according to claim 1, characterized in that, S100 includes: S110: Deploy a drone acquisition device in the power scenario, install the lidar at the reserved position at the bottom of the drone, and connect the lidar to the data acquisition device through a data cable to ensure that the lidar can be powered normally and transmit point cloud data; S120: Start the lidar to collect point cloud data of the target device in the power scenario, adjust the lidar scanning parameters to obtain high-quality 3D point cloud data covering the entire scenario, and save the data as a standard format file; S130: Use a point cloud annotation tool to annotate 50% of the collected three-dimensional point cloud data. Among them, annotate the target categories in the point cloud data, and make detailed annotations on the normal state and common abnormal states of the targets to generate corresponding annotation files, denoted as the dataset , and do not annotate the remaining 50% of the data, denoted as ; S140: For the labeled dataset make a division, randomly select 70% of the data as the training set, denoted as , for model training; use 30% of the data as the validation set, denoted as , for model performance evaluation; use the unlabeled dataset as the test set, denoted as , for testing the generalization and detection effect of the model.

3. The method according to claim 2, wherein S300 includes: S310: Input the point cloud data after multi-sensor fusion, represent the point cloud data as a vector of size N×4 (x, y, z, i), where (x, y, z) are the three-dimensional position coordinates of the point cloud data respectively, i is the reflection intensity of the point cloud data, and N is the number of point clouds; S320: Take an N × (d + C) matrix as input, which comes from N points with d-dimensional coordinates and C-dimensional point features. Given the input points {x1, x2, …, x n} to the sampling layer, use iterative farthest point sampling to select a subset of points {x i1 , x i2 , …, x im} such that x ij is the point farthest from the set {x i1 , x i2 , …, x ij-1}; S330: Input a point set of size N×(d + C) and the coordinates of a set of centroids of size into the grouping layer for grouping, and the output is a point set group of size , where each group corresponds to a local area. Here, N is the number of points in the input point set, d is the spatial dimension, C is the feature dimension, is the number of centroids, and K is the number of points in the neighborhood of the centroid points; S340: Input the local region input point net layer with a data size of and convert the coordinates of the points in the local region into a local frame relative to the centroid point: , where i = 1, 2, …, K and j = 1, 2, …, d, and is the coordinate of the centroid. Use the point net as the basic building block for local pattern learning, capture the point-to-point relationship in the local region through relative coordinates and point features, and each local region in the output is abstracted by its centroid and local features encoding the centroid neighborhood. The output data volume is , where , and is the local feature dimension after point net abstraction.

4. The method according to claim 3, wherein S400 includes: S410: Input the output of the point cloud layer into the local context fusion block. The local context fusion block follows the ball query algorithm to find the k neighbors of a certain point among all points and, under the measure of the 3D Euclidean distance between scattered points, define a local geometric graph in 3D space by combining the edges between itself and its k neighbors . Specifically: ​ ； The local geometry of all points P is represented as ; In addition, by sharing encode the local geometry graph and aggregate the local geometric context by applying a max pooling function over k neighbors: ； Among them, is the MLP-encoded local feature map shared in the first local context fusion block; when implementing the shared MLP operation, it specifically involves 1×1 convolution, batch normalization, and activation layer processing; S420: In parallel, the local feature pattern in the C-dimensional space becomes: ； Among them, is the corresponding feature; The local feature map of the feature map F is specifically: ； By sharing Encode the local feature map and aggregate the local feature context by applying a max pooling function over k neighbors, specifically: ； Among them is another shared MLP-encoded local feature map; S430: Connect the local geometric context and the local feature context to be the output of the local context fusion block : ； S440: Input the output of the local context fusion block into the global bilinear regularization block, and apply the weight matrix to reduce the fused output in channel dimension, where r is a reduction factor; then utilize the ReLU function to provide the non-linearity after the linear mapping and the non-negativity requirement of the low-rank global bilinear response; finally, compress the spatial information into a global channel descriptor by performing average pooling operation on N elements along the spatial axis : ； Use another weight matrix , the ReLU function, and average pooling operation on the elements along the channel axis to generate a global point direction descriptor : ； By taking and the square root of the outer product to capture the low-rank global bilinear response: ； S450: After sharing and two shortcut connections, the channel dimension was finally restored, and a full-size global perception map was generated: ； Fuse and output the local context Subtract , and use activation to add more non - linearity to the feature map of the final output, obtaining the final output of the local - context - based feature refinement module: 。 5. The method according to claim 4, wherein S510 is specifically: ； wherein, represents the true offset of the point to the center point, is an indicator function for determining whether the point is used to estimate the instance center, is the number of points used to predict the instance center, that is, the set of valid points filtered within the expanded true bounding box, and the centroid is predicted using the surrounding representative points in the context, represents the foreground points, i.e., the points belonging to the target instance, is the predicted offset, is the average predicted value of the instance points.

6. The method according to claim 5, wherein S600 includes: S610: Feed the aggregated centroid point features into the proposal generation head to generate predicted candidate boxes containing the target position, size, and orientation, and simultaneously predict the category and status of the target, where the status includes normal status and abnormal status; S620: Combine the prediction results of the abnormal status to preliminarily filter the predicted candidate boxes, apply 3D-NMS processing to the preliminarily screened candidate boxes and then filter them to generate preliminary detection results; S630: Further analyze the bounding boxes filtered by 3D-NMS processing to obtain the final detection results; among them, the final detection results include the target category, predicted box position, size, orientation, and the status information of the target, where the status information of the target includes normal and abnormal; for the abnormal status, based on the tilt angle of the bounding box or the internal point cloud distribution pattern, calculate the tilt value of the target structure, generate a quantization result, and output the confidence score of the abnormal event.

7. The method according to claim 6, wherein S700 includes: S710: The multitask loss is used in the model for joint optimization, and the total loss includes the downsampling strategy loss , the centroid prediction loss , the classification loss and the box generation loss : ； The box generation loss is further decomposed into position, size, angle-bin, angle-res, and corner parts: ； S720: Input the training set of the large-scale assembly scene point cloud dataset into the model, optimize the model parameters through the loss function, after training is completed, use the validation set to evaluate the performance of the hyperparameters, and adjust the network hyperparameters according to the results to find the optimal local hyperparameter solution, so as to obtain the network model with the best performance under the current dataset.

8. The method according to claim 7, characterized in that, The evaluation of the data in the test set in S700 includes: S730: Test set Input the trained 3D detection model, and obtain the test results after the model inference ; S740: Download and install the open3d platform, and perform offline visualization based on the obtained test result file and the original point cloud file of the dataset on the open3d platform to verify the accuracy of the detection results and the generalization ability of the model.

9. A three-dimensional point cloud object detection and abnormal state evaluation system using the method according to any one of claims 1 to 8, characterized in that, It includes a dataset acquisition module, a three-dimensional point cloud target detection and abnormal status evaluation model construction module, a point set extraction module, a feature refinement module based on local context, a centroid prediction module and an aggregation module, a detection head and an abnormal status evaluation module, a training and testing module; The dataset acquisition module is used to build a three-dimensional perception platform for power scenes based on lidar, collect the three-dimensional point cloud data of target devices in power scenes, perform data annotation and division on the collected three-dimensional point cloud data, make a power scene target detection dataset, and divide it into a training set, a validation set, and a test set; The three-dimensional point cloud target detection and abnormal status evaluation model construction module is used to construct a three-dimensional point cloud target detection and abnormal status evaluation model, and the model includes a point set extraction module, a feature refinement module based on local context, a centroid prediction and aggregation module, a detection head and an abnormal status evaluation module, a training and testing module; The point set extraction module is used to receive the training set in the power scenario object detection dataset, process the point cloud data after multi-sensor fusion, and extract the local structure and point-to-point relationship of the point cloud data; The feature refinement module based on local context extracts object features using extended bounding box context clues, and enhances the feature expression ability through a shared multi-layer perceptron and a bilinear regularization block; The centroid prediction module obtains the core position of the object, estimates the instance center offset in combination with the representative points within the extended bounding box, and further improves the ability to capture the global and local features of the point cloud object using the aggregation module; The detection head and abnormal state evaluation module realizes accurate prediction of object categories and states, classifies abnormal states and quantitatively evaluates them; The training and testing module uses a multi-task loss function, a training set and a validation set to optimize the parameters of the model until the model converges, completes the model training, inputs the test set into the trained model, evaluates the detection performance of the model, and evaluates the data in the test set to obtain the test results.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images