Power transmission line R-type closed pin loss detection method and system based on multi-sensor fusion

Abstract

The invention relates to the field of power equipment inspection, and discloses a power transmission line R-type closed pin loss detection method and system based on multi-sensor fusion, and the method comprises the steps: synchronously collecting the image and point cloud data of a power transmission line through multiple sensors, carrying out the preprocessing, and achieving the data collection and preprocessing; establishing a mapping relation between the preprocessed image and the point cloud, coarsely positioning a key area based on point cloud spatial features, projecting the key area to the image to obtain an area of interest, and realizing data fusion and target coarse positioning; performing fine identification and positioning on the R-type closed pin in the region of interest by adopting a target detection network to realize fine identification and positioning; associating the point cloud cluster corresponding to the recognition target, calculating the three-dimensional geometric features of the point cloud cluster, and inputting the three-dimensional geometric features into a classifier to judge the in-place or missing state of the R-type closed pin; integrating the three-dimensional coordinates; the defect of single visual inspection is effectively overcome, and all-weather, high-precision and automatic inspection and state diagnosis of the R-type closed pin of the power transmission line are realized.

Description

Technical Field

[0001] This invention relates to the field of power equipment inspection, and in particular to a method and system for detecting missing R-type closed pins in transmission lines based on multi-sensor fusion. Background Technology

[0002] Transmission lines are the backbone of the power system, and their safe and stable operation is crucial. R-type closed pins, as key components in transmission line suspension hardware used for locking pins, are small in size but play an irreplaceable role. If they are missing or detached, the pins can easily come loose, potentially leading to serious power grid accidents such as conductor slippage or falls, threatening power supply safety. Currently, the inspection of R-type closed pins mainly relies on traditional manual ground inspections or telescope inspections. These methods are not only inefficient but also highly dependent on the inspectors' eyesight and experience, especially in complex terrain conditions, leading to a high risk of missed inspections and misjudgments. With the development of drone technology, using drones to take high-definition photos followed by manual interpretation has improved data collection efficiency to some extent. However, the massive amounts of image data still require significant manpower for screening and identification, resulting in low automation and detection speeds that still cannot meet the needs of large-scale intelligent power grid inspections. In recent years… In addition, some research has emerged on automatic recognition technologies based on single visible light images, attempting to use deep learning object detection algorithms for recognition. However, these methods have obvious limitations: their recognition performance is easily affected by factors such as lighting, weather, and background complexity. The recognition rate drops significantly in strong light, shadow, backlight, or when the R-shaped pin is similar in color to the background. At the same time, relying solely on two-dimensional color and texture features, the model's generalization ability and robustness are insufficient for R-shaped pins with varied shapes, extremely small sizes, or partial occlusion. More importantly, it is difficult to accurately distinguish between "present" and "missing" states based solely on two-dimensional image information. It is easy to misjudge normal R-shaped pins that are not obvious in the image due to shooting angle as missing, resulting in a high false alarm rate. Summary of the Invention

[0003] In view of the problems existing in the prior art, the present invention is proposed.

[0004] The above-mentioned technical problems are solved by the following technical solution: This invention proposes a method and system for detecting missing R-type closed pins in transmission lines based on multi-sensor fusion. The method includes: simultaneously acquiring images and point cloud data of the transmission line through multiple sensors, and preprocessing them separately to achieve data acquisition and preprocessing; establishing a mapping relationship between the preprocessed images and point clouds, coarsely locating key areas based on the spatial features of the point clouds and projecting them onto the images to obtain regions of interest, thus achieving data fusion and coarse target localization; using a target detection network to perform fine identification and localization of R-type closed pins within the region of interest, thus achieving fine identification and localization; associating the point cloud clusters corresponding to the identified targets, calculating their three-dimensional geometric features, and inputting them into a classifier to determine the presence or absence status of the R-type closed pins; integrating the three-dimensional coordinates, identification results, and status judgment results to generate a detection report and provide early warnings for missing statuses, thus achieving result output and early warning.

[0005] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines according to the present invention, the multi-sensor includes a synchronously triggered visible light camera and a lidar.

[0006] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines according to the present invention: images and point cloud data of the transmission line are simultaneously acquired by multiple sensors, including visible light images and laser point cloud data.

[0007] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines according to the present invention: the target detection network is a YOLOv10 network with a coordinate attention mechanism.

[0008] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines according to the present invention, the three-dimensional geometric features include point cloud density, normal vector distribution variance, and local depth change rate.

[0009] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closing pins in transmission lines according to the present invention, the point cloud density is calculated using the following formula: Where D is the point cloud density, N is the number of points in the point cloud cluster P, and V is the bounding box volume of the point cloud cluster.

[0010] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closing pins in transmission lines according to the present invention, the formula for calculating the variance of the normal vector distribution is as follows: in, s 2 normalLet be the variance of the normal vector distribution, N be the number of points in the point cloud cluster P, ni be the normal vector of point i in the point cloud cluster, and μ be the mean of all normal vectors.

[0011] In a preferred embodiment of the multi-sensor fusion-based method for detecting missing R-type closing pins in transmission lines according to the present invention, the formula for calculating the local depth change rate is as follows: in, s depth Let N be the local depth change rate, and N be the number of points in the point cloud cluster P. z i For point i The depth value, m z This is the average depth value for all points.

[0012] In a preferred embodiment of the multi-sensor fusion-based transmission line R-type closed pin missing detection system described in this invention: the UAV subsystem includes a synchronously triggered high-resolution visible light camera and lidar for data acquisition; An edge computing unit is used to load and run the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines as described in any one of claims 1-8.

[0013] In a preferred embodiment of the multi-sensor fusion-based transmission line R-type closed pin missing detection system of the present invention: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of claims 1 to 9.

[0014] The beneficial effects of this invention are as follows: By fusing visible light images and laser point clouds, a joint analysis framework from two-dimensional texture to three-dimensional space is constructed. Utilizing precise pixel-to-point cloud mapping, an efficient detection process of "coarse localization of three-dimensional point clouds guiding fine recognition of two-dimensional images" is achieved. Furthermore, innovatively, state judgment is performed based on multi-dimensional geometric features such as the density, normal vector, and depth of three-dimensional point cloud clusters. This effectively overcomes the inherent limitations of single visible light sensors, which are susceptible to interference from lighting, weather, and background. It significantly improves the recognition rate and reliability of state judgment for small targets like R-type closed pins in complex natural environments. Simultaneously, this two-stage strategy avoids global searching within full-size images, greatly improving data processing efficiency. It achieves full automation from data acquisition, fusion processing, intelligent recognition, state judgment to report generation, significantly reducing manual intervention. It not only outputs defect reports containing precise three-dimensional geographic coordinates to guide accurate maintenance but also meets the urgent need of power grids for large-scale, high-frequency, and intelligent inspections of transmission lines. Ultimately, it achieves comprehensive benefits including reduced manual inspections, shortened maintenance cycles, and improved power grid safety and intelligent operation and maintenance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention. Wherein: Figure 1 A flowchart of a method and system for detecting missing R-type closed pins in transmission lines based on multi-sensor fusion is shown. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0017] The terminology used in this invention is that which is currently widely used in the art in consideration of the function of the invention; however, these terms may vary according to the intent of those skilled in the art, precedent, or new technology in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the invention. Therefore, the terms used in this specification should not be construed as simple names, but rather based on their meanings and the overall description of the invention.

[0018] Example 1 Reference Figure 1 This embodiment provides a method and system for detecting missing R-type closed pins in transmission lines based on multi-sensor fusion, including synchronously acquiring images and point cloud data of the transmission line through multiple sensors, and performing preprocessing on each data to achieve data acquisition and preprocessing; Specifically, visible light images of power transmission lines are acquired using a visible light camera and lidar mounted on a drone that are triggered simultaneously. I raw and laser point cloud data P raw Distortion correction and color enhancement are performed on the visible light image to obtain the enhanced image. I enhanced The laser point cloud data is denoised, downsampled, and stitched together to generate a 3D point cloud model. M cloud .

[0019] Establish a mapping relationship between the preprocessed image and the point cloud, coarsely locate the key region based on the spatial features of the point cloud and project it onto the image to obtain the region of interest, thereby realizing data fusion and coarse target localization; Specifically, using the pre-calibrated camera intrinsic parameter matrix K, the LiDAR rotation matrix R, and the translation vector T from the camera, the 3D point cloud model is... M cloudPoints in the image are projected onto the enhanced image. I enhanced Above, establish pixel-point cloud mapping relationships; based on a 3D point cloud model. M cloud Based on the spatial geometric features, extract the three-dimensional region of the key connection point for installing the R-type cotter pin. R 3D And use mapping relationships to divide the three-dimensional region R 3D Back projection to enhanced image I enhanced Obtain the two-dimensional region of interest. R 2D .

[0020] A target detection network is used to perform fine identification and localization of R-shaped closed pins within the region of interest, achieving precise identification and localization. Specifically, the area of ​​interest R 2D The input is fed into the object detection network to identify the R-shaped closed pin and output its two-dimensional bounding box. B 2D .

[0021] The point cloud clusters corresponding to the target are associated and identified, their three-dimensional geometric features are calculated and input into the classifier to determine the presence or absence status of the R-type closed pin, thus realizing the absence status judgment. Specifically, for each identified R-type closed pin, based on its two-dimensional bounding box... B 2D Based on the established pixel-point cloud mapping relationship, the corresponding 3D point cloud clusters are extracted. P ; Calculate 3D point cloud clusters P The three-dimensional geometric features are obtained and input into a pre-trained state classifier to obtain the "in-situ" or "missing" state of the R-type closed pin.

[0022] By integrating 3D coordinates, recognition results, and status judgment results, a detection report is generated and a warning is issued for missing statuses, thus achieving result output and warning.

[0023] Example 2 Reference Figure 1 The difference between this embodiment and the first embodiment lies in the detection step: S1. Data Acquisition and Preprocessing: S1.1 Before conducting the test, system initialization must be completed: Hardware deployment: A DJI Matrice 350RTK drone was selected, equipped with a Zenmuse H20T payload, which integrates a 20-megapixel visible light camera and a LiDAR with a detection range of ≥450m. The drone's high-precision POS system provides a spatiotemporal reference for data fusion.

[0024] Joint sensor calibration: This is a prerequisite for data fusion. In a laboratory or specific environment, using a standard calibration board, an optimization algorithm is used to simultaneously solve for the camera's intrinsic parameter matrix K, and the relative position and attitude (i.e., rotation matrix R and translation vector T) between the LiDAR and the camera. This calibration result will be embedded as core parameters in the system for use in the fusion calculation of all subsequent data frames.

[0025] After initialization, the drone is controlled to fly along a preset route. During flight, the system controls the visible light camera and lidar to trigger synchronously, acquiring visible light images of the power transmission line at the same time. I raw and laser point cloud data P raw .

[0026] S1.2 Image Preprocessing: Distortion correction utilizes the camera intrinsic parameter matrix K and the distortion coefficient vector D (containing radial distortion). k 1 , k 2 , k 3 and tangential distortion p 1 , p 2 ), for the original image I raw Correction is performed to obtain a distortion-free image. I undistort For any pixel in the image ( u , v Project it onto the normalized camera coordinate system, apply the distortion model, and then reproject it back onto the image plane; Color enhancement employs a contrast-limited adaptive histogram equalization method. The image is first divided into several small grids, and histogram equalization is performed within each grid, while a contrast-limited threshold is used. clipLimit To avoid noise amplification, the results from each grid are finally merged using bilinear interpolation to obtain the enhanced image. I enhanced .

[0027] S1.3 Point cloud preprocessing: 1. Statistical outlier filtering: Calculate the number of outliers for each point in the point cloud. p i To its k The average distance between the nearest neighbors d i Calculate the global average distance m and standard deviation s Remove all that meet the criteria. di > m + a⋅s The point, among which α This is the scaling factor (usually taken as 1.0~2.0). 2. Voxel mesh downsampling: Divide the point cloud space into segments of size . v × v × v The voxel mesh is used to replace all points within each voxel with the centroid of all points within that voxel, resulting in a downsampled point cloud. P down Represented as: in, V i Let i represent the set of points within the i-th voxel. p Representing a voxel grid V i A three-dimensional point within the point cloud; Voxels represent the set of voxel meshes obtained after the entire point cloud is divided. 3. Point Cloud Stitching: Based on the POS data (position and attitude) provided by the UAV, multiple frames of point clouds are registered to a unified coordinate system through the iterative nearest point algorithm or its variants to form a complete 3D point cloud model of the power transmission line corridor. M cloud .

[0028] At this point, we obtained two core preprocessing results: a 3D point cloud model. M cloud and a series of enhanced images I enhanced .

[0029] S2. Data Fusion and Coarse Target Localization: This step is a crucial bridge connecting two-dimensional and three-dimensional information; its input is the result of the previous step. M cloud and I enhanced .

[0030] S2.1 Pixel-Point Cloud Association: For point cloud models M cloud any point in P world = ( x , y , z ) T Using the parameters (R, T, K) calibrated during system initialization, their coordinates are projected onto the image through perspective projection transformation. I enhanced superior: in,( u , v ) represents the projected pixel coordinates. s The scale factor is used. This formula applies to each 3D point. P world It was found in two-dimensional images I enhanced The corresponding pixel p pixel = ( u , v A precise coordinate mapping relationship was established.

[0031] S2.2, Extraction of key 3D regions: In the 3D point cloud model M cloud In this process, based on prior knowledge of transmission lines (such as conductor routing and vertical arrangement characteristics of insulator strings), a region growing algorithm or rule model is used to automatically identify and segment 3D point cloud clusters of key connection points such as suspension clamps and tension clamps. R 3D .

[0032] S2.3, Generate Image ROI: The 3D key regions extracted in the previous step... R 3D Using the established pixel-point cloud mapping relationship, it is back-projected onto the two-dimensional enhanced image. I enhanced This allows for the definition of the corresponding region of interest (ROI) within the image. R 2D .

[0033] The output of this step is a series of Regions of Interest (ROIs). R 2D Each ROI corresponds to a pre-located hardware connection point in three-dimensional space that is highly likely to contain an R-pin. This greatly narrows the search range for subsequent fine-tuning.

[0034] S3, Fine-grained identification and positioning: For each (ROI) R 2D The YOLOv10 network incorporates a coordinate attention mechanism; this mechanism enhances features by performing global average pooling on the input feature map X along both the horizontal and vertical directions to obtain Z. h and Z w The attention weights g are obtained after convolution, activation, and splitting. h and g w The final output feature map Y is calculated as follows: in, c It is the channel index of the feature map, indicating the channel number of the feature map. c One channel; i , j These are the spatial coordinates on the feature map, where i Corresponding to the height direction, j Corresponding width direction; y c ( i , j ) indicates that the output feature map is in the channel c ,Location( i , j The enhanced value at () x c ( i , j ) indicates that the input feature map is in the channel c ,Location( i , j The original value at () Indicates channel c At height i The horizontal attention weights calculated at that location; Indicates channel c In width j The vertical attention weights are calculated at this point.

[0035] The network ultimately outputs a 2D pixel-level bounding box for each R-pin within the ROI. B 2D = ( x min , y min , x max , y max ) and its confidence level.

[0036] S4. Missing status determination: Associated 3D point cloud clusters, for each detected 2D bounding box B 2D Based on the mapping relationship established in step S2, all 3D point clouds falling within the bounding box are found, forming a 3D point cloud cluster corresponding to the specific R-shaped pin. P ={p1,p2,...,p N}

[0037] Calculate 3D geometric features, including point cloud density. D Variance of normal vector distribution s 2 normaland local depth change rate s depth The calculation method is as follows: Point cloud density D The calculation formula is as follows: D = N / V in N For point cloud clusters P The number of points in the middle, V Let V be the bounding box volume of the point cloud cluster; Variance of normal vector distribution s 2 normal The calculation formula is as follows: Where n i Midpoint of a point cloud cluster i The normal vector, m The mean of all normal vectors; Local depth change rate s depth The calculation formula is as follows: in z i For point i The depth value, m z This is the average depth value for all points.

[0038] State classification: The feature vector v = [ D , s 2 normal , s depth Input a pre-trained random forest classifier and output the final state of the R-type pin: "in place" or "missing".

[0039] S5. Results Output and Early Warning Result synthesis: The recognition results (bounding boxes) from step S3 are synthesized. B 2D The status judgment result (in place / missing) in step S4 and the three-dimensional geographic coordinates corresponding to the target in step S2. P world Bind the data to create a complete detection record.

[0040] Output and Early Warning: Generate structured reports from all records. On the monitoring center platform, visualize the results (e.g., highlight missing points on point cloud models and images), and automatically issue early warnings for defects to guide maintenance.

[0041] Finally, it should be noted that the methods and devices described in detail above are merely embodiments, and those skilled in the art can modify these embodiments in different ways as long as they do not depart from the scope of the present invention.

Claims

1. A method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion, characterized in that: include, By simultaneously acquiring images and point cloud data of the power transmission line using multiple sensors and performing preprocessing on each data source, data acquisition and preprocessing can be achieved. Establish a mapping relationship between the preprocessed image and the point cloud, coarsely locate the key region based on the spatial features of the point cloud and project it onto the image to obtain the region of interest, thereby realizing data fusion and coarse target localization; A target detection network is used to perform fine identification and localization of R-type closed pins within the region of interest, thereby achieving fine identification and localization. The point cloud clusters corresponding to the target are associated and identified, their three-dimensional geometric features are calculated and input into the classifier to determine the presence or absence status of the R-type closed pin, thus realizing the absence status judgment. By integrating 3D coordinates, recognition results, and status judgment results, a detection report is generated and a warning is issued for missing statuses, thus achieving result output and warning.

2. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 1, characterized in that: The multi-sensor system includes a synchronously triggered visible light camera and a lidar.

3. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 2, characterized in that: Images and point cloud data of the power transmission line are collected synchronously by multiple sensors, resulting in visible light images and laser point cloud data.

4. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 1, characterized in that: The target detection network is a YOLOv10 network that incorporates a coordinate attention mechanism.

5. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 1, characterized in that: The three-dimensional geometric features include point cloud density, normal vector distribution variance, and local depth change rate.

6. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 5, characterized in that: The formula for calculating the point cloud density is as follows: Where D is the point cloud density, N is the number of points in the point cloud cluster P, and V is the bounding box volume of the point cloud cluster.

7. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 5, characterized in that: The formula for calculating the variance of the normal vector distribution is as follows: in, σ 2 normal Let be the variance of the normal vector distribution, N be the number of points in the point cloud cluster P, ni be the normal vector of point i in the point cloud cluster, and μ be the mean of all normal vectors.

8. The method for detecting missing R-type closing pins in transmission lines based on multi-sensor fusion according to claim 5, characterized in that: The formula for calculating the local depth change rate is as follows: in, σ depth Let N be the local depth change rate, N be the number of points in point cloud cluster P, and ρ be the depth value of the point. μ z This is the average depth value for all points.

9. A multi-sensor fusion-based detection system for missing R-type closing pins in transmission lines, characterized in that: Including the multi-sensor fusion-based method for detecting missing R-type closing pins in transmission lines as described in any one of claims 1 to 8, and, The unmanned aerial vehicle (UAV) subsystem includes a synchronously triggered high-resolution visible light camera and lidar for data acquisition; An edge computing unit is used to load and run the multi-sensor fusion-based method for detecting missing R-type closed pins in transmission lines as described in any one of claims 1-8; The monitoring center platform is used to receive and display detection results and early warning information.

10. A computing device, comprising: Memory and processor; The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images