A deep learning-based power transmission tower structure damage automatic detection method

Abstract

The application relates to the technical field of power equipment state monitoring, and provides a power transmission tower structure damage automatic detection method based on deep learning, which comprises the following steps: acquiring visible light images and infrared thermal imaging images of a power transmission tower through a drone to form an original image set; performing background segmentation processing and image registration processing on the original image set to generate a tower body region image set; constructing a deep convolutional neural network model, training the deep convolutional neural network model based on a tower image sample set with pixel-level damage annotation, and obtaining a damage detection model; inputting the tower body region image set into the damage detection model, outputting damage category identification and a pixel-level segmentation mask image; and calculating three-dimensional space coordinates of a damage position based on the pixel-level segmentation mask image to generate a structure damage detection report containing damage types and spatial positions. The application realizes integrated intelligent diagnosis of power transmission tower structure damage by fusing dual-optical imaging and deep learning technology.

Description

Technical Field

[0001] This invention belongs to the field of power equipment condition monitoring technology, and in particular relates to an automatic detection method for structural damage of transmission towers based on deep learning. Background Technology

[0002] Traditional methods for detecting structural damage to power transmission towers mainly rely on manual climbing inspections or ground-based telescope observations, which suffer from inherent drawbacks such as low efficiency, high safety risks, strong subjectivity, and difficulty in quantification. In recent years, automatic identification technology based on visible light images from drones has been developed, but it is limited by a single mode and has a high rate of missed detection when dealing with damage with low contrast to the background, such as rust and cracks, or invisible defects such as internal overheating.

[0003] Meanwhile, existing methods mostly remain at the level of judging the "presence" or "absence" of damage in two-dimensional images, unable to output the precise location of damage in the three-dimensional space of the tower, making it difficult to directly guide precise maintenance, and lacking a quantitative assessment of the severity of damage. Furthermore, background interference in complex field environments, inaccurate registration of multi-source images, and the difficulty in identifying small target damage remain key bottlenecks restricting the practical application of this technology. Therefore, there is an urgent need for an intelligent detection method for power transmission tower structural damage that can be automated, accurate, quantitative, and capable of three-dimensional localization. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the existing technology, this invention proposes an automatic detection method for structural damage of power transmission towers based on deep learning. The technical solution designed in this invention includes the following steps: S1: Obtain visible light and infrared thermal images of power transmission towers using drones to form the original image set; S2: Perform background segmentation and image registration processing on the original image set to generate an image set of the tower area; S3: Construct a deep convolutional neural network model, and train the deep convolutional neural network model based on a sample set of iron tower images with pixel-level damage annotations to obtain a damage detection model; S4: Input the image set of the tower area into the damage detection model, and output the damage category identifier and pixel-level segmentation mask; S5: Calculate the three-dimensional spatial coordinates of the damaged area based on the pixel-level segmentation mask image, and generate a structural damage detection report that includes the damage type and spatial location.

[0005] Preferably, the background segmentation process in S2 is based on a region filtering method with a preset height threshold, including: S21: Obtain the acquisition height information corresponding to each original image; S22: Set the background segmentation height threshold; S23: Image regions with a height greater than the background segmentation height threshold are initially identified as background sky regions and filtered out.

[0006] Preferably, S21 further includes: If valid height information cannot be obtained for a certain region in the original image, the data for that region will be marked as invalid and removed.

[0007] Preferably, the formula for setting S22 is as follows: In the formula, Set a height threshold for background segmentation. The minimum acquisition height corresponding to all original images. K represents the maximum acquisition height corresponding to all original images, and K is an empirical constant.

[0008] Preferably, the image registration process in S2 includes: S201: Extract key feature point sets from the original image set respectively; S202: Construct a reference plane and plane distance tolerance, and calculate the distance from each key feature point to the reference plane; S203: Filter out key feature points whose distance to the reference plane is less than the tolerance, and retain the key feature points for image registration.

[0009] Preferably, the planar distance tolerance in S202 is calculated using the following formula: In the formula, For planar distance tolerance, Let N be the distance from the i-th key feature point to the reference plane, N be the total number of key feature points, and M be the adjustment coefficient.

[0010] Preferably, the deep convolutional neural network model in S3 includes a parallel dual-channel feature extraction network, a feature fusion module, and an attention enhancement module based on spatial distance; The parallel dual-channel feature extraction network is used to extract deep features from the input visible light image and infrared image respectively, and output a first feature map and a second feature map. The feature fusion module is used to receive the first feature map and the second feature map, fuse them by channel splicing and / or weighted addition, and output a fused feature map; The spatial distance-based attention enhancement module is used to receive the fused feature map, generate a spatial attention weight map based on the relative distance between each spatial position in the fused feature map and the initial damage localization area, and multiply the spatial attention weight map element-wise with the fused feature map to output the enhanced feature map.

[0011] Preferably, the calculation of the three-dimensional spatial coordinates of the damaged area in step S5 includes: S51: Obtain the pose parameters of the UAV when acquiring the image to be detected, including the position and attitude angle of the camera in the world coordinate system, and the camera intrinsic parameter matrix; S52: Based on the pixel-level segmentation mask, determine the pixel coordinates of the damaged area in the image; S53: Based on the camera imaging model, the pixel coordinates are projected backward into three-dimensional space to form a ray originating from the camera's optical center; S54: Perform an intersection calculation between the ray and the pre-stored three-dimensional reference model of the power transmission tower. The intersection point is the actual location coordinate of the damaged part in three-dimensional space.

[0012] Beneficial effects: 1. This application overcomes the limitations of single-modal perception by fusing visible light texture and infrared thermal imaging features through a parallel dual-channel deep network. It introduces a Gaussian attention mechanism based on spatial distance, enabling the model to adaptively focus computational resources on vulnerable key parts such as bolt nodes and connectors, just like an expert. This significantly improves the detection rate of minor damage and effectively suppresses interference from complex backgrounds. 2. This application obtains the precise contour of the damage through pixel-level segmentation, and combines the UAV pose, camera model and the three-dimensional digital model of the tower to realize the accurate conversion of the damage point from two-dimensional pixel coordinates to three-dimensional world coordinates. 3. This application uses a dynamic thresholding method based on the extreme value of the flight altitude, rather than a fixed threshold, which can adapt to different flight missions and terrains. In image registration, a feature point filtering method based on planar distance tolerance is adopted, which can automatically filter out a large number of invalid in-plane points, thereby improving registration efficiency and accuracy. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the background segmentation process according to a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the image registration process according to a preferred embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of calculating the three-dimensional spatial coordinates of the damaged area according to a preferred embodiment of the present invention. Detailed Implementation

[0014] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0015] This invention designs an automatic detection method for structural damage of power transmission towers based on deep learning, such as... Figure 1-4 As shown, the technical solution specifically includes: S1: Obtain visible light and infrared thermal images of power transmission towers using drones to form the original image set; S2: Perform background segmentation and image registration on the original image set to generate an image set of the tower area; S3: Construct a deep convolutional neural network model. Train the deep convolutional neural network model based on a sample set of iron tower images with pixel-level damage annotations to obtain a damage detection model. S4: Input the tower area image set into the damage detection model, and output the damage category label and pixel-level segmentation mask; S5: Calculate the three-dimensional spatial coordinates of the damaged area based on the pixel-level segmentation mask image, and generate a structural damage detection report that includes the damage type and spatial location.

[0016] Specifically, for the S1, the drone uses the DJI Matrice 350 RTK industrial-grade drone platform, equipped with a Zenmuse H20N dual-light payload that can simultaneously acquire 20-megapixel visible light images and 640×512 resolution infrared thermal images. The drone flies around the tower along a preset route to ensure that it acquires multi-angle images of all sides of the tower, and simultaneously records the POS data (latitude, longitude, altitude, pitch angle, roll angle, and yaw angle) of each frame.

[0017] Preferably, the background segmentation process in S2 is based on a region filtering method with a preset height threshold, including: S21: Obtain the acquisition height information corresponding to each original image; S22: Set the background segmentation height threshold; S23: Image regions with a height greater than the background segmentation height threshold are initially identified as background sky regions and filtered out.

[0018] Preferably, S21 further includes: If valid height information cannot be obtained for a certain region in the original image, the data for that region will be marked as invalid and removed.

[0019] Preferably, the formula for S22 is as follows: In the formula, Set a height threshold for background segmentation. The minimum acquisition height corresponding to all original images. K represents the maximum acquisition height corresponding to all original images, and K is an empirical constant.

[0020] Preferably, the image registration process in S2 includes: S201: Extract key feature point sets from the original image set respectively; S202: Construct a reference plane and plane distance tolerance, and calculate the distance from each key feature point to the reference plane; S203: Filter out key feature points whose distance to the reference plane is less than the tolerance; retain key feature points for image registration.

[0021] Preferably, the planar distance tolerance in S202 is calculated using the following formula: In the formula, For planar distance tolerance, Let N be the distance from the i-th key feature point to the reference plane, N be the total number of key feature points, and M be the adjustment coefficient.

[0022] Specifically, for S2, feature points and their descriptors that characterize stable structures such as tower corners and edge intersections are extracted from visible light and infrared images acquired at the same time, forming two key feature point sets. The S22 formula achieves adaptive background segmentation. Dynamic calculations are performed using the altitude range of this flight mission, replacing a fixed threshold. The empirical constant K is typically taken as 3-5, which makes the threshold... It can adapt to different flight altitude plans and terrain undulations, automatically filtering out most of the sky background at higher positions, while retaining the main body of the tower and the surrounding ground scenery, making it more robust.

[0023] Furthermore, regarding the planar distance tolerance in S202, the advantage of this formula lies in its ability to automatically eliminate a large number of non-critical matching points located on approximate planes (such as the tower body plane). In the scenario of power transmission towers, many feature points on the tower surface may lie on the same geometric plane, contributing little to registration and potentially introducing noise. This formula calculates the average distance from all feature points to the fitted plane and sets a tolerance by adjusting the coefficient M (usually set to 2~4). Distance less than Points that are not in the plane are filtered out, and the points that are retained are the feature points that are more critical to registration, such as those located at different depths and corners, thereby improving the efficiency and accuracy of subsequent registration algorithms.

[0024] In addition, the image registration process can use the Scale Invariant Feature Transform (SIFT) algorithm or the Oriented FAST and Rotated BRIEF (ORB) algorithm to extract feature points and their descriptors that can characterize stable structures such as tower corners and edge intersections from visible light images and infrared images acquired at the same time, respectively, forming two sets of key feature points.

[0025] Preferably, the deep convolutional neural network model in S3 includes a parallel dual-channel feature extraction network, a feature fusion module, and a spatial distance-based attention enhancement module; A parallel dual-channel feature extraction network is used to extract deep features from the input visible light image and infrared image respectively, and output a first feature map and a second feature map. The feature fusion module is used to receive the first feature map and the second feature map, fuse them by channel concatenation and / or weighted addition, and output the fused feature map. The spatial distance-based attention enhancement module receives the fused feature map, generates a spatial attention weight map based on the relative distance between each spatial location in the fused feature map and the initial damage localization region, and multiplies the spatial attention weight map element-wise with the fused feature map to output the enhanced feature map.

[0026] Specifically, the enhanced feature map is a high-level semantic feature used internally by the model for final prediction. This feature map is simultaneously input into two parallel task branches: a classification branch (usually connected to a global pooling layer and a fully connected layer) is responsible for outputting damage category labels; the other segmentation decoder branch (usually a decoder with a U-Net or FPN structure) is responsible for upsampling the feature map and classifying it pixel by pixel, ultimately outputting a pixel-level segmentation mask map. Therefore, the attention enhancement module improves the quality of features, thus jointly optimizing the performance of both classification and segmentation tasks.

[0027] Additionally, the formula for generating the spatial attention weight map is as follows: In the formula, To fuse the spatial coordinates on the feature map, and These are the coordinates of the center of the damaged region estimated based on the results of the previous iteration or initial segmentation. and Preset parameters to control the range of attention distribution. It should be noted that the spatial attention weight map is an application function. To all spatial locations of the feature map Then, a complete two-dimensional matrix is ​​generated, consisting of all the calculated weight values.

[0028] Furthermore, the formula for generating the spatial attention weight map provides a smooth and interpretable attention mechanism. It simulates the physical intuition that the influence of a damaged region decays from its center outwards. Coordinates The closer to the damage estimation center Weight The closer the value is to 1, the more fully the feature is preserved; the farther away the value is, the more exponentially the weight decays to near 0, and the feature is suppressed. Parameters and By controlling the decay rate, the attention area can be flexibly adjusted. This mechanism allows the model to not only focus on the damage point itself, but also adaptively capture important contextual information around it, such as crack extension and corrosion boundaries, while suppressing irrelevant background noise.

[0029] In addition, for S4, damage category identifiers include, but are not limited to: corrosion, coating peeling, missing bolts, structural deformation, foreign object hanging, and insulator damage. The pixel-level segmentation mask is a binary image with the same size as the input image, where areas with a pixel value of 1 represent the damaged parts identified by the model, and areas with a pixel value of 0 represent the background or intact structures, thereby achieving pixel-level precise localization of the damaged areas.

[0030] Preferably, calculating the three-dimensional spatial coordinates of the damaged area in S5 includes: S51: Obtain the pose parameters of the UAV when acquiring the image to be detected, including the position and attitude angle of the camera in the world coordinate system, and the camera intrinsic parameter matrix; S52: Determine the pixel coordinates of the damaged region in the image based on the pixel-level segmentation mask map; S53: Based on the camera imaging model, the pixel coordinates are projected backward into three-dimensional space to form a ray originating from the camera's optical center; S54: Perform intersection calculations between the ray and the pre-stored three-dimensional reference model of the transmission tower. The intersection point is the actual coordinate of the damaged part in three-dimensional space.

[0031] Specifically, for S5, generating a structural damage detection report involves: structurally integrating the damage category and confidence level output by S4 with the three-dimensional coordinates calculated by S5. The report automatically generates a PDF document containing the tower ID, inspection time, damage details list (type, location, confidence level), and damage area annotation map, and uploads it to the operation and maintenance management system.

[0032] In addition, the 3D reference model of the transmission tower is a pre-built triangular mesh model or a lightweight CAD model based on the design drawings. The intersection calculation uses the Möller–Trumbore ray-triangle intersection algorithm, traversing all triangular faces of the model and calculating the intersection point of the ray with each face. The final selected valid intersection points must satisfy the following conditions: 1) Located in the positive direction of the ray; 2) The angle between the normal direction of the face where the intersection point is located and the ray direction satisfies certain conditions (such as excluding the intersection point on the back side).

[0033] The coordinates of the first intersection point that meets the conditions are identified as the three-dimensional world coordinates of the damage point.

[0034] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An automatic detection method for structural damage of power transmission towers based on deep learning, characterized in that, include: S1: Obtain visible light and infrared thermal images of power transmission towers using drones to form the original image set; S2: Perform background segmentation and image registration processing on the original image set to generate an image set of the tower area; S3: Construct a deep convolutional neural network model, and train the deep convolutional neural network model based on a sample set of iron tower images with pixel-level damage annotations to obtain a damage detection model; S4: Input the image set of the tower area into the damage detection model, and output the damage category identifier and pixel-level segmentation mask; S5: Calculate the three-dimensional spatial coordinates of the damaged area based on the pixel-level segmentation mask image, and generate a structural damage detection report that includes the damage type and spatial location.

2. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 1, characterized in that, The background segmentation process in S2 is based on a region filtering method with a preset height threshold, including: S21: Obtain the acquisition height information corresponding to each original image; S22: Set the background segmentation height threshold; S23: Image regions with a height greater than the background segmentation height threshold are initially identified as background sky regions and filtered out.

3. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 2, characterized in that, S21 further includes: If valid height information cannot be obtained for a certain region in the original image, the data for that region will be marked as invalid and removed.

4. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 2, characterized in that, The formula for setting S22 is as follows: In the formula, Set a height threshold for background segmentation. The minimum acquisition height corresponding to all original images. K represents the maximum acquisition height corresponding to all original images, and K is an empirical constant.

5. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 1, characterized in that, The image registration process in S2 includes: S201: Extract key feature point sets from the original image set respectively; S202: Construct a reference plane and plane distance tolerance, and calculate the distance from each key feature point to the reference plane; S203: Filter out key feature points whose distance to the reference plane is less than the tolerance, and retain the key feature points for image registration.

6. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 5, characterized in that, The planar distance tolerance in S202 is calculated using the following formula: In the formula, For planar distance tolerance, Let N be the distance from the i-th key feature point to the reference plane, N be the total number of key feature points, and M be the adjustment coefficient.

7. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 1, characterized in that, The deep convolutional neural network model in S3 includes a parallel dual-channel feature extraction network, a feature fusion module, and an attention enhancement module based on spatial distance. The parallel dual-channel feature extraction network is used to extract deep features from the input visible light image and infrared image respectively, and output a first feature map and a second feature map. The feature fusion module is used to receive the first feature map and the second feature map, fuse them by channel splicing and / or weighted addition, and output a fused feature map; The spatial distance-based attention enhancement module is used to receive the fused feature map, generate a spatial attention weight map based on the relative distance between each spatial position in the fused feature map and the initial damage localization area, and multiply the spatial attention weight map element-wise with the fused feature map to output the enhanced feature map.

8. The method for automatic detection of structural damage in power transmission towers based on deep learning according to claim 1, characterized in that, The calculation of the three-dimensional spatial coordinates of the damaged area in S5 includes: S51: Obtain the pose parameters of the UAV when acquiring the image to be detected, including the position and attitude angle of the camera in the world coordinate system, and the camera intrinsic parameter matrix; S52: Based on the pixel-level segmentation mask, determine the pixel coordinates of the damaged area in the image; S53: Based on the camera imaging model, the pixel coordinates are projected backward into three-dimensional space to form a ray originating from the camera's optical center; S54: Perform an intersection calculation between the ray and the pre-stored three-dimensional reference model of the power transmission tower. The intersection point is the actual location coordinate of the damaged part in three-dimensional space.

Images