Insulator defect identification method based on cross-modal data fusion

Abstract

The invention provides an insulator defect identification method based on cross-modal data fusion. The insulator defect identification method comprises the following steps: acquiring and preprocessing an insulator visible light image, a thermal infrared image and point cloud data which are synchronously acquired; projecting all points in the point cloud to obtain a projection image and a projection mapping matrix; registering the visible light image and the thermal infrared image with the projection image to obtain a corresponding affine transformation matrix, and constructing a shielding matrix for shielding the background according to the projection image; performing feature extraction on the pre-processed visible light image, thermal infrared image and point cloud data to respectively obtain a texture feature map, a temperature feature map and a geometric feature map; fusing the texture feature map, the temperature feature map and the geometric feature map according to the shielding matrix and the affine transformation matrix to obtain cross-modal fusion features; and the cross-modal fusion features are used to discriminate and position insulator defects. According to the method, the problem of leak detection of insulator defects in different scenes by a traditional single-mode method can be solved.

Description

Technical Field

[0001] This invention belongs to the field of insulator detection technology, specifically relating to a method for identifying insulator defects based on cross-modal data fusion. Background Technology

[0002] Insulators play a vital role in power transmission lines, providing both support and electrical insulation. They are numerous, diverse, and constantly exposed to harsh environments, making them susceptible to defects such as cracks, chipping, contamination, and localized overheating due to weather conditions like sandstorms and wind. A single failure, even just one, insulator in a transmission line can lead to a power outage across the entire area, potentially causing grid failures. Therefore, efficient and accurate insulator inspections are crucial for timely detection of insulator defects and improving the stability and reliability of the power system, ensuring grid safety.

[0003] Currently, drones are widely used for insulator inspections, but the traditional method still relies on manual judgment based on visible light and thermal infrared images collected by drones. This manual approach depends on the experience and attention of the staff, and has drawbacks such as strong subjectivity, low efficiency, and poor continuity.

[0004] In recent years, with the continuous development of deep learning theory, deep learning-based target detection algorithms have been applied in insulator defect detection. However, these methods are mostly based on single-modal data such as visible light or infrared images for defect identification, which are susceptible to background interference and have low accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for identifying insulator defects based on cross-modal data fusion.

[0006] To solve one, some, or all of the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for insulator defect identification based on cross-modal data fusion includes: acquiring visible light images, thermal infrared images, and point cloud data of a target insulator simultaneously collected by a UAV, as well as the position and attitude information of the UAV during the acquisition; preprocessing the visible light images, thermal infrared images, and point cloud data; projecting all points in the point cloud onto the visible light image as a reference to obtain a projected image and a projection mapping matrix; registering both the visible light image and the thermal infrared image with the projected image to obtain the corresponding affine transformation matrix, and constructing a masking matrix for masking the background based on the projected image; extracting features from the preprocessed visible light image, thermal infrared image, and point cloud data to obtain texture feature maps, temperature feature maps, and geometric feature maps, respectively; fusing the texture feature maps, temperature feature maps, and geometric feature maps according to the masking matrix and the affine transformation matrix to obtain cross-modal fusion features; using the cross-modal fusion features to identify insulator defects and inversely mapping the defect locations to the point cloud data.

[0007] Furthermore, the method for obtaining the corresponding affine transformation matrix after registering both the visible light image and the thermal infrared image with the projection image includes: identifying key geometric feature points from point cloud data; aligning the visible light image and the thermal infrared image with the projection image using local structure matching based on the projection points of the key geometric feature points on the visible light image and the projection image, thereby obtaining the affine transformation matrix of the visible light image and the thermal infrared image for aligning the projection image.

[0008] Furthermore, the method for identifying key geometric feature points from point cloud data includes: for each point in the point cloud, searching for all other points within a specified radius centered on that point to form a point set; for each point set, obtaining the normal vector of each point, and using the variance of the normal vectors of all points in the point set as the curvature of the center point of that point set; filtering out points whose curvature exceeds a set curvature threshold as candidate points; initializing an empty anchor point set; selecting candidate points in descending order of curvature, and for each selected target candidate point, if the distance between the target candidate point and all points in the anchor point set is greater than a set distance threshold, then adding the target candidate point to the anchor point set; dividing the anchor points into a suspension point set and a grounding wire intersection set based on the position of the anchor points in the anchor point set; the suspension point set and the grounding wire intersection set are the identified key geometric feature points.

[0009] Furthermore, the method for aligning visible light images and thermal infrared images with projected images using local structure matching includes: calculating the theoretical projection points of key geometric feature points on the thermal infrared image using perspective projection formulas, based on intrinsic parameters such as the focal length and pixel size of the UAV's onboard thermal imaging camera and extrinsic parameters such as the position and attitude of the UAV when capturing the corresponding images; obtaining the corresponding points of key geometric feature points within the projected image as the theoretical projection points of the key geometric feature points on the visible light image; truncating calibration windows of specified sizes on both the visible light and thermal infrared images, centered on the theoretical projection points; and truncating calibration windows on the projected image, centered on the key geometric feature points. A reference window of specified size is defined; the gradient direction distribution of pixels within each calibration window and the reference window is analyzed using the AKAZE algorithm to obtain several key points and their corresponding feature vectors; for each calibration window of the visible light image and the thermal infrared image, the one-to-one similarity between the feature vectors of all key points in the corresponding calibration window and the feature vectors of all key points in the corresponding reference window is calculated, and the pair of key points with the highest similarity is selected as the matching point pair; based on all matching point pairs of the visible light image and the thermal infrared image, the affine transformation matrix of the visible light image and the thermal infrared image is calculated using the least squares method, and the visible light image and the thermal infrared image are aligned to the projected image.

[0010] Furthermore, the method for constructing a masking matrix for masking the background based on the projected image includes: constructing a masking matrix of the same size based on the projected image, wherein the element values ​​corresponding to the projected points and non-projected points of the point cloud in the projected image in the masking matrix represent unmasked and masked points, respectively; and performing dilation followed by erosion processing on the masking matrix.

[0011] Furthermore, the method for feature extraction from preprocessed point cloud data includes: iteratively selecting key points from the point cloud using farthest sampling of the PointNet++ algorithm; constructing a neighborhood sphere for each key point with an initial radius; calculating the coordinate offset and intensity difference of each point in the neighborhood relative to the key point for each neighborhood sphere, forming a feature vector; processing the feature vectors of all neighborhood points using a multilayer perceptron with shared weights to obtain the local feature vector of each point; aggregating the local feature vectors of all points in the neighborhood using max pooling to obtain the local features of the key point; gradually increasing the radius of the neighborhood sphere and repeating the previous step to obtain the local features of each key point again; and performing hierarchical feature fusion on the extracted multilayer local features of the key points to obtain the feature vector of each key point. The key point fusion features are calculated as follows: For each non-key point in the point cloud, the three nearest key points are obtained. The reciprocal of the distance between the non-key point and the key points is used as the weight. The fusion features of the three nearest key points are weighted and summed to obtain the fusion features of the non-key point. The fusion features of all points in the point cloud are globally pooled to obtain the point cloud features. A geometric feature map with the same size as the texture feature map and all zeros is created. For each point in the point cloud data, the corresponding projection coordinates are obtained according to the projection mapping matrix. The element value in the occlusion matrix is ​​determined according to the projection coordinates to determine whether it represents occlusion. If not, the projection coordinates are downsampled to obtain the corresponding new coordinates. The fusion features of the point are written into the corresponding position of the geometric feature map according to the new coordinates.

[0012] Furthermore, the method for fusing texture feature maps, temperature feature maps, and geometric feature maps includes: aligning the texture feature maps and temperature feature maps using an affine transformation matrix; constructing a geometric attention map using an occlusion matrix and the aligned texture and temperature feature maps; weighting the aligned texture and temperature feature maps using the geometric attention map; concatenating the weighted texture, temperature, and geometric feature maps into a cross-modal concatenated feature; estimating the confidence level of each modality in real time, and using the confidence level to perform modal weighting on the cross-modal concatenated feature to obtain the cross-modal fused feature.

[0013] Furthermore, the method for aligning the texture feature map and temperature feature map using an affine transformation matrix includes: upsampling the texture feature map and temperature feature map to obtain the corresponding original image coordinates; mapping each original image coordinate to the projected coordinates according to the affine transformation matrix; and downsampling the projected coordinates to the same size as the texture feature map to obtain the aligned feature layer coordinates.

[0014] Furthermore, the method for constructing a geometric attention map using the occlusion matrix and the aligned texture and temperature feature maps includes: downsampling the occlusion matrix and pooling the size of the occlusion matrix to match the size of the texture feature map; if the pooled value is greater than a set threshold, the corresponding element value is 1, otherwise it is 0; adding the aligned texture and temperature feature maps pixel by pixel to obtain a temporary overlay feature; multiplying the downsampled occlusion matrix and the temporary overlay feature pixel by pixel to obtain the final overlay feature; and applying a 1×1 convolution to the final overlay feature and using the Sigmoid activation function to obtain a single-channel geometric attention map.

[0015] Furthermore, the method of real-time estimation of the confidence level of each modality and modal weighting of cross-modal stitching features using the confidence level includes: constructing a modal graph containing visible light nodes, thermal infrared nodes, and point cloud nodes using a graph neural network, where the corresponding node values ​​are the global average pooling value of the aligned texture feature map, the global average value of the aligned temperature feature map, and the global average value of the geometric feature map, respectively; the similarity of the three edges of the modal graph is defined as the cosine similarity between two corresponding node values; performing local variance statistics on the original visible light image, taking the overall average variance and normalizing it to obtain the confidence level of the visible light nodes; and first calculating So on the original thermal infrared image. The Bel gradient is calculated, and the average gradient of the entire image is obtained and normalized to obtain the confidence score of the thermal infrared node. The number of geometric feature points in the point cloud data is counted. If the number exceeds the threshold, the confidence score of the point cloud node is 1; otherwise, the confidence score of the point cloud node is the ratio of the number of geometric feature points to the threshold. For each directed edge, the similarity and the confidence score of the source node are weighted and summed, and then processed using the softmax function to obtain the corresponding adaptive edge weight. For each node, the features of its directly adjacent nodes are weighted according to the corresponding adaptive edge weights and then summed to update the new node features. The average value of the three new node features is used as the cross-modal fusion feature.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention simultaneously acquires three complementary modal data: visible light images reflecting surface texture defects of insulators, thermal infrared images reflecting temperature anomalies, and laser point cloud data reflecting geometric deformation. Combined with a masking matrix to dynamically filter background interference, a cross-modal dynamic weighted fusion mechanism is adopted to adaptively balance the modal reliability under different environments. It can dynamically weight and fuse texture feature maps, temperature feature maps, and geometric feature maps, overcoming the problem of missed detection of insulator defects in different scenarios by traditional single-modal methods, and significantly improving the detection rate of various defects such as cracks, stains, glue peeling, overheating, and structural tilting.

[0017] This invention utilizes point cloud data to identify the region where the insulator is located, and finally generates a masking matrix to mask the outer region of the insulator in visible light images and thermal infrared images. During cross-modal feature fusion, only the unmasked insulator region is considered, thus filtering background interference, improving the accuracy of defect identification, and increasing computational efficiency.

[0018] This invention employs ResNet50+ deformable convolution to extract surface texture features from visible light images that reflect insulator cracks, stains, and adhesive loss. Deformable convolution addresses the non-rigid deformation of insulators caused by wind, making crack detection insensitive to bending. Multi-layer deformable convolution can simultaneously capture fine cracks (small receptive field) and long fractures (large receptive field). This invention uses MobileNetV3 to extract temperature features reflecting localized overheating and uneven heating of insulators. Temperature gradient enhancement suppresses background thermal noise (such as cloud reflection) and accurately locates hotspot boundaries. This invention uses PointNet++ layered fusion to extract geometric structural features from point cloud data that reflect insulator tilt, fracture, and missing parts. A geometric feature projection mechanism maps sparse point clouds into dense two-dimensional feature maps while preserving geometric details. Multi-scale layering enables joint capture of local defects (single broken ceramic piece) and overall anomalies (entire string tilt).

[0019] This invention uses the insulator's inherent suspension point and its intersection with the grounding wire as registration references. It aligns the pixel coordinates of cross-modal data through a projection mapping matrix. During final defect localization, a reverse projection mechanism maps the two-dimensional defect area back to point cloud spatial coordinates, ultimately outputting the three-dimensional spatial coordinates of the insulator defect location. This provides maintenance personnel with precise spatial guidance, avoiding the blind replacement of entire insulator strings. The final output consists of a defect probability vector, a defect feature map, and three-dimensional location coordinates, forming a triple interpretable output that meets the power industry's needs for defect tracing. Detailed Implementation

[0020] To better understand the present invention, the following embodiments further illustrate the content of the invention, but the scope of protection of the present invention is not limited to the following embodiments. Numerous specific details are set forth in the following description to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without one or more of these details.

[0021] Example 1: The purpose of this example is to provide a method for insulator defect identification based on cross-modal data fusion. It includes: Step S1: Acquire visible light images, thermal infrared images, and point cloud data of the target insulator simultaneously collected by the UAV, as well as the position and attitude information of the UAV during the data collection.

[0022] Acquire visible light images captured by the drone's onboard digital camera. Visible light images provide information such as the texture, shape, and color of the insulator surface, allowing for the identification of defects such as cracks, stains, and adhesive residue.

[0023] Acquire thermal infrared images captured by an airborne thermal imaging camera on a drone. Thermal infrared images can detect temperature anomalies on the surface of insulators, which often indicate discharge, dirt, or internal damage.

[0024] Acquire point cloud data collected by the airborne lidar of the UAV. Point cloud data can provide the three-dimensional geometry of the insulator and its suspension wire, providing geometric information such as spatial position, curvature, and normal vector.

[0025] The visible light images, thermal infrared images, and point cloud data collected must be acquired by the UAV at the same location, in the same attitude, and at the same time. Therefore, after the UAV flies to the target location and becomes stationary and stable, the three types of data can be collected simultaneously, and the position and attitude information of the UAV at the current moment can be recorded at the same time.

[0026] Step S2 involves preprocessing the visible light image, thermal infrared image, and point cloud data respectively.

[0027] Specifically, step S2 includes: S201. Perform bilateral filtering on the visible light image and the thermal infrared image.

[0028] During actual drone camera shooting, uneven lighting, electronic noise from the camera sensor, and slight shaking caused by airflow can all produce subtle and irregular fluctuations in brightness or color in the image. These fluctuations and noises can interfere with subsequent feature extraction, so image data filtering is necessary. This step first performs bilateral filtering on the visible light image and the thermal infrared image.

[0029] Conventional smoothing processes (such as mean filtering) tend to approximate the color of all pixels in an image towards the average of their neighboring pixels, resulting in blurred image edges (such as textures and contours). Bilateral filtering, however, determines whether adjacent pixels belong to the same object during smoothing. If two pixels have significantly different colors, they are considered to be on different object boundaries and are not smoothed. Bilateral filtering considers both spatial distance and color similarity. For each pixel, it first identifies its surrounding neighborhood; pixels that are closer in space and more similar in color receive greater weight. Finally, it calculates a weighted average of the colors of pixels satisfying both conditions to obtain a new pixel value.

[0030] Specifically, bilateral filtering methods include: For each pixel in the image to be processed, a K×K pixel target window is selected with that pixel as the center pixel (K is an odd number, such as K values ​​of 5, 7, or 9, and the size of K determines the maximum smoothing range); if the center pixel is close to the edge of the image to be processed, the blank pixels in the target window are filled with zero padding or mirror padding. Obtain the Euclidean distance d between the coordinates of each pixel in the target window and the center pixel, and then use the function... Calculate the spatial weight of each pixel in the target window. The spatial standard deviation is used to control the weight decay rate of pixels far from the center, and its value is one-third of the target window width K. Obtain the color difference between each color channel of each pixel in the target window and the corresponding color channel of the center pixel. Then, use the L2 norm of all color channel differences as the color difference I between that pixel and the center pixel. Finally, according to the function... Calculate the intensity weight of each pixel in the target window. The value is the standard deviation of intensity, used to control tolerance for color differences, with a value of 10-30. The spatial weight and intensity weight of each pixel in the target window are multiplied and then normalized to obtain the corresponding weight; The filtered pixel value is obtained by weighting all pixels in the target window according to their corresponding weights. After traversing all pixels of the image to be processed according to the above steps, the bilaterally filtered image is obtained.

[0031] S202. Upsample the thermal infrared image to the same size as the visible light image using bilinear interpolation.

[0032] S203. Normalize the processed visible light image and thermal infrared image.

[0033] S204. Perform voxel mesh filtering on the point cloud data.

[0034] When using a drone's lidar for ranging, atmospheric scattering, insufficient reflection intensity, and terrain obstruction can cause isolated, sparsely distributed noise points, necessitating noise reduction. Simultaneously, dense redundant points may appear on the same surface, requiring merging. This step employs voxel mesh filtering for both noise reduction and redundant point merging.

[0035] Specifically, voxel mesh filtering methods include: Divide the point cloud space into equal-sized cubic voxel meshes: set the side length of the voxel mesh, traverse all points in the point cloud, find the minimum and maximum coordinate values, obtain the total length of the three-dimensional space in the XYZ directions based on the difference between the minimum and maximum values, divide the total length by the side length of the voxel mesh and round up to obtain the number of voxel meshes in each direction. All points are assigned to a voxel grid based on the coordinates of each point in the point cloud: the difference between each point and the point with the smallest coordinate is calculated, then divided by the side length of the voxel grid and rounded down to determine the voxel to which the point belongs. Traverse all voxel grids. If the number of points in a voxel grid is less than 3, generate a representative point for the voxel grid based on the geometric center coordinates of the voxel grid and the average intensity of all points. Otherwise, generate a representative point for the voxel grid based on the average coordinates and average intensity of all points in each voxel grid. The filtered point cloud data is composed of all representative points.

[0036] The side length of the voxel grid determines the coarseness of the downsampling. A smaller side length retains more geometric details but results in a smaller reduction in the number of points, while a larger side length reduces the number of points faster but may lose geometric information of small cracks. Preferably, the side length of the voxel grid is 0.02m to ensure coverage of the smallest structures of the insulator (such as ceramic cracks) without losing curvature information of the suspension points.

[0037] Step S3: Obtain the projected image of the point cloud data, register the visible light image and thermal infrared image with the projected image to obtain the corresponding affine transformation matrix, and construct the occlusion matrix based on the projected image.

[0038] The suspension point and grounding wire intersection of the insulator string are inherent structures of the insulator and are not affected by light or temperature. These geometric points have high recognizability in different modal data, so they are used as references for cross-modal data or different image registration.

[0039] Specifically, step S3 includes: S301. Using the visible light image as a reference, project all points in the point cloud to obtain a projected image and a projection mapping matrix.

[0040] Based on intrinsic parameters such as focal length and pixel size of the UAV's onboard digital camera and extrinsic parameters such as the UAV's position and attitude when capturing visible light images, the projection coordinates of each point in the point cloud onto the visible light image are calculated using perspective projection formulas. If multiple points are projected onto the same pixel, only the point closest to the UAV is retained to avoid occlusion covering the surface. A two-dimensional projection image is constructed based on the projection coordinates of all points in the point cloud. The size of the projection image is consistent with the size of the visible light image. In the projection image, pixels corresponding to projected points can be set to black or white, while pixels of non-projected points can be set to their inverted colors.

[0041] After completing the projection calculation of all points in the point cloud, the projection mapping matrix corresponding to the points in the point cloud and the pixels in the projected image can be obtained.

[0042] S302. Identify key geometric feature points from point cloud data.

[0043] For each point in the point cloud, search for all other points within a specified radius (e.g., 0.1m) centered on that point to form a point set; the number of points in the point set is the same as the number of points in the point cloud. For each point set, the least squares plane fitting method is used to obtain the normal vector of each point, and the variance of the normal vectors of all points in the point set is used as the curvature of the center point of the point set. Points with curvature exceeding a set curvature threshold (e.g., 0.30) are selected as candidate points. Initialize an empty set of anchor points, and select candidate points in descending order of curvature; for each selected target candidate point, if the distance between the target candidate point and all points in the anchor point set is greater than the set distance threshold (set according to the point cloud density or actual needs, such as 0.05m), then add the target candidate point to the anchor point set. Based on the location of the anchor points in the anchor point set, the anchor points are divided into a suspension point set (points located vertically above the insulator string) and a grounding wire intersection set (points located near the ground or fittings and connected to the supporting structure); the suspension point set and the grounding wire intersection set are the key geometric feature points for identification.

[0044] S303. Based on the projection points of key geometric feature points on the visible light image and the projection image, the visible light image and the thermal infrared image are aligned with the projection image using local structure matching to obtain the affine transformation matrix of the visible light image and the thermal infrared image for aligning the projection image.

[0045] Based on the intrinsic parameters such as focal length and pixel size of the UAV's onboard thermal imaging camera and the extrinsic parameters such as the position and attitude of the UAV when capturing the corresponding image, the theoretical projection points of key geometric feature points on the thermal infrared image are calculated using the perspective projection formula. Obtain the corresponding points of key geometric feature points in the projected image, and use them as the theoretical projection points of the key geometric feature points on the visible light image; Centered on the theoretical projection point, a calibration window of a specified size (e.g., 16×16 pixels) is cropped from the visible light image and the thermal infrared image, respectively. Centered on key geometric feature points, a reference window of a specified size (e.g., 16×16 pixels) is cropped from the projected image; The gradient direction distribution of pixels in each calibration window and reference window is analyzed using the AKAZE algorithm to obtain several (30-50) key points and their corresponding feature vectors. For each calibration window of the visible light image and the thermal infrared image, the one-to-one similarity between the feature vectors of all key points in the corresponding calibration window and the feature vectors of all key points in the corresponding reference window is calculated using Hamming distance, Euclidean distance or cosine similarity. The pair of key points with the highest similarity is selected as the matching point pair. Based on all matching point pairs of the visible light image and the thermal infrared image, the affine transformation matrix of the visible light image and the thermal infrared image is calculated using the least squares method, and the visible light image and the thermal infrared image are aligned to the projected image.

[0046] S304. Construct a masking matrix based on the projected image.

[0047] Point clouds can capture the visible surface of an insulator, while visible light and thermal infrared images often contain a large number of background elements (sky, trees, ground). In subsequent steps, more attention needs to be paid to the pixels covered by the point cloud (i.e., the area corresponding to the insulator). Therefore, this step constructs a masking matrix based on the projected image. The masking matrix is ​​used to reflect the key areas in the visible light and thermal infrared images and reduce background interference.

[0048] A masking matrix is ​​constructed based on the projected image. The size of the masking matrix is ​​the same as the pixel size of the projected image. The element corresponding to the pixel of the projected point in the projected image has a value of 1 (indicating that it is not masked) in the masking matrix, and the other elements have a value of 0 (indicating that it is masked). The occlusion matrix is ​​first expanded and then eroded (using 3×3 square structuring elements) to fill the small holes caused by the sparse point cloud in order to obtain a continuous occlusion area.

[0049] By combining the masking matrix and the affine transformation matrix, it is possible to directly segment key regions from visible light images and thermal infrared images.

[0050] Step S4: Feature extraction is performed on the visible light image, thermal infrared image, and point cloud data, respectively.

[0051] Before subsequent feature fusion, the most relevant features are extracted from the visible light image, thermal infrared image and point cloud data respectively. Surface texture features are extracted from the visible light image, which can reflect the insulator's cracks, stains, glue peeling and other conditions. Temperature features are extracted from the thermal infrared image, which can reflect the insulator's local overheating, uneven heating and cooling and other conditions. Geometric structure features are extracted from the point cloud data, which can reflect the insulator's tilt, breakage and missing parts and other conditions.

[0052] Specifically, step S4 includes: S401. Use a texture feature extraction model to extract the texture features of insulators from visible light images.

[0053] The backbone network of the texture feature extraction model uses a ResNet50 convolutional neural network (without fully connected layers) to extract general visual features (such as edges, corners, and texture primitives). The input is a normalized visible light image (H×W×3), and the output is a feature map of H / 16×W / 16×1024.

[0054] The backbone network is followed by a deformable convolutional module. This module first performs a 2x bilinear upsampling on the backbone network's output, then processes it through two layers of deformable convolutions. Insulators may bend and deform due to wind force, or their geometry may change (stretching, tilting) due to camera angle. The fixed sampling grid of standard convolutions may not be suitable for this. Therefore, at each sampling point of the standard 3×3 convolution kernel, additional offsets in the x and y directions are predicted for each kernel position. These offsets are automatically generated using a small convolutional network based on the current feature map content. The two deformable convolutional layers have dilation rates of 2 and 4, respectively, continuously expanding the receptive field to capture a wider range of cracks or stains on entire ceramic plates.

[0055] The output of the deformable convolution module is compressed to obtain a texture feature map (H / 4×W / 4×256) carrying multi-scale texture information. Feature compression is implemented using 1×1 convolution and ReLU activation function, compressing the number of channels from 1024 to 256.

[0056] S402. Use the temperature feature extraction module to extract the temperature features of the insulator from the thermal infrared image.

[0057] The backbone network of the temperature feature extraction module uses the MobileNetV3-Small lightweight convolutional neural network (the first 7 modules) to quickly extract basic thermal distribution features. The input is a normalized thermal infrared image (H×W×1), and the output is a feature map of H / 8×W / 8×96.

[0058] The output of the backbone network first uses a Sobel operator with a gradient kernel of 3×3 to calculate the temperature gradient, highlighting regions of abrupt temperature changes and suppressing uniform background; then it undergoes channel attention processing to reduce the 96 channels to 256 channels. After channel attention processing, the output is subjected to 2x bilinear upsampling and a 3×3 convolution layer, resulting in a temperature feature map (H / 4×W / 4×256) carrying the edges and intensity of thermal anomalies. The size of the temperature feature map is the same as that of the texture feature map.

[0059] S403. Use the point cloud feature extraction module to extract point cloud features from the point cloud data, and obtain the geometric features of the insulator based on the point cloud features, projection mapping matrix and occlusion matrix.

[0060] Methods for extracting point cloud features from point cloud data using a point cloud feature extraction module include: Key points are iteratively selected from the point cloud using the PointNet++ algorithm's Farthest Point Sampling. For each key point, construct a neighborhood sphere with an initial radius; For each neighborhood sphere, calculate the coordinate offset and intensity difference of each point in the neighborhood relative to the key point (center point) to form a feature vector; use a multilayer perceptron (MLP) with shared weights to process the feature vectors of all neighborhood points to obtain the local feature vector of each point; use max pooling to aggregate the local feature vectors of all points in the neighborhood to obtain the local features of the key point. Increase the radius of the neighborhood sphere step by step (e.g., 0.05m → 0.1m → 0.2m), repeat the previous step, and extract the local features of each key point again; The extracted multi-layer local features of key points (each layer corresponds to a radius of a neighborhood sphere) are fused hierarchically to obtain the fused features of each key point; For each non-key point in the point cloud, obtain the three closest key points. Use the reciprocal of the distance between the non-key point and the key points as the weight, and sum the fusion features of the three closest key points using the weight to obtain the fusion features of the non-key point. Global pooling is performed on the fused features of all points in the point cloud to obtain the global features of the point cloud data, i.e., point cloud features.

[0061] When performing hierarchical feature fusion, the low-resolution features are upsampled layer by layer back to the highest resolution (the number of points in the first layer), and then the multi-layer features are concatenated along the channel dimension. Finally, a lightweight MLP is used to compress the concatenated features back to a unified dimension (such as 256 dimensions).

[0062] Methods for obtaining the geometric features of insulators based on point cloud features, projection mapping matrices, and occlusion matrices include: Create a geometry feature map with all zeros, the size of which is the same as the temperature feature map and the texture feature map (H / 4×W / 4×256). For each point in the point cloud data, obtain the corresponding projection coordinates (pixel coordinates on the projected image) according to the projection mapping matrix. Determine whether the corresponding element value in the occlusion matrix is ​​1 (indicating that it is not occluded) based on the projection coordinates. If so, downsample the projection coordinates by 4 times (corresponding to the size of the geometric feature map) to obtain the corresponding new coordinates. Write the fused features of the point into the corresponding position of the geometric feature map according to the new coordinates.

[0063] Step S5: Based on the occlusion matrix and affine transformation matrix, the features extracted from the visible light image, thermal infrared image and point cloud data are fused to obtain cross-modal fusion features.

[0064] Specifically, this step includes: S501. Align the texture feature map and temperature feature map with feature layers using an affine transformation matrix.

[0065] The texture feature map and temperature feature map are upsampled (multiplied by 4 for each pixel position) to obtain the corresponding original image coordinates; each original image coordinate is mapped to the projected coordinates according to the affine transformation matrix; finally, the projected coordinates are downsampled (divided by 4) to obtain the aligned feature layer coordinates, thus achieving the alignment of the texture feature map and temperature feature map on the feature layer.

[0066] S502. Construct a geometric attention map using the occlusion matrix and the aligned texture and temperature feature maps.

[0067] The occlusion matrix is ​​downsampled, and the size of the occlusion matrix is ​​averaged by pooling (4×4 pooling kernel, stride 4) to H / 4×W / 4 so as to match the size of the feature map; if the pooled value is greater than the set threshold (e.g. 0.3), the corresponding element value is 1, otherwise it is 0.

[0068] The aligned texture feature map and temperature feature map are added pixel by pixel to obtain a temporary overlay feature. The downsampled occlusion matrix is ​​then multiplied pixel by pixel with the temporary overlay feature to obtain the final overlay feature. The final overlay feature retains only the features of the geometrically visible area, suppressing background interference.

[0069] The final superimposed features are processed using 1×1 convolution and activated by the Sigmoid activation function to obtain a single-channel geometric attention map of size H / 4×W / 4.

[0070] S503. Use geometric attention maps to weight the aligned texture feature maps and temperature feature maps.

[0071] The texture feature map and temperature feature map are multiplied pixel-by-pixel and channel-by-channel using a geometric attention map to obtain a weighted texture feature map and temperature feature map.

[0072] S504. The weighted texture feature map, the weighted temperature feature map, and the geometric feature map are concatenated into a cross-modal concatenation feature.

[0073] S505. Real-time estimation of the credibility of each modality, dynamic modal weighting of cross-modal splicing features, and obtaining cross-modal fusion features.

[0074] Although cross-modal attention (geometric attention graph) has incorporated spatial geometric information into the features, the reliability of each modality can vary under different field conditions (such as insufficient lighting, infrared noise, and sparse point clouds). Therefore, a graph neural network (GNN) is used to estimate the reliability of each modality in real time, and the features of each modality are used for feature fusion.

[0075] A modal graph containing visible light nodes, thermal infrared nodes, and point cloud nodes is constructed using GNN. The corresponding node values ​​are the global average pooling value of the aligned texture feature map, the global average value of the aligned temperature feature map, and the global average value of the geometric feature map, respectively. The three nodes are interconnected to form three edges, and the similarity of each edge is defined as the cosine similarity between the two corresponding node values.

[0076] Initialize the confidence scores of the three nodes. Perform local variance statistics on the original visible light image, take the overall average variance and normalize it to obtain the confidence score of the visible light node; calculate the Sobel gradient for the original thermal infrared image, then obtain the average gradient of the entire image and normalize it to obtain the confidence score of the thermal infrared node; count the number of geometric feature points in the point cloud data. If the number exceeds a threshold, the confidence score of the point cloud node is 1; otherwise, the confidence score of the point cloud node is the ratio of the number of geometric feature points to the threshold.

[0077] In modal graphs, edges are undirected. During computation, each undirected edge can be split into two directed edges. For each directed edge, the similarity and the confidence of the source node are weighted and summed, and then processed using the softmax function to obtain the corresponding adaptive edge weight. During the weighted summation, the weights of similarity and confidence are learnable scalars, automatically determined during network training via gradient descent.

[0078] Each node sums the features of its directly adjacent nodes according to adaptive edge weights (adaptive edge weights correspond to the convergence direction, i.e., the directed edge), and updates the new node features.

[0079] The average of the three new node features is used as the cross-modal fusion feature.

[0080] Step S6: Use cross-modal fusion features to identify insulator defects and inversely map the defect locations to point cloud data.

[0081] This step uses a defect discrimination module to identify insulator defects. The defect discrimination module includes a first feature compression layer, an anomaly feature extraction layer, a channel attention weighting layer, a spatial attention focusing layer, a second feature compression layer, a first output layer, and a second output layer. The first feature compression layer uses a 1×1 convolutional kernel and, through batch normalization and the ReLU activation function, compresses the 256-dimensional features into 128 dimensions. The anomaly feature extraction layer uses a two-level depth 3×3 convolutional kernel with a stride of 1 and an inflation factor of 1 to extract local anomaly details. The channel attention weighting layer first performs global average pooling, then passes through a fully connected layer and a sigmoid activation function to obtain channel weights, which are then multiplied channel-by-channel with the original features. The spatial attention focusing layer performs maximum and average stacking along the channel axis, and after processing with a 7×7 convolutional kernel and a sigmoid activation function, obtains the spatial... The weights are multiplied pixel-by-pixel with the original features; the second feature compression layer again uses 1×1 convolution, batch normalization, and ReLU activation function to compress the number of channels from 128 dimensions to 64 dimensions; the first output layer performs global average pooling on the features and then processes them through a fully connected layer to output a k-dimensional vector, which is then normalized by softmax to obtain the probability of k types of defects; the second output layer performs 4x deconvolution upsampling on the features, and after processing with a 1×1 convolution kernel and Sigmoid activation function, a defect feature map of the same size as the input features is obtained, where the value of each pixel in the defect feature map represents the probability that the point belongs to any defect.

[0082] For training the defect discrimination module, a sample set is first constructed. For insulators installed in a distribution network with known defect types, a UAV simultaneously acquires visible light images, thermal infrared images, and point cloud data at different times (covering daytime / morning / dusk, sunny / cloudy / rainy days), different observation angles, and different observation distances using step S1. Steps S2-S5 are then used to obtain the cross-modal fusion features of the acquired data. This cross-modal fusion feature is used to construct the sample set, and the corresponding defect category labels are assigned to the cross-modal fusion features. Finally, the constructed sample set is used to train the defect discrimination module.

[0083] After obtaining the defect feature map, it is binarized (e.g., using 0.5 as a threshold, where 1 indicates a defect and 0 indicates normal) to obtain the defect feature matrix. A connectivity algorithm is then used to extract connected defect regions from the defect feature matrix. Finally, a projection mapping matrix is ​​used to project the defect regions onto the point cloud data. Therefore, the three-dimensional spatial coordinates of the defect location can be directly output, or a three-dimensional display can be performed.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A method for insulator defect identification based on cross-modal data fusion, characterized in that, include: Acquire visible light images, thermal infrared images, and point cloud data of the target insulator simultaneously collected by the UAV, as well as the position and attitude information of the UAV during the data collection; Preprocessing of visible light images, thermal infrared images, and point cloud data; Based on the visible light image, all points in the point cloud are projected to obtain the projected image and the projection mapping matrix; After registering both the visible light image and the thermal infrared image with the projected image, the corresponding affine transformation matrix is ​​obtained, and a masking matrix for masking the background is constructed based on the projected image. Feature extraction was performed on the preprocessed visible light image, thermal infrared image and point cloud data to obtain texture feature map, temperature feature map and geometric feature map respectively; The texture feature map, temperature feature map, and geometric feature map are fused based on the occlusion matrix and affine transformation matrix to obtain cross-modal fused features; Cross-modal fusion features are used to identify insulator defects, and the defect locations are inversely mapped to point cloud data.

2. The insulator defect identification method based on cross-modal data fusion according to claim 1, characterized in that, Methods for obtaining the corresponding affine transformation matrix by registering both visible light and thermal infrared images with a projection image include: Identify key geometric feature points from point cloud data; Based on the projection points of key geometric feature points on the visible light image and the projected image, the visible light image and the thermal infrared image are aligned with the projected image using local structure matching, thereby obtaining the affine transformation matrix of the visible light image and the thermal infrared image for aligning the projected image.

3. The insulator defect identification method based on cross-modal data fusion according to claim 2, characterized in that, Methods for identifying key geometric feature points from point cloud data include: For each point in the point cloud, search for all other points within a specified radius centered on that point to form a point set; For each point set, obtain the normal vector of each point, and use the variance of the normal vectors of all points in the point set as the curvature of the center point of the point set; Points with curvature exceeding a set curvature threshold are selected as candidate points. Initialize an empty anchor point set; select candidate points in descending order of curvature; for each selected target candidate point, if the distance between the target candidate point and all points in the anchor point set is greater than the set distance threshold, then add the target candidate point to the anchor point set. Based on the location of the anchor points in the anchor point set, the anchor points are divided into a suspension point set and a grounding wire intersection set; the suspension point set and the grounding wire intersection set are the key geometric feature points for identification.

4. The insulator defect identification method based on cross-modal data fusion according to claim 2, characterized in that, Methods for aligning visible light and thermal infrared images with projection images using local structure matching include: Based on the intrinsic parameters such as focal length and pixel size of the UAV's onboard thermal imaging camera and the extrinsic parameters such as the position and attitude of the UAV when capturing the corresponding image, the theoretical projection points of key geometric feature points on the thermal infrared image are calculated using the perspective projection formula. Obtain the corresponding points of key geometric feature points in the projected image, and use them as the theoretical projection points of the key geometric feature points on the visible light image; Using the theoretical projection point as the center, a calibration window of a specified size is cropped from the visible light image and the thermal infrared image, respectively. Centered on key geometric feature points, a reference window of a specified size is cropped from the projected image; The gradient direction distribution of pixels within each calibration window and reference window is analyzed using the AKAZE algorithm to obtain several key points and their corresponding feature vectors. For each calibration window of the visible light image and the thermal infrared image, calculate the one-to-one similarity between the feature vectors of all key points in the corresponding calibration window and the feature vectors of all key points in the corresponding benchmark window, and select the pair of key points with the highest similarity as the matching point pair; Based on all matching point pairs of the visible light image and the thermal infrared image, the affine transformation matrix of the visible light image and the thermal infrared image is calculated using the least squares method, and the visible light image and the thermal infrared image are aligned to the projected image.

5. The insulator defect identification method based on cross-modal data fusion according to claim 1, characterized in that, Methods for constructing a masking matrix for masking the background based on the projected image include: Construct a masking matrix of the same size based on the projected image. The element values ​​of the projected points and non-projected points of the point cloud in the projected image in the masking matrix represent unmasked and masked points, respectively. The shading matrix is ​​first expanded and then eroded.

6. The insulator defect identification method based on cross-modal data fusion according to claim 1, characterized in that, Methods for feature extraction from preprocessed point cloud data include: Key points are iteratively selected from the point cloud using farthest sampling of the PointNet++ algorithm; For each key point, construct a neighborhood sphere with an initial radius; For each neighborhood sphere, calculate the coordinate offset and intensity difference of each point in the neighborhood relative to the key point to form a feature vector; use a multilayer perceptron with shared weights to process the feature vectors of all neighborhood points to obtain the local feature vector of each point; use max pooling to aggregate the local feature vectors of all points in the neighborhood to obtain the local features of the key point. Increase the radius of the neighborhood sphere step by step, repeat the previous step, and obtain the local features of each key point again. The extracted multi-layer local features of key points are fused hierarchically to obtain the fused features of each key point. For each non-key point in the point cloud, obtain the three closest key points. Use the reciprocal of the distance between the non-key point and the key points as the weight, and sum the fusion features of the three closest key points using the weight to obtain the fusion features of the non-key point. Global pooling is performed on the fused features of all points in the point cloud to obtain the point cloud features; Create a geometric feature map with the same size as the texture feature map and all zeros; For each point in the point cloud data, the corresponding projected coordinates are obtained according to the projection mapping matrix. Based on the projected coordinates, it is determined whether the corresponding element value in the occlusion matrix indicates occlusion. If not, the projected coordinates are downsampled to obtain the corresponding new coordinates. The fused features of the point are written into the corresponding position of the geometric feature map according to the new coordinates.

7. The insulator defect identification method based on cross-modal data fusion according to claim 1, characterized in that, Methods for fusing texture feature maps, temperature feature maps, and geometric feature maps include: The texture feature map and temperature feature map are aligned using an affine transformation matrix; A geometric attention map is constructed using the occlusion matrix and the aligned texture and temperature feature maps. Geometric attention maps are used to weight the aligned texture feature maps and temperature feature maps; The weighted texture feature map, weighted temperature feature map, and geometric feature map are concatenated into a cross-modal concatenated feature; The credibility of each modality is estimated in real time, and the credibility is used to perform modal weighting on the cross-modal splicing features to obtain cross-modal fusion features.

8. The insulator defect identification method based on cross-modal data fusion according to claim 7, characterized in that, Methods for aligning texture feature maps and temperature feature maps using affine transformation matrices include: The original image coordinates are obtained by upsampling the texture feature map and the temperature feature map; Map each original image coordinate to the projected coordinates using the affine transformation matrix; The aligned feature layer coordinates are obtained by downsampling the projected coordinates to the same size as the texture feature map.

9. The insulator defect identification method based on cross-modal data fusion according to claim 7, characterized in that, Methods for constructing geometric attention maps using occlusion matrices and aligned texture and temperature feature maps include: The occlusion matrix is ​​downsampled, and the size of the occlusion matrix is ​​averaged and pooled to match the size of the texture feature map; if the pooled value is greater than the set threshold, the corresponding element value is 1, otherwise it is 0; The aligned texture feature map and temperature feature map are added pixel by pixel to obtain a temporary superimposed feature; The downsampled masking matrix is ​​multiplied pixel by pixel with the temporary overlay feature to obtain the final overlay feature; The final superimposed features are processed using 1×1 convolution and activated by the Sigmoid function to obtain a single-channel geometric attention map.

10. The insulator defect identification method based on cross-modal data fusion according to claim 7, characterized in that, Methods for real-time estimation of the credibility of each modality and modality-weighted summarization of cross-modal concatenation features based on this credibility include: A modal graph containing visible light nodes, thermal infrared nodes, and point cloud nodes is constructed using a graph neural network. The corresponding node values ​​are the global average pooling value of the aligned texture feature map, the global average value of the aligned temperature feature map, and the global average value of the geometric feature map, respectively. The similarity of the three edges of the modal graph is defined as the cosine similarity between two corresponding node values. Local variance statistics are performed on the original visible light image, the overall average variance is taken and normalized to obtain the confidence score of the visible light node; the Sobel gradient is first calculated on the original thermal infrared image, then the average gradient of the whole image is obtained and normalized to obtain the confidence score of the thermal infrared node; the number of geometric feature points in the point cloud data is counted. If the number exceeds the threshold, the confidence score of the point cloud node is 1; otherwise, the confidence score of the point cloud node is the ratio of the number of geometric feature points to the threshold. For each directed edge, the similarity and the confidence of the source node are weighted and summed, and then the softmax function is used to process the edges to obtain the corresponding adaptive edge weights. Each node sums up the features of its directly adjacent nodes according to their corresponding adaptive edge weights, and then updates the new node features. The average of the three new node features is used as the cross-modal fusion feature.