Power distribution line fault diagnosis method based on multi-modal information complementation

Abstract

This invention discloses a fault diagnosis method for power distribution lines based on multimodal information complementarity, belonging to the field of power equipment condition monitoring technology. This method goes beyond simple registration, mapping, temperature extraction, or general fusion of visible light images and infrared thermal images. Instead, it addresses the complex outdoor scenarios of power distribution lines. On one hand, it improves the positioning accuracy of fault subjects in complex backgrounds, weak contrast, and fine-grained structures by combining the Scharr operator with an adaptive multi-scale edge enhancement mechanism for target detection. On the other hand, it preserves temperature edges and subtle thermal textures while suppressing infrared thermal image noise through wavelet denoising. Finally, it achieves a deep complementary expression of visible light image features and infrared thermal imaging features using a cross-self-attention mechanism fusion model containing interference cancellation factors, ultimately obtaining more stable and accurate fault identification results.

Description

Technical Field

[0001] This invention relates to the field of power equipment condition monitoring technology, specifically to a method for fault diagnosis of distribution lines based on multimodal information complementarity. Background Technology

[0002] Power distribution lines are crucial carriers of electrical energy transmission, and their operational stability directly impacts the overall reliability and safety of the power grid. With the rapid development of intelligent inspection technology, the industry has raised higher standards for the non-contact, visual, real-time, and accurate detection of power distribution line faults. Currently, traditional fault detection methods for power distribution lines mainly employ electrical measurement, impedance methods, traveling wave methods, and manual inspection. While these methods have some engineering practical basis, they generally suffer from drawbacks such as high dependence on invasive measurements, slow detection response, weak adaptability to complex scenarios, and low accuracy in fault spatial location, failing to meet the demands of current intelligent inspection operations.

[0003] To address the shortcomings of traditional detection technologies, non-contact dual-modal detection technology based on visible light imaging and infrared thermography has become a key research direction in the field of power distribution line condition monitoring and fault diagnosis. Visible light imaging can clearly characterize the geometric contours and appearance defects of power distribution equipment such as conductors, insulators, and fittings; infrared thermography can accurately capture the surface temperature field distribution of equipment, exhibiting high sensitivity to hidden thermal faults such as insulation aging, poor contact, and localized overheating. The complementary advantages of these two modalities make visible light imaging-infrared thermography joint detection technology the mainstream research direction for intelligent fault diagnosis of power distribution lines. Existing visible light imaging-infrared thermography joint diagnostic schemes mostly follow the technical approach of extracting visible light images and infrared temperature features, and fusing shallow features for identification. However, the overall maturity of existing technologies is low, with several core technical shortcomings, making it difficult to achieve high-precision and high-stability intelligent fault diagnosis of power distribution lines.

[0004] First, single-modal detection has inherent limitations. Visible light images can only identify external defects in equipment and cannot detect hidden thermal faults such as abnormal temperature rise and poor contact resistance. Single infrared thermal images are susceptible to interference from solar radiation, background heat sources, and sensor noise, resulting in problems such as blurred image edges, texture distortion, and poor fault interpretation stability. Second, existing image preprocessing techniques are not targeted enough. They directly use raw visible light images for target detection without setting edge enhancement mechanisms for weak faults and fine-grained structures in complex scenes, which can easily lead to target positioning errors and unclear defect boundary recognition. At the same time, infrared thermal images only use simple filtering and noise reduction processing, which cannot simultaneously meet the core requirements of noise suppression and preservation of temperature edge features.

[0005] Finally, existing dual-modal fusion technologies suffer from insufficient fusion depth, often limited to pixel-level weighting, shallow feature stitching, and basic attention modeling. They fail to deeply explore the complementary correlation between visible light structural features and infrared thermal anomaly features, and lack a scientific feature saliency allocation mechanism. Current technologies have failed to construct a complete technical chain integrating visible light image enhancement and localization, infrared thermal imaging latent thermal fault identification, cross-modal deep interaction, and accurate fault discrimination, significantly limiting the accuracy and stability of power distribution line fault diagnosis. Summary of the Invention

[0006] In view of at least one of the above technical problems, this application provides a fault diagnosis method for distribution lines based on multimodal information complementarity. The aim is to construct a complete technical process of "synchronous acquisition of visible light images and infrared thermal images through dual channels - edge enhancement and target localization by Scharr operator with adaptive multi-scale edge enhancement mechanism - infrared thermal image radiometric correction and wavelet edge preservation denoising - fusion model with cross self-attention mechanism containing interference elimination factor - output of distribution network fault diagnosis results". This enables visible light image features and infrared thermal image features to achieve collaborative modeling around the same fault target, thereby realizing high-precision identification and condition assessment of faults in distribution components such as insulator missingness, insulator damage, foreign object suspension, abnormal and loose hardware structure, local overheating of conductors or hardware, and poor contact.

[0007] The technical solution of this invention is:

[0008] A fault diagnosis method for power distribution lines based on multimodal information complementarity, the method comprising the following steps:

[0009] Simultaneously acquire visible light images and infrared thermal images in power distribution line inspection scenarios;

[0010] Edge enhancement processing is performed on the visible light image to enhance the edge features and structural details of the target component, thus obtaining the edge-enhanced visible light image. ;

[0011] Will Input the pre-trained target detection network to automatically identify and locate the target region;

[0012] Perform radiometric correction and multi-scale wavelet denoising on the infrared thermal image to obtain an infrared thermal image. ;

[0013] Will Target area and infrared thermal image Spatial registration is performed on the corresponding regions in the image, and then cropped according to the boundary of the target region to generate visible photon images corresponding to the same faulty target. and infrared thermal images ;

[0014] Will and Input a cross-self-attention mechanism fusion model containing interference cancellation factors to obtain a fused image with visible light image features and infrared thermal image features;

[0015] The fused images are input into a pre-trained fault diagnosis neural network for power distribution line fault identification.

[0016] Combining the target area location information and the power distribution line fault identification results, the system outputs power distribution line fault diagnosis information including fault location, fault type, and confidence level.

[0017] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, the edge enhancement processing of the visible light image specifically includes the following steps:

[0018] Step 2.1: Perform multi-scale transformation on the visible light image acquired in Step 1, and apply Scharr horizontal convolution kernels at each scale. With vertical convolution kernel Extract the gradient response of the image;

[0019] Step 2.2: Calculate the corresponding gradient magnitude based on the gradient response of the image at each scale;

[0020] Step 2.3: Based on the gradient magnitude at each scale and the adaptive local structure perception mechanism, the edge enhancement intensity of different regions of the visible light image is adaptively adjusted to obtain the edge prior map;

[0021] Step 2.4: Create an edge prior map Acting on the original visible light image The edge-enhanced visible light image is obtained. .

[0022] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, step 2.1 includes:

[0023] Visible light images Perform multi-scale transformation to obtain image representations at different scales. Specifically, this is achieved through the following formula:

[0024] , ;

[0025] ;

[0026] in , and These represent the height, width, and color channels of a visible light image, respectively. Indicates scale index. Indicates the total number of scales. Indicates the first Pyramid scaling operator at various scales This represents the image size resampling function. This represents the corresponding scale factor. This represents the preset scaling reference factor for adjacent scales. Indicates the first The cumulative scaling ratio of each scale relative to the original image;

[0027] Scharr horizontal convolution kernels were used at each scale. With vertical convolution kernel The gradient response of the image is extracted using the following formula:

[0028] ;

[0029] in Indicates the first Images at various scales are processed by Scharr horizontal convolution kernels. Extracted horizontal gradient response; Indicates the first Images at various scales are processed by vertical convolution kernels Extracted vertical gradient response.

[0030] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, step 2.3 includes:

[0031] The method for solving the dynamic weights of each pixel at each scale is as follows: Define the first... The current pixel position in each scale image is Its local neighborhood is The average gray value of the neighborhood is Then pixel The dynamic weight is defined as:

[0032] ;

[0033] in, Indicates the first Pixels at each scale Dynamic weights at each location; Indicates the activation mapping function; , , and Indicates the adjustment coefficient; Indicates the first Pixels at each scale Texture complexity at that location, Indicates the first Pixels at each scale Contrast at that location Indicates the first Pixels at each scale Noise intensity at the location; Indicates the first Pixels at each scale Gradient magnitude at;

[0034] Based on the dynamic weights of each pixel at each scale, the gradient magnitude of each pixel at each scale is weighted and enhanced. Furthermore, the dynamic weights of each pixel at different scales are fused across scales to obtain the edge prior response of each pixel in the visible light image. Finally, the edge prior response of all pixels is combined to obtain the edge prior map of the visible light image of the power distribution line. The edge response of each pixel in the visible light image is obtained through the following formula:

[0035] ;

[0036] in This represents the final edge prior map. Represents pixels Edge prior response at the location; Indicates the first Fusion weights at various scales; Indicates the first An upsampling operator that maps the edge response at each scale to the original resolution is used to restore the edge response map at that scale to the original visible light image resolution. Indicates the first Pixels at each scale The gradient magnitude at that point.

[0037] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, step 2.4 specifically includes:

[0038] edge prior graph Compared with the original visible light image Spatial size uniformity and pixel value normalization are performed to ensure that the edge prior map and the original visible light image correspond one-to-one in pixel position;

[0039] The processed edge prior map The structurally guided weights are then reapplied to the processed visible light image to enhance structurally significant regions while suppressing background false edges and random noise interference, resulting in an edge-enhanced visible light image. ; where pixels in a visible light image The visible light image response after edge enhancement is calculated using the following formula:

[0040] ;

[0041] in, Represents pixels Visible light image response after edge enhancement; This represents the normalization operator; Represents the original visible light image in pixels Pixel value at; Represents pixels Edge prior response at the location; This represents the edge enhancement intensity adjustment coefficient.

[0042] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, the multi-scale wavelet denoising operation includes:

[0043] Radiocorrected thermal imaging signals Two-dimensional wavelet decomposition is performed using the following formula:

[0044] ;

[0045] in For the first Low-frequency approximate subband, They represent the first Horizontal, vertical, and diagonal high-frequency coefficients of the layer;

[0046] An adaptive threshold function is used to shrink the high-frequency coefficients of each layer, specifically implemented using the following formula:

[0047] ;

[0048] in Indicates the number after shrinkage treatment High frequency coefficient of layer, Indicates the number of steps before shrinkage treatment. High-frequency wavelet coefficients of the layer, These correspond to the horizontal, vertical, and diagonal directions, respectively.

[0049] ;

[0050] in This represents the number of pixels in the infrared thermal image. Indicates the first Estimated noise standard deviation for high-frequency wavelet coefficients at each layer; Indicates the median operation;

[0051] Perform multi-scale wavelet reconstruction to obtain an infrared thermal image after multi-scale wavelet denoising. Specifically, it is expressed as follows:

[0052] ;

[0053] in This indicates the number of denoised cells retained after multi-scale wavelet denoising. Low-frequency approximate sub-band; These represent the values ​​after threshold shrinkage. The high-frequency coefficients of the horizontal, vertical and diagonal layers.

[0054] Optionally, according to the power distribution line fault diagnosis method based on multimodal information complementarity, the following steps are performed in the cross-self-attention mechanism fusion model containing interference cancellation factors:

[0055] Step 6.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Let it be mode A, and Let this be mode B. Multidimensional representations of mode A and mode B are extracted using convolutional layers and linear transformations, respectively, to obtain the query vectors corresponding to mode A and mode B. Key vector Sum value vector Specifically, it is expressed by the following formula:

[0056] ;

[0057] Step 6.2: Construct a cross-modal attention mapping so that modality A is updated with weighted features based on the features of modality B, and modality B is updated with weighted features based on the features of modality A, thereby establishing a bidirectional explicit feature dependency relationship, specifically expressed by the following formula:

[0058] ;

[0059] in This means mapping the features of B to A; This means mapping the features of A to B; , which is the scaling factor for the feature dimension;

[0060] Step 6.3: Introduce a learnable interference cancellation factor during the mode A and mode B fusion stage. Dynamic weighting adjustment is applied to the bidirectional explicit feature dependencies, and the fusion result is expressed as follows:

[0061] ;

[0062] Step 6.4: Process the fusion result through a normalization layer and a linear mapping layer to obtain the final fused image with visible light image features and infrared thermal image features.

[0063] The beneficial effects of adopting the above technical solution are as follows:

[0064] (1) The present invention adopts a non-contact detection method that combines visible light images and infrared thermal images, which can complete fault location, thermal anomaly perception and status recognition without affecting the normal operation of the equipment, reducing manual dependence and improving the level of intelligent inspection.

[0065] (2) This invention improves the ability to characterize structural boundaries under complex backgrounds, weak targets and low contrast conditions by combining the Scharr operator of the adaptive multi-scale edge enhancement mechanism, edge prior embedding and target detection in visible light front-end processing, thereby helping to improve the accuracy of fault location and the stability of subsequent cross-modal regions.

[0066] (3) The present invention uses the infrared thermal image preprocessing strategy of "radiative correction, two-dimensional wavelet multi-scale denoising and adaptive threshold shrinkage" to suppress noise and background interference while retaining temperature gradient changes and fine thermal textures in the infrared thermal image, thereby achieving edge preservation and noise reduction of the infrared thermal image and enhancing the distinguishability of thermal anomaly information.

[0067] (4) Based on the visible light detection results, establish the correspondence between the visible light image and the infrared thermal image dual-modal region, so that the subsequent fusion process always faces the same physical target and reduces the diagnostic error caused by cross-modal mismatch.

[0068] (5) By constructing a cross-self-attention mechanism fusion model containing interference cancellation factor, bidirectional interactive modeling of visible light image and infrared thermal image is performed, an explicit dependency relationship between visible light image features and infrared thermal image features is established, and a learnable interference cancellation factor is introduced to dynamically adjust the bimodal contribution, so as to achieve more sufficient cross-modal complementary expression and stable alignment, thereby more effectively suppressing the interference of background heat source, environmental noise and modal redundancy information, and improving the accuracy and robustness of fault diagnosis.

[0069] (6) The fusion features obtained by the fusion model through the cross self-attention mechanism can be connected to different diagnostic networks such as convolutional neural networks, residual networks, Transformer networks, and Swing Transformer networks, and have good model compatibility, engineering deployment flexibility and promotion application value. Attached Figure Description

[0070] Figure 1 This is a flowchart of the power distribution line fault diagnosis method based on multimodal information complementarity in this embodiment. Detailed Implementation

[0071] To facilitate understanding of this application, a more comprehensive description of this application will be provided below with reference to the accompanying drawings.

[0072] The core idea of ​​this application is as follows: The method proposed in this application does not merely stop at simple registration, mapping, temperature extraction, or general fusion of visible light images and infrared thermal images. Instead, it is designed for complex outdoor scenarios of power distribution lines and constructs a complete technical process: "Simultaneous acquisition of visible light images and infrared thermal images through dual channels — edge enhancement and target localization using the Scharr operator with an adaptive multi-scale edge enhancement mechanism — infrared thermal image radiometric correction and wavelet edge-preserving denoising — fusion model with a cross-self-attention mechanism containing interference elimination factors — output of power distribution network fault diagnosis results." On the one hand, by combining the Scharr operator with an adaptive multi-scale edge enhancement mechanism for target detection, the positioning accuracy of the fault subject under complex backgrounds, weak contrast, and fine-grained structure conditions is improved. On the other hand, wavelet denoising suppresses infrared thermal image noise while preserving temperature edges and subtle thermal textures. Then, by using a fusion model with a cross-self-attention mechanism containing interference elimination factors, a deep complementary expression of visible light image features and infrared thermal imaging features is achieved, ultimately obtaining more stable and accurate fault identification results.

[0073] Figure 1 This is a flowchart of the power distribution line fault diagnosis method based on multimodal information complementarity in this embodiment. Figure 1 As shown, this embodiment of the power distribution line fault diagnosis method based on multimodal information complementarity includes the following steps:

[0074] Step 1: Simultaneously acquire visible light images and infrared thermal images in the power distribution line inspection scenario;

[0075] This implementation uses a high-resolution industrial camera to acquire images of the target area of ​​the power distribution line. Through visible light imaging technology, it comprehensively acquires the spatial layout, geometric contours, and surface texture information of key power distribution components such as conductors, insulators, hardware, and various connectors, providing structured data support for subsequent equipment component identification and appearance defect detection. Simultaneously, a thermal imager is used to synchronously acquire infrared thermal images of the same monitoring area, accurately capturing the global temperature distribution information of the power distribution equipment surface and effectively extracting potential thermal anomalies such as localized overheating, poor contact, and insulation aging.

[0076] In a preferred embodiment, both visible light and thermal infrared sensors undergo time synchronization calibration and spatial positioning calibration to ensure complete registration of multimodal images in terms of acquisition time, imaging field of view, and target area. This provides a reliable data foundation for subsequent image preprocessing, cross-modal feature fusion, and accurate fault determination. The image resolution can be flexibly set according to the detection distance, target size, and recognition accuracy. Preferably, the visible light image resolution is no less than 1920×1080 pixels, and the infrared thermal image resolution is no less than 320×240 pixels.

[0077] Step 2: Perform edge enhancement processing on the visible light image acquired in Step 1 to enhance the edge features and structural details of the target component, weaken the interference of complex background, and obtain the edge-enhanced visible light image;

[0078] This implementation addresses the problems of low image contrast, blurred edges, and complex backgrounds in outdoor scenes, as well as the limitations of traditional Scharr operators, which have fixed kernels, single scales, and insufficient adaptability to complex backgrounds. It applies an adaptive multi-scale edge enhancement mechanism to visible light images using the Scharr operator. By calculating the gray-level gradients in the horizontal and vertical directions, it extracts gradient responses under different receptive fields, reducing interference from local texture complexity, abnormal contrast, and noise, and generating adaptive dynamic weights. It selectively enhances target edges while suppressing false edges and noise responses in the background. This enables edge enhancement and detail optimization of acquired visible light images of power distribution lines, highlighting the target edge contours and fine structural features of power distribution components while mitigating interference from complex backgrounds.

[0079] This step specifically includes the following steps:

[0080] Step 2.1: Perform multi-scale transformation on the visible light image acquired in Step 1, and apply Scharr horizontal convolution kernels at each scale. With vertical convolution kernel Extract the gradient response of the image;

[0081] In this embodiment, the visible light image acquired in step 1 is represented as... ,in, , and These represent the height, width, and color channels of a visible light image, respectively. First, the image... Perform multi-scale transformation to obtain image representations at different scales. And Scharr horizontal convolution kernels are used at each scale. With vertical convolution kernel The gradient response of the image is extracted using the following formula:

[0082] ;

[0083] in, Indicates scale index. Indicates the total number of scales. Indicates the first Pyramid scaling operator at various scales; Represents the image size resampling function; This represents the corresponding scale factor. This represents the preset scaling reference factor for adjacent scales; Indicates the first The cumulative scaling ratio of each scale relative to the original image; Indicates the first Images at various scales are processed by Scharr horizontal convolution kernels. Extracted horizontal gradient response; Indicates the first Images at various scales are processed by vertical convolution kernels Extracted vertical gradient response.

[0084] Step 2.2: Calculate the corresponding gradient magnitude based on the gradient response of the image at each scale:

[0085] ;

[0086] in, Indicates the first Gradient magnitude at each scale This is a preset minimum positive number.

[0087] Step 2.3: Based on the gradient magnitude at each scale and the adaptive local structure perception mechanism, the edge enhancement intensity of different regions of the visible light image is adaptively adjusted to obtain the edge prior map;

[0088] Unlike the traditional single-scale fixed response method, this implementation further introduces an adaptive local structure perception mechanism to adaptively adjust the edge enhancement intensity of different regions of the visible light image.

[0089] First, the dynamic weights of each pixel at each scale are calculated. The method is as follows: Let the weight of the pixel at each scale be... The current pixel position at each scale is Its local neighborhood is The average gray value of the neighborhood is Then the first Pixels at each scale The dynamic weight is defined as:

[0090] ;

[0091] in, Indicates the first Pixels at each scale Dynamic weights at each location; The activation mapping function can be represented by the Sigmoid function. , , and The adjustment coefficient is obtained through the training process of the adaptive local structure perception mechanism. Indicates the first Pixels at each scale Texture complexity at that location, Indicates the first Pixels at each scale Contrast at that location Indicates the first Pixels at each scale Noise intensity at the location; Indicates the first Pixels at each scale The gradient magnitude at that point.

[0092] By using the aforementioned adaptive local structure perception mechanism with adjustable coefficients, image regions with clear target contours and high local contrast can receive greater enhancement weights. At the same time, it suppresses background false edges and areas with strong noise interference, reducing the adverse effects of complex backgrounds and image noise on component detection and fault location.

[0093] After obtaining the dynamic weights of each pixel at each scale, the gradient magnitudes of each pixel at each scale are weighted and enhanced based on these dynamic weights. Furthermore, cross-scale fusion of the dynamic weights of pixels at different scales is performed to obtain the edge prior responses of each pixel in the visible light image. Finally, the edge prior responses of all pixels are combined to obtain the edge prior map of the visible light image of the power distribution line. The edge response of each pixel in the visible light image is obtained through the following formula:

[0094] ;

[0095] in, Represents pixels Edge prior response at the location; Indicates the first Fusion weights at various scales; Indicates the first An upsampling operator that maps the edge response at each scale to the original resolution is used to restore the edge response map at that scale to the original visible light image resolution. Indicates the first Pixels at each scale The gradient magnitude at that point.

[0096] This method can simultaneously preserve fine-grained edge details and large-scale contour information, improving the integrity and stability of edge representation.

[0097] Step 2.4: Create an edge prior map Acting on the original visible light image The edge-enhanced visible light image is obtained. ;

[0098] Edge prior map for obtaining visible light images of power distribution lines Then, the edge prior map Compared with the original visible light image Spatial size uniformity and pixel value normalization are performed to ensure a one-to-one correspondence between the edge prior map and the original visible light image at pixel locations. Subsequently, the processed edge prior map... As structural guiding weights, these weights are reapplied to the processed visible light image to enhance structurally significant regions such as conductor boundaries, insulator skirt outlines, hardware connection locations, and connector outer edges. This process also suppresses background false edges and random noise interference, resulting in an edge-enhanced visible light image. Pixels in the visible light image The visible light image response after edge enhancement is calculated using the following formula:

[0099] ;

[0100] in, Represents pixels Visible light image response after edge enhancement; This represents the normalization operator; Represents the original visible light image in pixels Pixel value at; Represents pixels Edge prior response at the location; This represents the edge enhancement intensity adjustment coefficient, which is obtained through the subsequent object detection network training process.

[0101] For color visible light images, the enhancement operations described above can be applied to the red, green, and blue color channels respectively. This processing method enhances the boundary structure and detailed features of key components in power distribution lines without compromising the overall brightness and texture information of the original visible light image. This provides clear-edged and structurally stable visible light images for subsequent image preprocessing, component identification, defect detection, and fault location. .

[0102] Step 3: Input the edge-enhanced visible light image into the pre-trained target detection network to automatically complete the identification and localization of the target region.

[0103] In this embodiment, the edge-enhanced visible light image is input into a pre-trained target detection network such as YOLOv8 to automatically complete the accurate identification and localization of the target region. Specifically, the target region in the image is selected and its category and corresponding confidence level are labeled.

[0104] Step 4: Perform radiometric correction and multi-scale wavelet denoising on the infrared thermal image acquired in Step 1.

[0105] Considering that infrared thermal images are susceptible to environmental radiation, specular reflection, background temperature changes, and sensor response non-uniformity, this embodiment first performs radiometric correction on the infrared thermal image acquired in step 1. Those skilled in the art will readily know that the radiometric correction operation includes emissivity setting, background temperature compensation, and non-uniformity correction. The radiometric correction operation can eliminate various interference factors and make the thermal image signal more accurately reflect the radiation temperature distribution of the target area surface.

[0106] Then, the radiation-corrected thermal image signal is Two-dimensional wavelet decomposition is performed using the following formula:

[0107] ;

[0108] in, For the first Low-frequency approximate subband, They represent the first The high-frequency coefficients of the horizontal, vertical and diagonal layers.

[0109] To effectively suppress random noise and preserve the edge features of infrared thermal images, this implementation design employs an adaptive threshold function to shrink the high-frequency coefficients of each layer, specifically implemented using the following formula:

[0110] ;

[0111] in Indicates the number of digits after threshold shrinkage. High frequency coefficients of the layer Indicates the number of seconds before threshold shrinkage. High-frequency wavelet coefficients of the layer, These correspond to the horizontal, vertical, and diagonal directions, respectively. Determined adaptively based on noise statistical characteristics:

[0112] ;

[0113] in This represents the number of pixels in the infrared thermal image. Indicates the first Estimated noise standard deviation for high-frequency wavelet coefficients at each layer; This indicates the median operation.

[0114] This implementation achieves feature-preserving denoising under noise constraints in the wavelet domain through an adaptive threshold function.

[0115] Performing multi-scale wavelet reconstruction operations yields an infrared thermal image after multi-scale wavelet denoising, which can be represented as... Specifically, it is expressed as follows:

[0116] ;

[0117] in This indicates the number of denoised cells retained after multi-scale wavelet denoising. Low-frequency approximate sub-band; These represent the values ​​after threshold shrinkage. The high-frequency coefficients of the horizontal, vertical and diagonal layers.

[0118] This processing effectively suppresses noise in infrared thermal images while preserving local thermal gradient changes and subtle texture information, thereby enhancing the structural discriminability and boundary continuity of the temperature field. Infrared thermal image after multi-scale wavelet denoising. It can more clearly reflect the temperature distribution characteristics of the surface of power distribution equipment and the location of potential abnormal heating points, providing a high-quality input data foundation for subsequent feature extraction, status identification and fault diagnosis.

[0119] Step 5: Obtain the visible light image determined in Step 3. The target region in the image is the infrared thermal image output from step 4 after multi-scale wavelet denoising. Spatial registration is performed on the corresponding regions in the image, and then cropped according to the boundary of the target region to generate visible photon images corresponding to the same faulty target. and infrared thermal images ;

[0120] This implementation method, based on the accurate identification of key components of the power distribution line and the location of the target area obtained from target detection in step 3, uses the visible light image determined in step 3... The target region in the image is the infrared thermal image output from step 4 after multi-scale wavelet denoising. Spatially register the corresponding regions in the image, then crop according to the target region boundary to generate spatially registered visible photon images of the same faulty target. and infrared thermal images This ensures that the subsequent fusion process is carried out on the same physical object.

[0121] Step 6: Construct a cross-self-attention mechanism fusion model containing an interference cancellation factor to obtain a fused image with visible light image features and infrared thermal image features.

[0122] This step involves spatially registered visible photon images. Dedicated to mode A, the spatially registered infrared thermal image sub-image Let this be modality B. The corresponding query vector, key vector, and value vector are extracted through convolutional layers and linear mapping. Then, a cross-modal attention mapping is constructed, enabling modality A to be updated with weighted features based on the features of modality B, and vice versa, thus establishing a bidirectional explicit feature dependency. To reduce background interference and the problem of excessive dominance by a single modality, a learnable interference cancellation factor is introduced in the modality A and modality B fusion stage to dynamically adjust the bidirectional explicit feature dependency. The fusion result is then processed through a normalization layer and a linear mapping layer to obtain a unified multi-source fusion feature representation. This step specifically includes the following steps:

[0123] Step 6.1: Spatially registered visible photon image Dedicated to mode A, the spatially registered infrared thermal image sub-image Let this be mode B. Multidimensional representations of mode A and mode B are extracted using convolutional layers and linear transformations, respectively, to obtain the query vectors corresponding to mode A and mode B. Key vector Value vector Specifically, it is expressed by the following formula:

[0124] ;

[0125] Step 6.2: Construct a cross-modal attention mapping so that modality A is updated with weighted features based on the features of modality B, and modality B is updated with weighted features based on the features of modality A, thereby establishing a bidirectional explicit feature dependency relationship;

[0126] in For learnable projection matrices, These correspond to modalities A and B, respectively, and are used to map image features from modalities A and B to the attention space. Unlike traditional self-attention mechanisms where (Q, K, V) come from the same modality, the cross-self-attention fusion model with interference cancellation factors uses a cross-self-attention mechanism to construct attention maps through cross-modal features to establish explicit feature dependencies. The calculation process for bidirectional explicit feature dependencies can be expressed as:

[0127] ;

[0128] in This means mapping the features of B to A; This means mapping the features of A to B; , which is a scaling factor for the feature dimension, is used to ensure the stable convergence of the cross-self-attention mechanism fusion model containing interference cancellation factors. The cross-self-attention mechanism guides the attention weights of modality A with the semantic features of modality B, and vice versa, thereby achieving bidirectional semantic response enhancement. Through this cross-modeling, complementary regions and structural correspondences between multimodal features can be adaptively captured, achieving explicit alignment of information and feature enhancement between different modalities.

[0129] Step 6.3: In the mode A and mode B fusion stage, a learnable interference elimination factor is introduced to dynamically weight and adjust the bidirectional explicit feature dependency relationship;

[0130] In the mode A and mode B fusion stage, to reduce significant background interference from visible light images, this implementation introduces an interference cancellation factor. This is to achieve dynamic weighting and adaptive adjustment of the output of the cross-self-attention mechanism. The final fusion result can be expressed as:

[0131] ;

[0132] in The network of the fusion model can be automatically learned and optimized through a cross-self-attention mechanism to ensure optimal information fusion in different scenarios. When the cross-self-attention mechanism fusion model containing interference cancellation factors outputs a fused image result biased towards the guided features of mode A; when At that time, the output fused image of the cross-self-attention mechanism fusion model containing interference cancellation factor depends more on the structural supplementation of mode B.

[0133] Step 6.4: Process the fusion result through a normalization layer and a linear mapping layer to obtain the final fused feature map;

[0134] The output fused image of the cross-self-attention mechanism fusion model containing interference cancellation factors is further optimized by a normalization layer and a linear mapping layer to restore channel consistency and enhance feature robustness.

[0135] Step 7: Input the fused image output by the cross-self-attention mechanism fusion model containing interference cancellation factor obtained in Step 6 into the pre-trained fault diagnosis neural network for power distribution line fault identification.

[0136] The fault diagnosis neural network can be a convolutional neural network, residual network, Transformer network, SwinTransformer network, or other deep network structures suitable for image classification. The fault identification results for power distribution lines may include, but are not limited to, missing insulators, damaged insulators, suspended foreign objects, abnormal and loose hardware structures, localized overheating of conductors or hardware, and poor contact faults.

[0137] Step 8: Combining the target area location obtained in Step 3 and the power line fault identification results obtained in Step 7, output power line fault diagnosis information including fault location, fault type, and confidence level. This implementation combines the target bounding box location in Step 3 and the power line fault identification results in Step 7 to form power line fault diagnosis information including fault location, fault type, and confidence level, which is used for online inspection, condition assessment, and operation and maintenance decision support of power lines.

[0138] In summary, the essential difference between the method of this application and existing technologies lies not in the improvement of a single detection model or a single fusion method, but in the establishment of a multimodal collaborative cognitive mechanism for power distribution line fault diagnosis. This mechanism ensures that the structural representation capability of visible light and the thermal anomaly characterization capability of infrared light are no longer simply superimposed, but rather unified in modeling and joint discrimination based on the intrinsic characteristics of the fault. Therefore, it can fundamentally improve the accuracy, robustness, and engineering practicality of fault identification in complex scenarios. Furthermore, the method of this application exhibits stable and superior identification performance under different data sources, different environmental conditions, and different subsequent diagnostic network structures, indicating that this invention not only improves the identification accuracy of potential faults in power distribution lines, but also has good versatility, scalability, and engineering application value.

[0139] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A method for fault diagnosis of power distribution lines based on multimodal information complementarity, characterized in that, The method includes the following steps: Simultaneously acquire visible light images and infrared thermal images in power distribution line inspection scenarios; Edge enhancement processing is performed on the visible light image to enhance the edge features and structural details of the target component, thus obtaining the edge-enhanced visible light image. ; Will Input the pre-trained target detection network to automatically identify and locate the target region; Perform radiometric correction and multi-scale wavelet denoising on the infrared thermal image to obtain an infrared thermal image. ; Will Target area and infrared thermal image Spatial registration is performed on the corresponding regions in the image, and then cropped according to the boundary of the target region to generate visible photon images corresponding to the same faulty target. and infrared thermal images ; Will and Input a cross-self-attention mechanism fusion model containing interference cancellation factors to obtain a fused image with visible light image features and infrared thermal image features; The fused images are input into a pre-trained fault diagnosis neural network for power distribution line fault identification. Combining the target area location information and the power distribution line fault identification results, the system outputs power distribution line fault diagnosis information including fault location, fault type, and confidence level.

2. The method for fault diagnosis of power distribution lines based on multimodal information complementarity according to claim 1, characterized in that, The edge enhancement processing of the visible light image specifically includes the following steps: Step 2.1: Perform multi-scale transformation on the visible light image acquired in Step 1, and apply Scharr horizontal convolution kernels at each scale. With vertical convolution kernel Extract the gradient response of the image; Step 2.2: Calculate the corresponding gradient magnitude based on the gradient response of the image at each scale; Step 2.3: Based on the gradient magnitude at each scale and the adaptive local structure perception mechanism, the edge enhancement intensity of different regions of the visible light image is adaptively adjusted to obtain the edge prior map; Step 2.4: Create an edge prior map Acting on the original visible light image The edge-enhanced visible light image is obtained. .

3. The method for fault diagnosis of power distribution lines based on multimodal information complementarity according to claim 2, characterized in that, Step 2.1 includes: Visible light images Perform multi-scale transformation to obtain image representations at different scales. Specifically, this is achieved through the following formula: ， ； ； in , and These represent the height, width, and color channels of a visible light image, respectively. Indicates scale index. Indicates the total number of scales. Indicates the first Pyramid scaling operator at various scales Represents the image size resampling function. This represents the corresponding scale factor. This represents the preset scaling reference factor for adjacent scales. Indicates the first The cumulative scaling ratio of each scale relative to the original image; Scharr horizontal convolution kernels were used at each scale. With vertical convolution kernel The gradient response of the image is extracted using the following formula: ； in Indicates the first Images at various scales are processed by Scharr horizontal convolution kernels. Extracted horizontal gradient response; Indicates the first Images at various scales are processed by vertical convolution kernels Extracted vertical gradient response.

4. The method for fault diagnosis of power distribution lines based on multimodal information complementarity according to claim 3, characterized in that, Step 2.3 includes: The method for solving the dynamic weights of each pixel at each scale is as follows: Define the first... The current pixel position in each scale image is Its local neighborhood is The average gray value of the neighborhood is Then the pixel The dynamic weight is defined as: ； in, Indicates the first Pixels at each scale Dynamic weights at each location; Indicates the activation mapping function; , , and Indicates the adjustment coefficient; Indicates the first Pixels at each scale Texture complexity at that location, Indicates the first Pixels at each scale Contrast at that location Indicates the first Pixels at each scale Noise intensity at the location; Indicates the first Pixels at each scale Gradient magnitude at; Based on the dynamic weights of each pixel at each scale, the gradient magnitude of each pixel at each scale is weighted and enhanced. Furthermore, the dynamic weights of each pixel at different scales are fused across scales to obtain the edge prior response of each pixel in the visible light image. Finally, the edge prior response of all pixels is combined to obtain the edge prior map of the visible light image of the power distribution line. The edge response of each pixel in the visible light image is obtained through the following formula: ； in This represents the final edge prior map. Represents pixels Edge prior response at the location; Indicates the first Fusion weights at various scales; Indicates the first An upsampling operator that maps the edge response at each scale to the original resolution is used to restore the edge response map at that scale to the original visible light image resolution. Indicates the first Pixels at each scale The gradient magnitude at that point.

5. The power distribution line fault diagnosis method based on multimodal information complementarity according to claim 4, characterized in that, Step 2.4 specifically includes: edge prior graph Compared with the original visible light image Spatial size uniformity and pixel value normalization are performed to ensure that the edge prior map and the original visible light image correspond one-to-one in pixel position; The processed edge prior map The structurally guided weights are then reapplied to the processed visible light image to enhance structurally significant regions while suppressing background false edges and random noise interference, resulting in an edge-enhanced visible light image. ; where pixels in a visible light image The visible light image response after edge enhancement is calculated using the following formula: ； in, Represents pixels Visible light image response after edge enhancement; This represents the normalization operator; Represents the original visible light image in pixels Pixel value at; Represents pixels Edge prior response at the location; This represents the edge enhancement intensity adjustment coefficient.

6. The method for fault diagnosis of power distribution lines based on multimodal information complementarity according to claim 1, characterized in that, The multi-scale wavelet denoising operation includes: Radiocorrected thermal imaging signals Two-dimensional wavelet decomposition is performed using the following formula: ； in For the first Low-frequency approximate subband, They represent the first Horizontal, vertical, and diagonal high-frequency coefficients of the layer; An adaptive threshold function is used to shrink the high-frequency coefficients of each layer, specifically implemented using the following formula: ； in Indicates the number after shrinkage treatment Layer high frequency coefficient, Indicates the number of steps before shrinkage treatment. High-frequency wavelet coefficients of the layer, These correspond to the horizontal, vertical, and diagonal directions, respectively. ； in This represents the number of pixels in the infrared thermal image. Indicates the first Estimated noise standard deviation for high-frequency wavelet coefficients at each layer; Indicates the median operation; Perform multi-scale wavelet reconstruction to obtain an infrared thermal image after multi-scale wavelet denoising. Specifically, it is expressed as follows: ； in This indicates the number of denoised cells retained after multi-scale wavelet denoising. Low-frequency approximate sub-band; These represent the values ​​after threshold shrinkage. The high-frequency coefficients of the horizontal, vertical and diagonal layers.

7. The method for fault diagnosis of power distribution lines based on multimodal information complementarity according to claim 1, characterized in that, The following steps are performed in the cross-self-attention mechanism fusion model containing the interference cancellation factor: Step 6.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Let it be mode A, and Let this be mode B. Multidimensional representations of mode A and mode B are extracted using convolutional layers and linear transformations, respectively, to obtain the query vectors corresponding to mode A and mode B. Key vector Sum value vector Specifically, it is expressed by the following formula: ； Step 6.2: Construct a cross-modal attention mapping so that modality A is updated with weighted features based on the features of modality B, and modality B is updated with weighted features based on the features of modality A, thereby establishing a bidirectional explicit feature dependency relationship, specifically expressed by the following formula: ； in This means mapping the features of B to A; This means mapping the features of A to B; , which is the scaling factor for the feature dimension; Step 6.3: Introduce a learnable interference cancellation factor during the mode A and mode B fusion stage. Dynamic weighting adjustment is applied to the bidirectional explicit feature dependencies, and the fusion result is expressed as follows: ； Step 6.4: Process the fusion result through a normalization layer and a linear mapping layer to obtain the final fused image with visible light image features and infrared thermal image features.

Images