Defect detection method and system based on insulator multi-modal image fusion

Abstract

The invention discloses a defect detection method and system based on insulator multi-modal image fusion, and relates to the technical field of electrical equipment defect detection, multi-modal image fusion is performed based on an attention mechanism improved RFN-Nest image fusion model, fusion of infrared image temperature anomaly features and visual image structure detail features is enhanced, and the defect detection accuracy is improved. The information entropy, mutual information and other indexes of the generated fusion image are remarkably superior to those of an original model and a traditional fusion method, high-quality data support is provided for a detection task, then defect detection is carried out based on an attention mechanism improved YOLOv8 target detection model, the defect feature discrimination capability and the anti-interference capability are improved, and the detection efficiency is improved. The problems of missing report and false report of insulator defects in a complex scene are effectively solved, the performance is remarkably improved compared with a traditional independent link design scheme, and the method can be directly applied to an actual electric power inspection scene.

Description

Technical Field

[0001] This application relates to the field of power equipment defect detection technology, specifically to a defect detection method and system based on insulator multimodal image fusion. Background Technology

[0002] Insulators, as core components providing electrical insulation and mechanical support in power systems, are widely used in critical facilities such as transmission lines and substations. Their performance stability directly determines the safety and efficiency of power transmission. Long-term exposure to complex outdoor environments makes insulators susceptible to strong electric fields, acid rain, temperature changes, and other factors, leading to defects such as cracks, damage, and abnormal heating. This can result in serious accidents such as flashovers and line tripping, causing significant economic losses and social impact. Therefore, developing efficient and accurate insulator defect detection technology is of great significance for ensuring the stable operation of power systems.

[0003] With the development of computer vision and deep learning technologies, detection schemes based on multimodal images have become a research hotspot. Although current related technologies have made some progress, significant bottlenecks still exist: In the field of image fusion, deep learning models represented by RFN-Nest can achieve multi-scale feature fusion, but in the encoder output features, key information related to insulator defects (such as infrared high-temperature points and visible crack edges) is easily submerged by background redundancy information, lacking a targeted feature enhancement mechanism; In the field of target detection, mainstream algorithms such as YOLOv8, while balancing speed and accuracy, are insufficient in identifying defects in complex environments such as fine cracks less than 5 pixels wide and foggy weather when applied to insulator detection. The core reason is that their backbone network has not been optimized for the feature sparsity of insulator defects; In addition, existing research often treats image fusion and target detection as independent steps, lacking a "fusion-detection" collaborative design, resulting in a low degree of matching between fused features and detection requirements, making it difficult to fully utilize the value of multimodal data.

[0004] Existing technologies and traditional detection methods suffer from low efficiency and poor robustness in complex environments, failing to meet the needs of real-time UAV inspection. Furthermore, they suffer from high rates of missed and false alarms. Therefore, there is an urgent need for an integrated technical solution that can enhance key feature fusion, improve defect discrimination capabilities, and achieve synergistic optimization of fusion and detection. Summary of the Invention

[0005] The purpose of this application is to provide a defect detection method and system based on multimodal image fusion of insulators, which solves the problems of high false alarm and high false false alarm rates and poor adaptability to complex environments of traditional methods.

[0006] This application is achieved through the following technical solution:

[0007] The first aspect of this application provides a defect detection method based on multimodal image fusion of insulators, including:

[0008] Infrared and visual images of the insulator are acquired, and the infrared and visual images are preprocessed to obtain standardized image data.

[0009] An improved RFN-Nest image fusion model based on an attention mechanism is constructed, and the RFN-Nest image fusion model is trained using a two-stage training strategy to obtain the trained RFN-Nest image fusion model.

[0010] The trained RFN-Nest image fusion model is invoked to fuse the standardized image data, resulting in a high-quality fused image.

[0011] An improved YOLOv8 target detection model based on an attention mechanism is constructed, and the YOLOv8 target detection model is trained using a pre-stored insulator defect dataset to obtain the trained YOLOv8 target detection model.

[0012] The trained YOLOv8 target detection model is invoked to identify the high-quality fused image, and feature extraction, attention enhancement, feature fusion, prediction and post-processing optimization are performed to obtain the insulator defect detection result;

[0013] Based on the defect type, confidence level, and bounding box information in the insulator defect detection results, the insulator operating status is determined and a detection report is generated.

[0014] In one possible implementation, the infrared image and the visual image are preprocessed to obtain standardized image data, including:

[0015] The infrared image and the visual image are sequentially denoised to obtain a denoised infrared image and a denoised visual image; the denoising process uses a bilateral filtering algorithm.

[0016] The denoised visual image is then converted to grayscale to obtain a grayscale visual image; the grayscale conversion process uses a weighted average method to convert the color visual image into a grayscale image.

[0017] The denoised infrared image and the grayscale visual image are used as standardized image data.

[0018] In one possible implementation, an improved RFN-Nest image fusion model based on an attention mechanism is constructed, and the RFN-Nest image fusion model is trained using a two-stage training strategy to obtain the trained RFN-Nest image fusion model, including:

[0019] An EMA attention mechanism module is embedded between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an improved RFN-Nest image fusion model based on the attention mechanism.

[0020] The RFN-Nest image fusion model is trained using a two-stage training strategy to obtain the trained RFN-Nest image fusion model.

[0021] In one possible implementation, an EMA attention mechanism module is embedded between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an attention-based improved RFN-Nest image fusion model, including:

[0022] Construct an encoder network, an EMA attention mechanism module, an RFN network, and a decoder network; wherein the encoder network is configured as a convolutional neural network;

[0023] The encoder network and the decoder network are connected sequentially through the EMA attention mechanism module and the RFN network to form an RFN-Nest image fusion model based on an improved attention mechanism.

[0024] In one possible implementation, training the RFN-Nest image fusion model using a two-stage training strategy to obtain the trained RFN-Nest image fusion model includes:

[0025] The parameters of the RFN network in the attention-based improved RFN-Nest image fusion model are frozen. The autoencoder network, composed of the encoder, EMA attention mechanism module, and decoder in the attention-based improved RFN-Nest image fusion model, is trained with the goal of minimizing the first loss function, resulting in the RFN-Nest image fusion model after one stage of training. The first loss function is: ; For pixel loss, The structural similarity loss is represented by λ, where λ is the weighting coefficient.

[0026] The parameters of the autoencoder network in the RFN-Nest image fusion model after the first stage of training are frozen, and the RFN network is trained with the goal of minimizing the second loss function to obtain the RFN-Nest image fusion model after the second stage of training, thus completing the training; the second loss function is: ; Loss of background detail Enhance the loss for the target features.

[0027] In one possible implementation, a YOLOv8 target detection model based on an improved attention mechanism is constructed, comprising: integrating the CBAM attention mechanism after the C2f module of the backbone network in the original YOLOv8 target detection model to obtain a YOLOv8 target detection model based on an improved attention mechanism; wherein the CBAM attention mechanism includes a channel attention submodule and a spatial attention submodule.

[0028] In one possible implementation, the YOLOv8 target detection model is trained using a pre-stored insulator defect dataset to obtain the following:

[0029] The experimental environment was built based on the PyTorch deep learning framework, with an initial learning rate of 0.001. Using a pre-stored insulator defect dataset as a foundation, the YOLOv8 target detection model was trained using the SGD optimizer. During training, the momentum coefficient was set to 0.9, the weight decay coefficient to 5e-4, the batch size to 16, and the total number of training iterations to 100 epochs. The first 80 epochs were the normal training phase, and the learning rate was linearly decayed to 1e-6 in the last 20 epochs. The average accuracy and loss values ​​corresponding to the pre-stored validation set were used as monitoring metrics during training. Model weights were saved every 5 epochs, triggering an early stopping mechanism to prevent invalid training.

[0030] In one possible implementation, the post-processing optimization includes: setting the confidence threshold to 0.5, the IoU threshold to 0.3, and using a non-maximum suppression algorithm to remove redundant prediction boxes. At the same time, for small target defects with a width of <5 pixels, the IoU threshold of the non-maximum suppression algorithm is reduced to 0.25.

[0031] The insulator defect detection results include target category, confidence level, and bounding box coordinates. The target category includes normal insulators, cracks, damage, contamination, and abnormal heating.

[0032] In one possible implementation, based on the defect type, confidence level, and bounding box information in the insulator defect detection results, the insulator operating status is determined and a detection report is generated, including:

[0033] Based on the defect type, confidence level, and bounding box information in the insulator defect detection results, the state of the target insulator is determined, and the target suggestion corresponding to the pre-stored target insulator state is obtained.

[0034] Based on the defect type, confidence level, bounding box information, and target suggestions, a detection report is generated that includes a defect location annotation map, defect type statistics, confidence level distribution, and processing suggestions.

[0035] Based on the same inventive concept, a second aspect of this application provides a defect detection system based on insulator multimodal image fusion, comprising:

[0036] The data acquisition and preprocessing module is used to acquire infrared images and visual images corresponding to the insulator, and to preprocess the infrared images and visual images to obtain standardized image data.

[0037] The image fusion module is used to construct an improved RFN-Nest image fusion model based on an attention mechanism, and to train the RFN-Nest image fusion model through a two-stage training strategy to obtain the trained RFN-Nest image fusion model; the trained RFN-Nest image fusion model is then used to perform fusion processing on the standardized image data to obtain a high-quality fused image;

[0038] The target detection module is used to construct a YOLOv8 target detection model based on an improved attention mechanism, and to train the YOLOv8 target detection model with a pre-stored insulator defect dataset to obtain a trained YOLOv8 target detection model; the trained YOLOv8 target detection model is then used to identify the high-quality fused image to perform feature extraction, attention enhancement, feature fusion, prediction, and post-processing optimization to obtain the insulator defect detection result;

[0039] The result output module is used to determine the operating status of the insulator and generate a test report based on the defect type, confidence level and bounding box information in the insulator defect detection results.

[0040] Compared with the prior art, this application has the following advantages and beneficial effects:

[0041] 1. Excellent fusion effect: The improved RFN-Nest model enhances the fusion of infrared image temperature anomaly features and visible image structural detail features by embedding the EMA attention mechanism. The information entropy, mutual information and other indicators of the generated fused image are significantly better than the original model and traditional fusion methods, providing high-quality data support for detection tasks.

[0042] 2. Outstanding detection performance: The improved YOLOv8 model enhances the ability to identify defect features and resist interference by integrating the CBAM attention mechanism. The mAP@0.5 reaches 93.2%, the recall rate of small defects reaches 89.7%, and the inference speed of 71 FPS is maintained. It balances accuracy and real-time performance to meet the real-time inspection needs of UAVs.

[0043] 3. High efficiency through collaborative design: An integrated solution of "image fusion-target detection" is constructed to achieve accurate matching between feature fusion and detection requirements, effectively solving the problems of missed and false alarms of insulator defects in complex scenarios, and significantly improving performance compared with traditional independent design solutions;

[0044] 4. High practicality: The dataset covers multiple types of insulators, multiple defect modes and multiple environmental scenarios. The model can still work stably in complex environments such as fog and low light. The detection accuracy of fine cracks (width < 5 pixels) and abnormal heating (temperature difference > 5℃) is over 85%, which can be directly applied to actual power inspection scenarios. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0046] Figure 1 A flowchart illustrating a defect detection method based on multimodal image fusion of insulators, provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram of the RFN-Nest image fusion model based on an improved attention mechanism provided in this application embodiment;

[0048] Figure 3 A schematic diagram of an improved YOLOv8 object detection model based on an attention mechanism, provided in an embodiment of this application;

[0049] Figure 4 A schematic diagram of a defect detection system based on multimodal image fusion of insulators provided in an embodiment of this application;

[0050] The attached diagram shows the markings and corresponding component names:

[0051] 401 - Data acquisition and preprocessing module, 402 - Image fusion module, 403 - Target detection module, 404 - Result output module. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0053] like Figure 1As shown in the figure, this application provides a defect detection method based on insulator multimodal image fusion, including:

[0054] S101. Acquire infrared images and visual images corresponding to the insulator, and preprocess the infrared images and visual images to obtain standardized image data.

[0055] In one possible implementation, the infrared image and the visual image are preprocessed to obtain standardized image data, including:

[0056] The infrared image and the visual image are sequentially denoised to obtain a denoised infrared image and a denoised visual image; the denoising process uses a bilateral filtering algorithm.

[0057] The denoised visual image is then converted to grayscale to obtain a grayscale visual image; the grayscale conversion process uses a weighted average method to convert the color visual image into a grayscale image.

[0058] The denoised infrared image and the grayscale visual image are used as standardized image data.

[0059] For example, a drone equipped with a high-definition visible light camera and an infrared thermal imager can be used to acquire multi-angle, all-around images of insulators on power transmission lines. During the acquisition process, the camera's shooting angle and distance should be kept relatively stable to ensure image consistency and comparability. The acquired raw images are preprocessed: a bilateral filtering algorithm is used for noise reduction, removing Gaussian noise, salt-and-pepper noise, and other interferences while preserving edge and detail information; a weighted average method is used to convert the color visible image into a grayscale image, simplifying the data volume and reducing computational complexity, so that the grayscale image can accurately reflect the brightness information of the original image.

[0060] S102. Construct an improved RFN-Nest image fusion model based on an attention mechanism, and train the RFN-Nest image fusion model using a two-stage training strategy to obtain the trained RFN-Nest image fusion model.

[0061] In one possible implementation, an RFN-Nest (end-to-end residual fusion network for infrared and visible light images) image fusion model based on an improved attention mechanism is constructed, and the RFN-Nest image fusion model is trained using a two-stage training strategy to obtain the trained RFN-Nest image fusion model, including:

[0062] An EMA attention mechanism module is embedded between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an improved RFN-Nest image fusion model based on the attention mechanism.

[0063] The RFN-Nest image fusion model is trained using a two-stage training strategy to obtain the trained RFN-Nest image fusion model.

[0064] In one possible implementation, an EMA (Efficient Multi-scale Attention) attention mechanism module is embedded between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an RFN-Nest image fusion model improved based on the attention mechanism, including:

[0065] Construct an encoder network, an EMA attention mechanism module, an RFN network, and a decoder network; wherein the encoder network is configured as a convolutional neural network;

[0066] The encoder network and the decoder network are connected sequentially through the EMA attention mechanism module and the RFN network to form an RFN-Nest image fusion model based on an improved attention mechanism.

[0067] like Figure 2 As shown, the improved RFN-Nest image fusion model's network architecture includes an encoder network, an EMA attention mechanism module, a residual fusion network (RFN), and a decoder network. The encoder network employs a convolutional neural network (CNN) structure, progressively reducing image resolution and increasing the number of feature map channels through multiple convolutional and pooling layers to extract multi-scale deep features of the image. The EMA attention mechanism module is embedded between the encoder and RFN, with a decay coefficient α=0.99. It adaptively generates an attention weight matrix by calculating the global exponential moving average statistics of the feature maps, weighting and enhancing the multi-scale features output by the encoder. The core formula is... ,in, This represents the attention weight of the EMA. The attention weight from the previous moment. As the global mean feature of the current feature map, this module can dynamically focus on key feature areas such as high-temperature points in infrared images and crack edges in visual images, while weakening interference from irrelevant background information.

[0068] The Residual Fusion Network (RFN) consists of six convolutional layers. Conv1 and Conv2 receive infrared image features and visual image features enhanced by EMA, respectively. They are then concatenated and input into Conv3 and Conv6, which directly fuse the two features to fully exploit the complementary information of the two image features. The decoder network consists of multiple DCBs (Decoder Convolutional Blocks). Through short connections and cross-layer connections, multi-scale deep features are connected together. The image resolution is gradually restored through upsampling and convolution operations to reconstruct the fused image.

[0069] In one possible implementation, training the RFN-Nest image fusion model using a two-stage training strategy to obtain the trained RFN-Nest image fusion model includes:

[0070] The parameters of the RFN network in the attention-based improved RFN-Nest image fusion model are frozen. The autoencoder network, composed of the encoder, EMA attention mechanism module, and decoder in the attention-based improved RFN-Nest image fusion model, is trained with the goal of minimizing the first loss function, resulting in the RFN-Nest image fusion model after one stage of training. The first loss function is: ; For pixel loss, The loss function is structural similarity loss, where λ is the weighting coefficient. By optimizing this loss function, the encoder gains strong feature extraction capabilities, the EMA attention mechanism accurately captures key features, and the decoder gains good feature reconstruction capabilities.

[0071] The parameters of the autoencoder network in the RFN-Nest image fusion model after the first stage of training are frozen, and the RFN network is trained with the goal of minimizing the second loss function to obtain the RFN-Nest image fusion model after the second stage of training, thus completing the training; the second loss function is: ; Loss of background detail Enhance the loss for the target features. Optimize the feature fusion effect by training an RFN.

[0072] The trained model generates fused images with information entropy ≥7.85 and mutual information ≥3.56, which are improvements of ≥4.4% and ≥10.9% respectively compared to the original RFN-Nest model, providing high-quality data support for subsequent detection tasks.

[0073] S103. The trained RFN-Nest image fusion model is invoked to perform fusion processing on the standardized image data to obtain a high-quality fused image.

[0074] S104. Construct an improved YOLOv8 target detection model based on the attention mechanism, and train the YOLOv8 target detection model with a pre-stored insulator defect dataset to obtain the trained YOLOv8 target detection model.

[0075] In one possible implementation, a YOLOv8 object detection model based on an improved attention mechanism is constructed, including: integrating the CBAM attention mechanism after the C2f module of the backbone network in the original YOLOv8 object detection model to obtain a YOLOv8 object detection model based on an improved attention mechanism; the CBAM (Convolutional Block Attention Module, a lightweight attention mechanism module) attention mechanism includes a channel attention submodule and a spatial attention submodule.

[0076] In one possible implementation, the YOLOv8 target detection model is trained using a pre-stored insulator defect dataset to obtain the following:

[0077] The experimental environment was built based on the PyTorch deep learning framework, with an initial learning rate of 0.001. Using a pre-stored insulator defect dataset as a foundation, the YOLOv8 target detection model was trained using the SGD optimizer. During training, the momentum coefficient was set to 0.9, the weight decay coefficient to 5e-4, the batch size to 16, and the total number of training iterations to 100 epochs. The first 80 epochs were the normal training phase, and the learning rate was linearly decayed to 1e-6 in the last 20 epochs. The average accuracy and loss values ​​corresponding to the pre-stored validation set were used as monitoring metrics during training. Model weights were saved every 5 epochs, triggering an early stopping mechanism to prevent invalid training.

[0078] like Figure 3 As shown, the improved YOLOv8 object detection model's network architecture includes a Backbone network, an FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) feature fusion layer, and a prediction head. The core improvement is embedding a CBAM attention mechanism at the output of each C2f module in the Backbone.

[0079] The CBAM attention mechanism consists of a channel attention submodule (CAM) and a spatial attention submodule (SAM), which sequentially apply attention weights to the channel and spatial dimensions of the feature map. The channel attention submodule calculates the importance weight of each channel through a squeeze-excitation operation, using the following formula: Where F is the input feature map, Avgpool() is global average pooling, MLP is multilayer perceptron (fully connected network), and MaxPool() is global max pooling.

[0080] The spatial attention submodule generates spatial attention weights by aggregating channel information, using the following formula: The overall output is It can enhance the channel features and spatial location features related to defects and suppress background interference; where ⊗ represents element-wise multiplication, σ is the Sigmoid function, Mc() is the channel attention map, Ms() is the spatial attention map, and [;] is splicing.

[0081] The FPN+PAN feature fusion layer achieves cross-scale fusion of high-level semantic features and low-level detail features through the top-down FPN path and the bottom-up PAN path, taking into account the detection needs of defects of different sizes.

[0082] The prediction head employs a decoupled design, separating the classification and regression tasks, and outputting the target class probability, bounding box coordinates, and confidence score respectively.

[0083] The model training process is based on the PyTorch deep learning framework, with an NVIDIA RTX 3090 GPU as the hardware configuration. YOLOv8 pre-trained weights are loaded, and transfer learning is used to reduce training difficulty and accelerate convergence. The hyperparameters are set as follows: initial learning rate 0.001, SGD optimizer (momentum coefficient 0.9, weight decay coefficient 5e-4), batch size 16, and a total of 100 training iterations (normal training for the first 80 epochs, and linear decay of the learning rate to 1e-6 for the last 20 epochs). During training, the mAP@0.5 of the validation set and the loss value are used as monitoring indicators. The model weights are saved every 5 epochs, and an early stopping mechanism is triggered when there is no significant improvement in the mAP of the validation set for 10 consecutive epochs.

[0084] The insulator defect dataset covers various insulator types (suspension, post, pin, etc.), defect types (cracks, damage, dirt, abnormal heating, etc.), and environmental conditions (sunny, cloudy, rainy, foggy, etc.). It is divided into training, validation, and test sets in an 8:1:1 ratio. The training set is augmented with random rotation, scaling, flipping, and cropping to improve the model's generalization ability.

[0085] S105. The trained YOLOv8 target detection model is invoked to identify the high-quality fused image, so as to perform feature extraction, attention enhancement, feature fusion, prediction and post-processing optimization to obtain the insulator defect detection result.

[0086] The specific reasoning process is as follows:

[0087] 1. Input preprocessing: The fused image is adjusted to a size of 640×640, and a pixel normalization strategy is used to map the pixel values ​​to [0,1] to maintain the aspect ratio and avoid feature distortion. Batch data is loaded and processed in parallel through PyTorch's DataLoader.

[0088] 2. Feature extraction and enhancement: Multi-scale features are extracted from the fused image through the backbone network. After each C2f module, the channel and spatial weights are dynamically generated by the CBAM attention mechanism to enhance the response of defect features such as cracks and high-temperature points.

[0089] 3. Feature Fusion and Prediction: The attention-enhanced feature map is input into the FPN+PAN structure to complete cross-scale fusion. The prediction head outputs the category probability, bounding box coordinates and confidence of 5 types of targets (normal insulator, crack, damage, pollution, and abnormal heating).

[0090] 4. Post-processing optimization: The NMS algorithm (non-maximum suppression algorithm) is used to remove redundant prediction boxes, with the confidence threshold set to 0.5 and the IoU threshold set to 0.3. For small defects with a width of less than 5 pixels, the IoU threshold of NMS is reduced to 0.25 to reduce the risk of missed detection. The final output includes the detection results containing the target category, confidence score, and bounding box coordinates.

[0091] The improved YOLOv8 target detection model has an mAP@0.5≥93.2%, a small defect recall rate≥89.7%, and an inference speed≥71FPS, meeting the real-time inspection requirements of UAVs.

[0092] S106. Based on the defect type, confidence level, and bounding box information in the insulator defect detection results, determine the insulator operating status and generate a detection report.

[0093] In one possible implementation, based on the defect type, confidence level, and bounding box information in the insulator defect detection results, the insulator operating status is determined and a detection report is generated, including:

[0094] Based on the defect type, confidence level, and bounding box information in the insulator defect detection results, the state of the target insulator is determined, and the target suggestion corresponding to the pre-stored target insulator state is obtained.

[0095] Based on the defect type, confidence level, bounding box information, and target suggestions, a detection report is generated that includes a defect location annotation map, defect type statistics, confidence level distribution, and processing suggestions.

[0096] For example, the specific determination rule is as follows:

[0097] 1. If the test results only include normal insulators and the confidence level is ≥0.8, it is determined to be in normal operation, and the test report recommends continuous monitoring;

[0098] 2. If the confidence level of the detected dirt category or other defect category is between 0.5 and 0.8, it is judged to be in a sub-healthy state. The test report recommends increasing the frequency of inspections and cleaning up dirt in a timely manner.

[0099] 3. If cracks, damage, or abnormal heating are detected with a confidence level ≥ 0.8, it is considered an abnormal condition. The inspection report recommends immediate shutdown and maintenance to prevent accidents.

[0100] The inspection report supports visualization (images with labeled defect locations, categories, and confidence levels) and export, making it easy for maintenance personnel to view and process.

[0101] Based on the technical solutions described in the above embodiments, this application has the following advantages and beneficial effects compared with the prior art:

[0102] 1. Excellent fusion effect: The improved RFN-Nest model enhances the fusion of infrared image temperature anomaly features and visible image structural detail features by embedding the EMA attention mechanism. The information entropy, mutual information and other indicators of the generated fused image are significantly better than the original model and traditional fusion methods, providing high-quality data support for detection tasks.

[0103] 2. Outstanding detection performance: The improved YOLOv8 model enhances the ability to identify defect features and resist interference by integrating the CBAM attention mechanism. The mAP@0.5 reaches 93.2%, the recall rate of small defects reaches 89.7%, and the inference speed of 71 FPS is maintained. It balances accuracy and real-time performance to meet the real-time inspection needs of UAVs.

[0104] 3. High efficiency through collaborative design: An integrated solution of "image fusion-target detection" is constructed to achieve accurate matching between feature fusion and detection requirements, effectively solving the problems of missed and false alarms of insulator defects in complex scenarios, and significantly improving performance compared with traditional independent design solutions;

[0105] 4. High practicality: The dataset covers multiple types of insulators, multiple defect modes and multiple environmental scenarios. The model can still work stably in complex environments such as fog and low light. The detection accuracy of fine cracks (width < 5 pixels) and abnormal heating (temperature difference > 5℃) is over 85%, which can be directly applied to actual power inspection scenarios.

[0106] To enable those skilled in the art to more easily understand the technical solutions described in the embodiments of this application, examples are provided based on the technical solutions described in the above embodiments.

[0107] A defect detection method based on insulator multimodal image fusion may include:

[0108] S1. Data Acquisition and Preprocessing;

[0109] A DJI M300 RTK drone, equipped with a Zenmuse P1 high-definition visible light camera and a Zenmuse XT2 infrared thermal imager, was used to inspect and photograph suspension insulator strings on a 500kV transmission line in Sichuan Province. The shooting height was 10-15 meters above the insulators, and the shooting angles covered the front, side, and obliquely above. The acquired images underwent preprocessing: a bilateral filtering algorithm (spatial standard deviation set to 5, grayscale standard deviation set to 0.5) was used to remove noise; a weighted average method (R channel weight 0.299, G channel weight 0.587, B channel weight 0.114) was used to convert the color visible images into grayscale images.

[0110] S2. Construction and training of the improved RFN-Nest image fusion model;

[0111] An RFN-Nest+EMA fusion model was constructed. The encoder network was set with 8 convolutional layers (kernel size 5×5) and 3 pooling layers. The decay coefficient α of the EMA attention mechanism module was 0.99. The dimension of the attention weight matrix was consistent with the number of output channels (256) of the last layer of the encoder. The number of Conv1-Con6 convolutional kernels of the residual fusion network (RFN) were 64, 64, 128, 128, 256 and 256, respectively. The decoder network was set with 6 DCB modules and upsampling was performed using deconvolution operation.

[0112] A two-stage training strategy was adopted: the first stage trained the autoencoder network with a loss function of λ=0.8, 50 epochs, and a batch size of 8; the second stage trained the RFN network with a loss function of λ=0.5, 30 epochs, and a batch size of 8. After training, the generated fused image had an information entropy of 7.85, a standard deviation of 12.65, and a mutual information of 3.56, representing improvements of 4.4%, 6.9%, and 10.9% respectively compared to the original RFN-Nest model.

[0113] S3. Construction and training of an improved YOLOv8 object detection model;

[0114] A YOLOv8+CBAM detection model was constructed, integrating the CBAM attention mechanism module after each of the four C2f modules in the YOLOv8 backbone network; the FPN+PAN feature fusion layer maintained its original structure, and the prediction head output the detection results for five types of targets (normal insulators, cracks, damage, pollution, and abnormal heating).

[0115] The insulator defect dataset is generated from the image annotations collected in step S1. The LabelImg annotation tool is used to accurately draw the defect bounding boxes and label the categories. The dataset is divided into training set, validation set and test set in an 8:1:1 ratio. The training set is subjected to data augmentation operations such as random rotation (0-360°), scaling (0.8-1.2 times), horizontal flipping (probability 0.5), and cropping (randomly cropped to 640×640).

[0116] Model training was conducted using the PyTorch 1.18 framework and an NVIDIA RTX 3090 GPU (24GB VRAM). YOLOv8 pre-trained weights were loaded with an initial learning rate of 0.001, an SGD (Stochastic Gradient Descent) optimizer (momentum 0.9, weight decay of 5e-4), a batch size of 16, and a total of 100 training iterations (the learning rate was kept at 0.001 for the first 80 epochs and then linearly decayed to 1e-6 for the last 20 epochs). Model weights were saved every 5 epochs. The validation set mAP@0.5 reached its highest value (93.2%) at 68 epochs (training rounds), with no significant improvement thereafter, triggering the early stopping mechanism to stop training.

[0117] S4. Integrated reasoning and defect detection;

[0118] The fused image generated in step S2 is resized to 640×640, and the pixels are normalized to [0,1] before being input into the trained YOLOv8+CBAM model. The batch size is set to 16. Multi-scale features are extracted from the fused image using Backbone, the CBAM module enhances the defect feature response, and FPN+PAN fuses cross-scale features. The prediction head outputs the class probability, bounding box coordinates, and confidence score. The NMS algorithm is used for post-processing, with a confidence threshold of 0.5, an IoU threshold of 0.3 for ordinary defects, and an IoU threshold of 0.25 for small defects (width < 5 pixels).

[0119] Test set validation results show that the model has an accuracy of 93.5%, a recall of 92.1%, an mAP@0.5 of 93.2%, and an inference speed of 71 FPS. Among them, the recall rate for fine cracks (width 2-5 pixels) is 89.7%, the detection accuracy for abnormal heating (temperature difference > 5℃) is 94.3%, and the accuracy in complex environments such as fog and low light reaches 93.5%.

[0120] S5. Status determination and report generation;

[0121] Based on the test set results, the insulator's operating status was determined: 620 test images were detected as normal (normal insulator category only, confidence level ≥ 0.8), and continuous monitoring is recommended; 105 images were detected as sub-healthy (pollution category or defect confidence level 0.5-0.8), and increased inspection and cleaning of contaminants are recommended; 75 images were detected as abnormal (cracks, damage, abnormal overheating category, confidence level ≥ 0.8), and immediate shutdown and maintenance are recommended. The results output module generates a test report including defect location annotations, defect type statistics, confidence level distribution, and handling suggestions, which can be exported in PDF format.

[0122] like Figure 4 Based on the same inventive concept, another embodiment of this application also provides a defect detection system based on insulator multimodal image fusion, comprising:

[0123] The data acquisition and preprocessing module 401 is used to acquire infrared images and visual images corresponding to the insulator, and to preprocess the infrared images and visual images to obtain standardized image data.

[0124] Image fusion module 402 is used to construct an improved RFN-Nest image fusion model based on an attention mechanism, and to train the RFN-Nest image fusion model through a two-stage training strategy to obtain the trained RFN-Nest image fusion model; the trained RFN-Nest image fusion model is then used to perform fusion processing on the standardized image data to obtain a high-quality fused image.

[0125] The target detection module 403 is used to construct a YOLOv8 target detection model based on an improved attention mechanism, and to train the YOLOv8 target detection model with a pre-stored insulator defect dataset to obtain a trained YOLOv8 target detection model; the trained YOLOv8 target detection model is then used to identify the high-quality fused image to perform feature extraction, attention enhancement, feature fusion, prediction, and post-processing optimization to obtain the insulator defect detection result;

[0126] The result output module 404 is used to determine the operating status of the insulator and generate a test report based on the defect type, confidence level and bounding box information in the insulator defect detection results.

[0127] The defect detection system based on insulator multimodal image fusion provided in this application embodiment can perform the above-described method and technical solution. Its principle and beneficial effects are similar, and will not be described again here.

[0128] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for defect detection based on insulator multi-modal image fusion, characterized in that, The method comprises the following steps: Collecting infrared images and visible images corresponding to insulators, and preprocessing the infrared images and visible images to obtain standardized image data; Constructing an RFN-Nest image fusion model improved based on an attention mechanism, and training the RFN-Nest image fusion model through a two-stage training strategy to obtain a trained RFN-Nest image fusion model; Calling the trained RFN-Nest image fusion model to perform fusion processing on the standardized image data to obtain a high-quality fusion image; Constructing a YOLOv8 target detection model improved based on an attention mechanism, and training the YOLOv8 target detection model with a pre-stored insulator defect dataset to obtain a trained YOLOv8 target detection model; Calling the trained YOLOv8 target detection model to identify the high-quality fusion image to perform feature extraction, attention enhancement, feature fusion, prediction, and post-processing optimization, and obtaining an insulator defect detection result; According to the defect type, confidence, and bounding box information in the insulator defect detection result, determining the running state of the insulator and generating a detection report.

2. The method for defect detection based on insulator multi-modal image fusion according to claim 1, characterized in that, The preprocessing of the infrared images and visible images to obtain standardized image data comprises the following steps: Sequentially performing denoising processing on the infrared images and visible images to obtain denoised infrared images and denoised visible images; the denoising processing adopts a bilateral filtering algorithm; Performing grayscale processing on the denoised visible images to obtain grayscale visible images; the grayscale processing converts the color visible images into grayscale images by using a weighted average method; Taking the denoised infrared images and grayscale visible images as standardized image data.

3. The method for defect detection based on insulator multi-modal image fusion according to claim 1, characterized in that, The construction of the RFN-Nest image fusion model improved based on an attention mechanism and the training of the RFN-Nest image fusion model through a two-stage training strategy to obtain a trained RFN-Nest image fusion model comprises the following steps: Embedding an EMA attention mechanism module between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an RFN-Nest image fusion model improved based on an attention mechanism; Training the RFN-Nest image fusion model through a two-stage training strategy to obtain a trained RFN-Nest image fusion model.

4. The method for defect detection based on insulator multi-modal image fusion according to claim 3, characterized in that, The embedding of the EMA attention mechanism module between the encoder and the residual fusion network of the original RFN-Nest image fusion model to obtain an RFN-Nest image fusion model improved based on an attention mechanism comprises the following steps: Constructing an encoder network, an EMA attention mechanism module, an RFN network, and a decoder network; wherein the encoder network is set as a convolutional neural network; Sequentially connecting the encoder network and the decoder network through the EMA attention mechanism module and the RFN network to form an RFN-Nest image fusion model improved based on an attention mechanism.

5. The method for defect detection based on insulator multi-modal image fusion according to claim 1 or 3, characterized in that, The RFN-Nest image fusion model is trained through the two-stage training strategy to obtain a trained RFN-Nest image fusion model, including: The parameters corresponding to the RFN network in the RFN-Nest image fusion model improved based on the attention mechanism are frozen, and the self-encoder network composed of the encoder, the EMA attention mechanism module and the decoder in the RFN-Nest image fusion model improved based on the attention mechanism is trained with the first loss function minimization as the target to obtain the RFN-Nest image fusion model after one-stage training; the first loss function is: ; is a pixel loss, is a structural similarity loss, and λ is a weight coefficient; The parameters corresponding to the auto-encoder network in the RFN-Nest image fusion model after the first-stage training are frozen, and the RFN network is trained with the second loss function minimization as the goal, to obtain the RFN-Nest image fusion model after the second-stage training, and the training is completed; the second loss function is: ; a loss for background details, a target feature enhancement loss.

6. The method for defect detection based on insulator multi-modal image fusion according to claim 1, characterized in that, The YOLOv8 target detection model based on the attention mechanism is improved, including: integrating the CBAM attention mechanism after the C2f module of the Backbone backbone network in the original YOLOv8 target detection model to obtain the YOLOv8 target detection model based on the attention mechanism; the CBAM attention mechanism includes a channel attention submodule and a spatial attention submodule.

7. The method for defect detection based on insulator multi-modal image fusion according to claim 6, characterized in that, The YOLOv8 target detection model is trained based on the pre-stored insulator defect dataset to obtain a YOLOv8 target detection model including: Based on the PyTorch deep learning framework, an experimental environment is built, the initial learning rate is set to 0.001, and the YOLOv8 target detection model is trained based on the pre-stored insulator defect dataset using the SGD optimizer to obtain the YOLOv8 target detection model; during the training process, the momentum coefficient is set to 0.9, the weight decay coefficient is set to 5e-4, the batch size is set to 16, the total number of training iterations is set to 100 Epochs, the first 80 Epochs are the normal training phase, and the last 20 Epochs linearly decay the learning rate to 1e-6; during the training process, the average precision mean and loss value corresponding to the pre-stored validation set are used as monitoring indicators, the model weight is saved once every 5 Epochs, and the early stopping mechanism is triggered to prevent invalid training.

8. The method for defect detection based on insulator multi-modal image fusion according to claim 1, characterized in that, The post-processing optimization includes: setting the confidence threshold to 0.5, the IoU threshold to 0.3, and using the non-maximum suppression algorithm to remove redundant prediction boxes, and for small target defects with a width < 5 pixels, the IoU threshold of the non-maximum suppression algorithm is reduced to 0.25; The insulator defect detection result includes target categories, confidence and bounding box coordinates, and the target categories include normal insulators, cracks, damage, contamination and heat anomalies.

9. The method for defect detection based on insulator multi-modal image fusion according to claim 1, characterized in that, According to the defect type, confidence and bounding box information in the insulator defect detection result, the running state of the insulator is determined and a detection report is generated, including: According to the defect type, confidence and bounding box information in the insulator defect detection result, the target insulator state is determined, and the target suggestion corresponding to the pre-stored target insulator state is obtained; According to the defect type, confidence, bounding box information and target suggestion, a detection report containing defect location labeling, defect type statistics, confidence distribution and processing suggestions is generated.

10. A defect detection system based on insulator multi-modal image fusion, characterized in that, including: A data acquisition and preprocessing module for acquiring infrared images and visible images corresponding to insulators and preprocessing the infrared images and visible images to obtain standardized image data; An image fusion module is configured to construct an RFN-Nest image fusion model improved based on an attention mechanism, train the RFN-Nest image fusion model through a two-stage training strategy, and obtain a trained RFN-Nest image fusion model; and the trained RFN-Nest image fusion model is called to perform fusion processing on the standardized image data, and a high-quality fusion image is obtained. A target detection module is configured to construct a YOLOv8 target detection model improved based on an attention mechanism, train the YOLOv8 target detection model with a pre-stored insulator defect dataset, and obtain a trained YOLOv8 target detection model; and the trained YOLOv8 target detection model is called to identify the high-quality fusion image to perform feature extraction, attention enhancement, feature fusion, prediction and post-processing optimization, and obtain an insulator defect detection result. A result output module is configured to determine an insulator operating state and generate a detection report according to defect types, confidence and bounding box information in the insulator defect detection result.

Images