Insulator defect detection method fusing infrared and visible light images and embedded recognition device

Abstract

The application discloses a kind of insulator defect detection method and embedded recognition device of fusion infrared and visible light image.The method comprises the following steps: constructing dual-light image dataset and spectral gradient perception cross-modal attention fusion network, generating fusion image by encoding module, cross-modal attention module, spectral gradient constraint module and decoding module;A multi-task detection model is constructed, which includes a defect classification head and a defect segmentation head, and simultaneously outputs a defect positioning frame and a segmentation mask;The feature extraction layer shared by the fusion network and the detection model constitutes a unified neural network, and the weighted sum of the minimum fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss is jointly trained;Output defect type, positioning frame coordinates and defect segmentation contour.The application preserves the temperature gradient characteristics of infrared image through the spectral gradient perception mechanism, and realizes accurate fusion of dual-light image by combining cross-modal attention, effectively improving the precision and robustness of insulator defect detection.

Description

Technical Field

[0001] This invention belongs to the field of power equipment testing technology, specifically relating to an insulator defect detection method and embedded identification device that integrates infrared and visible light images. It can be used for intelligent detection of defects such as dirt, cracks, skirt damage, and core rod dampness in transmission line insulators. Background Technology

[0002] Insulators are critical components widely used in power systems, and their performance directly affects the operational safety of transmission lines. Due to long-term exposure to the outdoor environment, insulator surfaces are prone to various defects such as dirt, cracks, damaged sheds, and moisture in the core rod. If these defects are not detected and addressed in a timely manner, they may lead to serious accidents such as flashover and breakdown, affecting the stable operation of the power grid.

[0003] Traditional insulator defect detection mainly relies on manual inspection, which suffers from low efficiency, high subjectivity, and high risk. In recent years, computer vision-based detection methods have been widely used, but single-modal visible light imaging is difficult to accurately identify defects under conditions such as insufficient lighting and complex backgrounds, especially in detecting early-stage micro-defects and internal defects. Infrared thermal imaging technology can reflect the temperature distribution characteristics of objects and has unique advantages in detecting thermal defects, but infrared images have low resolution and lack texture information, making it difficult to distinguish defects from the inherent structure of the insulator.

[0004] To address the aforementioned issues, researchers have proposed a detection method that fuses infrared and visible light images. Existing fusion methods are mainly categorized into multi-scale transform-based methods and deep learning-based methods. Multi-scale transform-based methods, such as pyramid transform and wavelet transform, can achieve preliminary fusion, but the fusion rules rely on manual design and are difficult to adapt to complex scenarios. Deep learning-based methods, such as DenseFuse, RFN-Nest, and SwinFusion, achieve end-to-end fusion through autoencoders or generative adversarial networks, improving fusion quality to some extent. However, these methods are prone to losing spectral gradient information from the infrared image during the fusion process, leading to weakened temperature characteristics of defect areas and affecting subsequent detection accuracy.

[0005] Furthermore, existing fusion and detection networks are typically designed separately, leading to a mismatch between the optimization goals of the fused image and the requirements of the detection task, making true end-to-end optimization difficult. Simultaneously, the differences in imaging principles between two-light images result in viewpoint deviations and spatial misalignments, directly impacting fusion performance and detection accuracy. Therefore, achieving accurate fusion of infrared and visible light images, effectively preserving spectral gradient features during the fusion process, and jointly optimizing the fusion and detection networks have become pressing technical challenges in this field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an insulator defect detection method and embedded recognition device that fuses infrared and visible light images, so as to solve the technical problems that existing fusion methods are prone to losing infrared image spectral gradient information and that the separate design of the fusion network and the detection network leads to inconsistent optimization objectives.

[0007] In a first aspect, embodiments of the present invention provide a method for detecting insulator defects by fusing infrared and visible light images, the method comprising: Step 1: Construct a dual-light image dataset for insulators, which includes registered visible light and infrared images; Step 2: Construct a spectral gradient-aware cross-modal attention fusion network. The network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module. It is used to extract features from the visible light image and the infrared image, perform cross-modal interaction, spectral gradient constraint and reconstruction, and generate a fused image. Step 3: Construct a multi-task detection model based on end-to-end joint optimization. The multi-task detection model includes a defect classification head and a defect segmentation head, which are used to simultaneously output defect localization boxes and defect segmentation masks after inputting the fused image. Step 4: End-to-end joint training and optimization. The spectral gradient-aware cross-modal attention fusion network and the multi-task detection model share the feature extraction layer to form a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss. Step 5: Output the detection results, which include the defect type, the coordinates of the location box, and the defect segmentation contour.

[0008] In a second aspect, embodiments of the present invention provide an embedded identification device for insulator defect detection that integrates infrared and visible light images, applied to the insulator defect detection method that integrates infrared and visible light images as described in the first aspect, the system comprising: A data construction unit is used to construct a dual-light image dataset of insulators, the dataset including registered visible light images and infrared images; The image fusion unit is equipped with a spectral gradient-aware cross-modal attention fusion network. The network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module, which are used to extract features, perform cross-modal interaction, constrain and reconstruct spectral gradients between the visible light image and the infrared image to generate a fused image. The multi-task detection unit is equipped with a multi-task detection model based on end-to-end joint optimization. The multi-task detection model includes a defect classification head and a defect segmentation head, which are used to input the fused image and simultaneously output the defect localization box and the defect segmentation mask. The joint training and optimization unit is used to perform end-to-end joint training and optimization of the spectral gradient sensing cross-modal attention fusion network and the multi-task detection model. The spectral gradient sensing cross-modal attention fusion network and the multi-task detection model share a feature extraction layer, forming a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss. The detection result output unit is used to output the detection result, which includes the defect type, the coordinates of the positioning box, and the defect segmentation contour.

[0009] Thirdly, embodiments of the present invention provide an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the insulator defect detection method by fusing infrared and visible light images as described in the first aspect when executing the instructions.

[0010] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a program that instructs a device to perform the insulator defect detection method by fusing infrared and visible light images as described in the first aspect.

[0011] Compared with the prior art, the present invention has the following beneficial effects: First, the spectral gradient sensing cross-modal attention fusion network of the present invention extracts multi-scale structural features, texture detail features and spectral supplementary features of images through the encoding module, realizes spatial alignment and bidirectional interaction of visible light and infrared features through the cross-modal attention module, and retains the temperature gradient information of infrared images through the spectral gradient constraint module, effectively solving the problem that existing fusion methods are prone to losing spectral gradient information.

[0012] Second, the fusion network of this invention shares a feature extraction layer with the multi-task detection model and is jointly trained to form a unified end-to-end neural network structure, which keeps the optimization goal of the fused image consistent with the requirements of the detection task and significantly improves the defect detection accuracy.

[0013] Third, the multi-task detection model of the present invention includes both a defect classification head and a defect segmentation head, and can output defect location boxes and defect segmentation masks in parallel, realizing integrated detection of defect location, classification and segmentation.

[0014] Fourth, the weighted loss function of this invention introduces spectral gradient constraint loss, which constrains the gradient direction of the fused image in the high-temperature region to be consistent with that of the infrared image through the similarity loss function, thereby further enhancing the ability of the fused image to retain spectral features.

[0015] Fifth, the embedded identification device of the present invention integrates the network and the detection model in the same processing unit, has real-time detection capability, and is suitable for outdoor power inspection scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of an insulator defect detection method that fuses infrared and visible light images, provided in an embodiment of the present invention.

[0017] Figure 2 This is an architectural diagram of an embedded identification device for insulator defect detection that fuses infrared and visible light images, provided in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0020] It should be noted that in the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0021] Based on the embodiments described in this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0022] Example 1

[0023] Figure 1 This is a schematic flowchart illustrating an insulator defect detection method that fuses infrared and visible light images, according to an embodiment of the present invention. Figure 1 As shown, an insulator defect detection method that integrates infrared and visible light images includes: Step 1: Construct a dual-mode image dataset for insulators, comprising registered visible light and infrared images. This step involves acquiring and registering visible light and infrared images of insulators to construct a dataset containing both modal images, providing fundamental data support for subsequent fusion and detection. In this embodiment, a UAV equipped with a visible light camera and an infrared thermal imager is used to acquire images of transmission line insulators. The acquired visible light images have a resolution of 8000×6000 pixels, and the infrared images have a resolution of 640×512 pixels. Subpixel-level registration of the two images is performed using SIFT feature point matching and progressive consistent sampling algorithms. The registered image pairs are then cropped to remove invalid edge regions, ultimately constructing a dual-mode image dataset for insulators containing 847 pairs of registered visible light and infrared images.

[0024] Specifically, in this embodiment, the step of constructing the insulator dual-light image dataset further includes an image registration sub-step: SIFT feature points are extracted from both visible light and infrared images, and coarse matching is performed using the nearest neighbor distance ratio. In this embodiment, SIFT feature points are first extracted from both visible light and infrared images. SIFT feature points are local feature points that are invariant to image scaling and rotation, effectively identifying key locations in the image. Subsequently, coarse matching is performed on the extracted feature points using the nearest neighbor distance ratio. The distance ratio between each feature point and its nearest and second nearest neighbors is calculated. When this ratio is less than a preset threshold, the two feature points are considered to form a matching pair, thus initially identifying corresponding feature point pairs between the visible light and infrared images.

[0025] A progressively consistent sampling algorithm is used to eliminate mismatched point pairs and retain high-confidence in-place matching pairs. In this embodiment, the progressively consistent sampling algorithm is used to further filter the feature point pairs obtained after coarse matching, eliminating mismatched point pairs. This algorithm sorts the feature points according to their matching quality, prioritizes sampling from high-quality matching points, and iteratively fits a geometric transformation model through multiple iterations, retaining matching points consistent with the model as high-confidence in-place matching pairs, effectively removing erroneous matches caused by changes in viewpoint or noise interference.

[0026] The homography transformation matrix is ​​calculated based on interior point matching pairs. In this embodiment, the homography transformation matrix is ​​calculated based on the retained high-confidence interior point matching pairs. The homography transformation matrix describes the projection transformation relationship between a visible light image and an infrared image, mapping pixels in the visible light image to their corresponding positions in the infrared image. This matrix is ​​solved using a direct linear transformation algorithm, achieving an accurate description of the geometric transformation relationship between the two images.

[0027] A perspective transformation is performed on the visible light image to precisely align it with the infrared image at the pixel level. In this embodiment, based on the calculated homography transformation matrix, a perspective transformation is performed on the visible light image, projecting it onto the coordinate system of the infrared image. The perspective transformation can correct image distortion caused by differences in shooting angle, ensuring that the insulator outline in the visible light image is precisely aligned with the corresponding area in the infrared image at the pixel level, achieving sub-pixel level registration accuracy.

[0028] The registered image pairs are cropped to remove invalid regions caused by the transformation at the edges. In this embodiment, the registered image pairs are cropped. Since perspective transformation can create invalid blank or deformed areas at the image edges, cropping removes these invalid parts, preserving the effective overlap area between the visible light and infrared images, thus ensuring the data quality for subsequent image fusion and defect detection.

[0029] Step 2: Construct a spectral gradient-aware cross-modal attention fusion network. This network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module. It is used to extract features from the visible light image and the infrared image, perform cross-modal interaction, constrain spectral gradients, and reconstruct the image to generate a fused image. In this embodiment, the constructed spectral gradient-aware cross-modal attention fusion network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module. The encoding module uses three parallel encoders to extract the structural features, texture details, and spectral supplementary features of the image, respectively. The cross-modal attention module achieves feature space alignment through a 4×4 local self-attention network and completes the interaction between visible light and infrared features through a bidirectional cross-attention mechanism. The spectral gradient constraint module applies constraints to the fusion process based on the Sobel gradient map of the infrared image. The decoding module reconstructs the fused image from multi-scale features using a feature pyramid network.

[0030] Specifically, in this embodiment, the encoding module of the spectral gradient-aware cross-modal attention fusion network includes: First Module: First Encoder (Multi-Scale Dilated Convolutional Structure). In this embodiment, the first encoder employs a multi-scale dilated convolutional structure to extract global contextual information from the image. Dilated convolution is a convolutional method that expands the receptive field by inserting holes into standard convolutional kernels, enabling the capture of a wider range of image information without increasing the number of parameters. The first encoder uses multiple dilated convolutional kernels with progressively increasing dilation rates to process the input image in parallel. Convolutional kernels with smaller dilation rates focus on local regions of the image, while convolutional kernels with larger dilation rates focus on broader regions of the image, thereby generating structural feature maps containing receptive fields of different scales, effectively extracting the overall contour and morphological features of the insulator.

[0031] The second module: the second encoder (edge-aware convolutional kernel). In this embodiment, the second encoder uses an edge-aware convolutional kernel based on gradient operators to extract texture detail features from the image. A gradient operator is an operator used to calculate the rate of change of image brightness. By calculating the gradient response of the image in the horizontal and vertical directions, it can effectively identify edges and textured regions in the image. The second encoder utilizes this edge-aware capability to enhance detailed information such as cracks and dirt boundaries on the insulator surface, generating a feature map containing fine texture details, providing rich texture information for subsequent defect detection.

[0032] The third module: the third encoder (channel attention mechanism). In this embodiment, the third encoder superimposes the output features of the first and second encoders along the channel dimension through residual connections, and introduces a channel attention mechanism to weight and enhance regions with spectral differences. The channel attention mechanism is a mechanism that allows the network to autonomously focus on important feature channels. First, global pooling is used to compress each feature channel into a single value, capturing the global information of that channel; then, after processing by a fully connected layer, weight coefficients corresponding to each channel are generated; finally, these weight coefficients are multiplied by the original feature map to weight regions with significant spectral differences (such as heat spots in infrared images), generating a spectral supplementary feature map that effectively preserves the temperature gradient information in the infrared image.

[0033] Module 4: Feature Aggregation Strategy. In this embodiment, a layer-by-layer progressive feature aggregation strategy is adopted to fuse the output features of the first, second, and third encoders. This aggregation strategy uses a hierarchical residual connection method. First, the number of channels in the shallow features is adjusted through convolution to match the channel dimension of the deep features. Then, the adjusted shallow features are added element-wise to the deep features to achieve progressive feature fusion. Through this layer-by-layer progressive approach, the overall structural features, texture detail features, and spectral supplementary features extracted by different encoders are organically combined to form a hierarchical feature representation containing rich information, providing high-quality input features for subsequent cross-modal interaction and image reconstruction.

[0034] Specifically, in this embodiment, the cross-modal attention module includes: The first module consists of two parallel feature processing branches: a visible light feature branch and an infrared feature branch. In this embodiment, the cross-modal attention module includes two parallel feature processing branches, processing visible light image features and infrared image features respectively. Each branch uses a multi-layer Transformer encoder structure to perform deep encoding of the features of its respective modality. The Transformer encoder is a neural network structure based on a self-attention mechanism. Each encoder layer contains two core sub-layers: a multi-head self-attention sub-layer and a feedforward neural sub-layer. The multi-head self-attention sub-layer uses multiple parallel attention computation heads to allow each feature location to pay attention to other relevant locations in the same feature map, capturing the global dependencies within the feature. The feedforward neural sub-layer performs a non-linear transformation on the attention output to enhance the expressive power of the features. After processing by each sub-layer, layer normalization is used to standardize the features, and the input features are added to the output features through residual connections to ensure the stability of network training and the integrity of feature propagation.

[0035] The second module: Hierarchical alignment module. In this embodiment, the cross-modal interaction layer first aligns the visible light features and infrared features in spatial dimension through the hierarchical alignment module. This module uses a local self-attention network to achieve spatial calibration of the features. A local self-attention network is a mechanism that calculates self-attention within a limited window. It uses a sliding window approach to divide the feature map into multiple windows of fixed size, and calculates self-attention independently within each window, significantly reducing computational complexity. At the same time, this network introduces a learnable positional bias term. The positional offset learned during training compensates for the spatial misalignment caused by the difference in shooting angles of the dual-light camera, enabling the visible light features and infrared features to achieve accurate spatial correspondence at the pixel level.

[0036] The third module: Bidirectional cross-attention mechanism. In this embodiment, the cross-modal interaction layer further achieves deep fusion of visible light features and infrared features through a bidirectional cross-attention mechanism. This mechanism includes two cross-attention sub-layers: the first cross-attention sub-layer uses visible light features as the query term and infrared features as the key and value. Through multi-head cross-attention calculation, the visible light features absorb complementary information from the infrared features, and after layer normalization and feedforward processing, the enhanced visible light features are obtained. The second cross-attention sub-layer uses infrared features as the query term and visible light features as the key and value, allowing the infrared features to absorb texture detail information from the visible light features. Similarly, after layer normalization and feedforward processing, the enhanced infrared features are obtained. Through this bidirectional interaction, the features of the two modalities achieve full exchange and complementary enhancement of information.

[0037] Module 4: Feature Weighted Fusion. In this embodiment, the output of the cross-modal interaction layer, after further processing by a feedforward neural network and activation function, enters the feature weighted fusion stage. This stage uses learnable weight parameters to adaptively weight and sum the enhanced visible light and infrared features. The weight parameters are automatically learned and optimized during network training, dynamically adjusting the contribution of the two modal features based on the content of the input image. Through this weighted fusion strategy, cross-modal fusion features that contain both rich texture information from the visible light image and retain the spectral gradient features of the infrared image are generated, providing high-quality input for subsequent image reconstruction and defect detection.

[0038] Specifically, in this embodiment, the decoding module adopts a hierarchical feature pyramid network structure, including: Module 1: Top-Down Upsampling Path. In this embodiment, the decoding module first constructs a top-down upsampling path to restore the resolution of the high-level semantic features output by the encoding module. High-level semantic features contain global information and semantic content of the image, but have low spatial resolution. The upsampling operation progressively enlarges the feature map size through interpolation. After each upsampling operation, a skip connection is established between the current level's feature map and the corresponding level's feature map in the encoding module. Skip connections combine shallow detail features with deep semantic features. First, the number of channels in the encoded features is adjusted to match the number of channels in the decoded features through convolution. Then, the two are added element-wise, allowing the reconstruction process to utilize high-level semantic information to guide overall structure restoration while also improving image clarity with the help of shallow detail information.

[0039] The second module: Bottom-up feature aggregation path. In this embodiment, the decoding module further constructs a bottom-up feature aggregation path to refine and optimize the feature maps after skip connections. This path inputs the feature maps of each level to the convolutional block for processing through lateral connections. The convolutional block adopts a depthwise separable convolutional structure. This convolutional method decomposes standard convolution into two independent steps: first, depthwise convolution is performed, and spatial convolution is performed independently on each input channel to capture the feature relationships in the spatial dimension; then, pointwise convolution is performed, and information from different channels is fused through a 1×1 convolutional kernel to adjust the number of feature channels. Each convolutional layer is followed by batch normalization to standardize the features, accelerating network training convergence, and an activation function is used to introduce nonlinear transformations to enhance the expressive power of the features. Through this bottom-up, layer-by-layer refinement, the quality of the feature maps at each level is continuously improved.

[0040] The third module: Multi-scale feature fusion. In this embodiment, the decoding module achieves full fusion of multi-scale features through the collaborative work of two paths: top-down and bottom-up. The top-down path transmits high-level semantic information to the lower level, guiding the generation of detailed features; the bottom-up path feeds back low-level detailed information to the higher level, enriching the texture expression of semantic features. The lateral connection between the two paths ensures the smooth flow of information across different scales, so that the final generated feature map retains the overall structural information extracted by the encoding module while incorporating rich texture details, laying the foundation for high-quality image reconstruction.

[0041] The fourth module: Output layer. In this embodiment, the final part of the decoding module is the output layer, which maps multi-scale feature maps to the final fused image through convolution operations. The output layer uses a 1×1 convolution kernel to convert the multi-channel feature maps into a three-channel color image, ensuring that its size and number of channels are consistent with the input visible light image. The convolution operation linearly combines features using learnable convolution kernel parameters, extracting the information most suitable for human visual perception and subsequent detection tasks from the rich multi-scale features, generating a high-quality fused image that contains both the texture details of the visible light image and retains the spectral gradient features of the infrared image.

[0042] Step 3: Construct a multi-task detection model based on end-to-end joint optimization. This model includes a defect classification head and a defect segmentation head, which simultaneously output defect localization boxes and defect segmentation masks after inputting the fused image. In this embodiment, the constructed multi-task detection model is based on the ResNet18 architecture and includes a defect classification head and a defect segmentation head. The defect classification head consists of a region proposal network and a classification sub-network. It generates anchor boxes on the multi-scale feature maps of the fused image and performs classification after extracting fixed-size feature maps using RoI Align. The defect segmentation head employs a fully convolutional network with an encoder-decoder structure. It captures multi-scale contextual information through dilated spatial pyramid pooling and generates a binarized segmentation mask after upsampling.

[0043] In this embodiment, the defect classification head first generates candidate defect regions on the multi-scale feature map of the fused image using a region proposal network. This network pre-sets multiple anchor boxes of different sizes and aspect ratios at each location in the feature map; these anchor boxes serve as initial detection windows covering various regions in the image. Subsequently, the region proposal network performs two tasks on each anchor box: first, foreground-background binary classification to determine whether the anchor box contains a defect target; second, bounding box regression to fine-tune the position of the anchor box containing the defect, making it more accurately frame the defect region. Through the collaborative work of these two tasks, the network filters out high-quality candidate defect regions from a large number of anchor boxes, providing a foundation for subsequent fine-grained classification.

[0044] In this embodiment, the classification sub-network performs feature extraction on the candidate defect regions generated by the region proposal network. Since the candidate regions vary in size, they cannot be directly input into subsequent fully connected layers for processing; therefore, they need to be normalized into feature maps of a fixed size. The classification sub-network uses a feature extraction method, employing interpolation to uniformly adjust candidate regions of different sizes to the same size, while preserving key feature information within the defect regions, thus preparing for subsequent classification.

[0045] In this embodiment, the normalized fixed-size feature map is input into a fully connected layer for further processing. The fully connected layer unfolds the feature map into a one-dimensional feature vector and performs nonlinear transformations and combinations on the features through multi-layer neural connections to extract high-level semantic features for defect identification. Finally, the classifier calculates the probability of each candidate region belonging to each type of defect based on the feature vector output by the fully connected layer, and selects the category with the highest probability as the defect type for that region.

[0046] In this embodiment, the defect classification head can identify multiple types of insulator defects, including one or more of the following: contamination, cracks, skirt damage, and core rod moisture. Contamination defects refer to a layer of dirt accumulated on the insulator surface; crack defects refer to fine cracks appearing on or inside the insulator surface; skirt damage refers to physical damage to the insulator's skirt portion; and core rod moisture refers to a performance degradation of the insulator's internal core rod due to moisture intrusion. The classification head associates the identified defect type with the corresponding location frame coordinates, providing complete defect information for subsequent detection result output.

[0047] In this embodiment, the defect segmentation head employs a fully convolutional network with an encoder-decoder structure to achieve pixel-level accurate segmentation of defect regions. The encoder portion of the defect segmentation head shares the feature extraction layer output of the multi-task detection model, eliminating the need for re-extracting features, thus reducing the number of computational parameters and ensuring feature consistency. The shared feature maps contain rich semantic information and spatial details, providing high-quality input features for subsequent fine-grained segmentation.

[0048] In this embodiment, the encoder portion of the defect segmentation head further captures multi-scale contextual information through a dilated spatial pyramid pooling module. The dilated spatial pyramid pooling module contains multiple parallel convolutional branches, each using convolutional kernels with different dilation rates to process feature maps in parallel. Convolutional kernels with smaller dilation rates focus on local details of the defect, while those with larger dilation rates focus on the overall contour of the defect and surrounding environmental information. The outputs of each branch, after undergoing convolution operations to adjust the number of channels, are concatenated with the original feature map along the channel dimension. This allows the network to simultaneously utilize feature information from different receptive fields, effectively improving its ability to segment defects of varying sizes.

[0049] In this embodiment, the decoder part of the defect segmentation head restores the spatial resolution of the feature map through progressive upsampling, gradually enlarging the low-resolution semantic feature map to a high resolution. Upsampling is achieved using transposed convolution, a method that learns to enlarge the feature map, offering stronger feature recovery capabilities than traditional interpolation methods. After each upsampling layer, the decoder makes skip connections with the corresponding layer's feature map in the encoder, introducing shallow, detailed features into deeper semantic features and compensating for spatial details lost during upsampling.

[0050] In this embodiment, the feature map after skip connections is refined using depthwise separable convolution. Depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs spatial convolution independently on each channel, capturing feature relationships in the spatial dimension; pointwise convolution fuses information from different channels using a 1×1 convolution kernel. This convolution method significantly reduces the number of parameters and computational complexity while maintaining feature extraction capabilities, enabling the defect segmentation head to run efficiently on resource-constrained embedded devices.

[0051] In this embodiment, the output layer of the defect segmentation head generates a binary segmentation mask through an activation function. The activation function maps the feature map output by the network to probability values ​​between 0 and 1, where the value of each pixel represents the probability that the location belongs to a defect region. By setting an appropriate threshold, the probability map is converted into a binary segmentation mask, where regions with a pixel value of 1 correspond to the precise outline of the defect, and regions with a pixel value of 0 correspond to the background. The generated binary segmentation mask, together with the defect localization box coordinates, constitutes a complete defect detection result, providing accurate pixel-level information for subsequent defect analysis, area calculation, and trend prediction.

[0052] Step 4: End-to-end joint training and optimization. The spectral gradient-aware cross-modal attention fusion network and the multi-task detection model share a feature extraction layer, forming a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of the fusion image quality loss, defect detection loss, defect segmentation loss, and spectral gradient constraint loss. In this embodiment, a weighted loss function is used for joint training, and the weight coefficients of each loss are adaptively adjusted according to the convergence speed during training. By sharing a feature extraction layer with the detection model to form a unified neural network, and performing joint training by minimizing the weighted sum of the fusion image quality loss, defect detection loss, defect segmentation loss, and spectral gradient constraint loss, end-to-end optimization is achieved.

[0053] In this embodiment, the weighted loss function used in end-to-end joint training consists of a weighted sum of four parts: fusion image quality loss, defect detection loss, defect segmentation loss, and spectral gradient constraint loss. These four loss parts correspond to different optimization objectives of the fusion network and the multi-task detection model, respectively. By weighting and summing, multiple training objectives are unified into a single loss function, enabling the network to learn both the fusion and detection tasks simultaneously, thus achieving end-to-end joint optimization.

[0054] Specifically, the system's total loss function is composed of a weighted sum of four main loss terms (fusion quality, detection, segmentation, and spectral gradient constraint), and the weight coefficients of each loss term are... This is a dynamically adjusted balance coefficient.

[0055] , in, These are dynamic weighting coefficients, calculated using an uncertainty-weighted method, used to balance the training progress of each sub-task and prevent any one task from dominating the model's learning. To fuse image quality loss, Losses due to defect detection For defect segmentation loss, This is the spectral gradient constraint loss.

[0056] Image quality loss during fusion The result is a weighted sum of pixel-level loss and perceptual loss. Pixel-level loss calculates the difference at each pixel position between the fused image and the source image, ensuring that the fused image remains consistent with the source image at the pixel level. Perceptual loss uses an image classification network pre-trained on a large-scale image dataset, extracting convolutional features from multiple intermediate layers of this network, and calculating the distance between the fused image and the source image in these feature spaces. Since the intermediate layer features of the pre-trained network can reflect human perception of image content, perceptual loss can make the fused image more consistent with human observation habits in terms of visual quality and semantic content.

[0057] Specifically, the fusion quality loss is a weighted sum of pixel-level L1 loss and perceptual loss. The formula is as follows: , in, These are the weighting coefficients for pixel-level loss (controlling the degree of matching of the base pixels). The weighting coefficients for perceptual loss (controlling structural and texture similarity). Calculate the fused image using pixel-level L1 loss. With target image The mean absolute difference between them: , in, This represents the total number of pixels in the image. Using pixel indices, iterate through each pixel location to compute the fused image. With target image The pixel difference. To merge images The 1 pixel, For target image The 1 pixel. It is L1 norm. The merged image... Each pixel in With the corresponding target image pixels The comparisons are performed, and the average of the sum of the absolute differences is calculated. This process ensures that the merged image is as close as possible to the real image as possible overall (for each pixel).

[0058] To perceive the loss, Euclidean distance is calculated based on intermediate layer features extracted from a pre-trained network (such as VGG-16): , in, For the first Layer feature extraction function. For the first The channels, height, and width of the layer feature map. This is the feature layer index. It represents a specific intermediate layer (layer 3, 8, or 15) extracted from a pre-trained network (such as VGG-16). It is the L2 norm. Although here... and These are whole images, but they are pre-trained by a network. Converted into a feature map. At this point... and This can be understood as the first in the feature map. A feature vector (instead of pixels) is used to capture higher-level structural similarity.

[0059] Spectral gradient constraint loss This method preserves temperature gradient information from infrared images. First, a gradient map is calculated from the infrared image, obtaining the gradient values ​​in the horizontal and vertical directions for each pixel location. The gradient reflects the severity and direction of temperature changes. A spectral gradient constraint loss imposes constraints on the gradients of corresponding pixel locations in the fused image, ensuring that the gradient direction in high-temperature regions (such as hot spots in defects) remains consistent with the original infrared image. This loss uses a similarity loss function to calculate the difference between the gradients of the fused image and the infrared image. By minimizing this difference, it ensures that the fusion process does not destroy key gradient features in the infrared image, allowing the fused image to retain both visible light texture details and the complete temperature distribution information of the infrared image.

[0060] Spectral gradient constraint loss Based on the gradient direction similarity calculation between the fused image and the infrared image, cosine similarity loss is used: , in, These are the weighting coefficients for the spectral gradient constraint loss. To merge images The gradient map (obtained by the Sobel operator). Infrared image The gradient map (calculated by the Sobel operator). The physical meaning of this loss is: in high-temperature regions (above the temperature threshold T), the gradient direction of the fused image should be consistent with that of the original infrared image to preserve spectral gradient information.

[0061] Defect detection loss The loss is a weighted sum of the classification loss and regression loss of the region proposal network, and the cross-entropy loss of the classification subnetwork. The classification loss of the region proposal network determines whether the anchor box contains a defect target, while the regression loss fine-tunes the anchor box position to more accurately frame the defect. The cross-entropy loss of the classification subnetwork is used to classify the defects within the candidate region into specific types. The region proposal network uses a focus loss for classification. This focus loss adjusts the weights of easy and difficult samples, causing the network to pay more attention to samples that are difficult to classify, effectively mitigating the problem of positive and negative sample imbalance in defect detection.

[0062] Specifically, defect detection loss It consists of RPN classification loss (using focus loss), regression loss, and cross-entropy loss of the classification subnetwork.

[0063] , in, The weighting coefficients for the RPN classification loss (focus loss). The weighting coefficients for the regression loss (usually using Smooth L1 loss). These are the weight coefficients for the cross-entropy loss of the RCNN classifier subnetwork. Focal Loss is used to handle foreground / background sample imbalance. , in, This represents the prospect probabilities predicted by the model. This is a category balancing factor (adjusting the weights of foreground / background samples). The weight of the RPN classification loss in the total detection loss. The total number of anchor frames generated by RPN. Use anchor box indices. Iterate through each candidate anchor box to calculate the classification error.

[0064] For regression loss (Smooth L1): , in, These are the predicted bounding box coordinates and the actual bounding box coordinates, respectively. This represents the number of positive anchor frames participating in the regression. This is the index for the anchor boxes that are positive samples. Only anchor boxes marked as positive samples are traversed. The Smooth L1 loss function (also known as the Huber loss function) combines the advantages of L1 and L2: it behaves like L2 loss (smoothing) when the error is small, and like L1 loss (robust) when the error is large, thus providing a more stable and efficient gradient during training. For the error of a single regression parameter... =Predicted value - Actual value, the Smooth L1 loss is defined as follows: , When | When | < 1: Using the quadratic term (L2), the function is smooth (i.e., the gradient is continuous), which is beneficial for gradient descent. When | When |≥1: Using the linear term (L1) slows down the function's growth rate and reduces the impact of outliers.

[0065] The RCNN classification loss (cross-entropy) is calculated using the following formula: , in, This represents the total number of samples in the RCNN classification head (i.e., the number of RoIs). Index RoIs. Iterate through each candidate region and classify it. This represents the total number of defect categories (such as dirt, cracks, broken skirts, and damp mandrel). This is the true label (one-hot encoded). This represents the predicted class probability.

[0066] Defect segmentation loss A weighted sum of overlap loss and focus loss is employed. The overlap loss optimizes the degree of overlap between the predicted and ground truth segmented regions, ensuring the segmentation mask accurately covers defect areas. The focus loss optimizes the classification accuracy of defect boundary pixels, resulting in clearer and more accurate segmentation boundaries. The combination of these two losses guarantees the overall accuracy of the segmented regions while improving the fineness of boundary details.

[0067] Defect segmentation loss It is composed of a weighted average of Dice loss (overlap) and focus loss: , in, These are the weighting coefficients for the Dice loss and the segmentation focus loss, respectively.

[0068] The Dice loss is used to measure the overlap of the segmentation masks. , in, This represents the total number of pixels in the mask image. The pixel index is used for the mask. Each pixel position is iterated to calculate the overlap. These are the predicted mask pixel values ​​and the actual mask pixel values, respectively.

[0069] The focus loss for the segmentation task has the same structure as the focus loss for the detection task, and the formula is as follows: .

[0070] The weights of each loss term in the loss function are dynamically adjusted balancing coefficients. Each loss term corresponds to a learnable noise parameter, and the weight coefficient is inversely proportional to this noise parameter. During training, the weight coefficients are automatically adjusted based on the convergence speed of each loss term: loss terms that converge slowly receive larger weights and are given more attention; loss terms that converge quickly receive smaller weights. This adaptive adjustment mechanism avoids the tedious process of manual parameter tuning, enabling the network to automatically balance multiple training objectives and achieve better joint optimization results.

[0071] To accommodate the varying convergence speeds of different loss terms during training, an uncertainty-weighted method is employed. This reduces the contribution of faster-converging loss terms to the total loss, preventing overfitting. The formula is: , in, This is the learnable noise parameter (standard deviation) for the corresponding loss term, used to measure the uncertainty of the task. If a task (e.g., detection) performs unstablely (high uncertainty) in the early stages of training, the corresponding... If it is larger, then If the value is small, the model will tend to prioritize optimizing that task to reduce uncertainty.

[0072] Step 5: Output the detection results, which include the defect type, bounding box coordinates, and defect segmentation contour. In this embodiment, the fused image is input into the trained multi-task detection model, which simultaneously outputs the defect bounding box and the defect segmentation mask. Canny edge detection is used to extract the defect contour from the segmentation mask, and the Douglas-Puk algorithm is used to vectorize and simplify the contour. The defect type, bounding box coordinates, and polygon contour vector are integrated into a structured detection report and output.

[0073] In this embodiment, the binary segmentation mask generated by the defect segmentation head is first input into the Canny edge detection algorithm to extract the closed contour curve of the defect. The Canny edge detection algorithm is a multi-stage edge detection method. Through steps such as calculating image gradient, non-maximum suppression, and double thresholding, it can accurately identify the boundary between the defect region and the background, generating a continuous and closed contour curve, which provides a foundation for subsequent contour vectorization.

[0074] In this embodiment, the Douglas-Puk algorithm is used to vectorize and simplify the extracted closed contour curve. The Douglas-Puk algorithm is a curve simplification algorithm that recursively retains key points on the curve and eliminates redundant points, significantly reducing the number of vertices while maintaining the basic shape of the contour. The simplified polygonal contour representation not only preserves the geometric features of the defect but also significantly reduces storage space requirements, facilitating subsequent data transmission and analysis.

[0075] In this embodiment, the simplified polygonal contour vertex coordinates are spatially aligned with the defect location box coordinates to establish a correspondence between pixel coordinates and image physical coordinates. Using pre-calibrated camera parameters, the contour vertex coordinates in the pixel coordinate system are converted into actual physical size coordinates, enabling the geometric parameters of the defect, such as size and area, to be quantitatively expressed in physical units such as millimeters and square millimeters, providing an objective basis for assessing the severity of the defect.

[0076] In this embodiment, a temporal analysis is performed on the contour changes of the same defect based on the detection results of consecutive frame sequences. By tracking and matching the same defect in multiple frames of images, the changes in its contour vertex coordinates, area, perimeter, and other geometric parameters over time are recorded. Based on this temporal data, a prediction algorithm is used to analyze the change patterns of the defect, predict its future diffusion trend and evolution rate, and determine whether the defect is in a rapid development stage, providing forward-looking guidance for operation and maintenance decisions.

[0077] In this embodiment, diverse information such as defect type, location frame coordinates, polygon contour vector, defect area, diffusion rate, and trend prediction results are integrated into a structured inspection report. The report supports multiple output formats and includes defect location information, geometric shape, severity assessment, and development trend prediction, facilitating inspection personnel to quickly understand the insulator's condition and formulate corresponding maintenance strategies, thus achieving intelligent processing throughout the entire process from defect detection to condition assessment.

[0078] Example 2 like Figure 2 As shown, the present invention provides an embedded identification device architecture diagram for insulator defect detection that integrates infrared and visible light images, applied to the embedded identification device for insulator defect detection that integrates infrared and visible light images as described in Embodiment 1, comprising:

[0079] Data construction unit 210 is used to construct a dual-light image dataset of insulators, the dataset including registered visible light images and infrared images.

[0080] The image fusion unit 220 is equipped with a spectral gradient-aware cross-modal attention fusion network. The network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module. It is used to extract features, perform cross-modal interaction, constrain and reconstruct spectral gradients from the visible light image and the infrared image to generate a fused image.

[0081] The multi-task detection unit 230 is equipped with a multi-task detection model based on end-to-end joint optimization. The multi-task detection model includes a defect classification head and a defect segmentation head, which are used to simultaneously output defect localization boxes and defect segmentation masks after inputting the fused image.

[0082] The joint training and optimization unit 240 is used to perform end-to-end joint training and optimization of the spectral gradient sensing cross-modal attention fusion network and the multi-task detection model. The spectral gradient sensing cross-modal attention fusion network and the multi-task detection model share a feature extraction layer, forming a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss.

[0083] The detection result output unit 250 is used to output the detection result, which includes the defect type, the coordinates of the positioning box, and the defect segmentation contour.

[0084] Figure 3 This is an electronic device provided in one embodiment of the present invention. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0085] In this embodiment of the invention, the memory 300 is used to store executable instructions of the processor 301, which is configured to implement the method as described in the first aspect when executing the instructions.

[0086] In an embodiment of the present invention, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0087] In one embodiment of the present invention, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0088] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0089] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0090] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0091] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0092] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for detecting insulator defects by fusing infrared and visible light images, characterized in that, Includes the following steps: Step 1: Construct a dual-light image dataset for insulators, which includes registered visible light and infrared images; Step 2: Construct a spectral gradient-aware cross-modal attention fusion network. The network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module. It is used to extract features from the visible light image and the infrared image, perform cross-modal interaction, spectral gradient constraint and reconstruction, and generate a fused image. Step 3: Construct a multi-task detection model based on end-to-end joint optimization. The multi-task detection model includes a defect classification head and a defect segmentation head, which are used to simultaneously output defect localization boxes and defect segmentation masks after inputting the fused image. Step 4: End-to-end joint training and optimization. The spectral gradient-aware cross-modal attention fusion network and the multi-task detection model share the feature extraction layer to form a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss. Step 5: Output the detection results, which include the defect type, the coordinates of the location box, and the defect segmentation contour.

2. The method according to claim 1, characterized in that, The encoding module of the spectral gradient-aware cross-modal attention fusion network includes: The first encoder employs a multi-scale dilated convolutional structure, using dilated convolutional kernels with increasing dilation rates to capture global contextual information of the input image and generate a structural feature map containing multi-scale receptive fields. The second encoder employs an edge-aware convolutional kernel based on gradient operators. This kernel generates a feature map containing texture details by calculating the gradient response of the image in the horizontal and vertical directions. The third encoder superimposes the output features of the first and second encoders in the channel dimension through residual connections, and introduces a channel attention mechanism. The channel attention mechanism captures channel information through global pooling, generates channel weights after processing by a fully connected layer, and weights the spectral difference regions to generate a spectral supplementary feature map. The output features of the first encoder, the second encoder, and the third encoder are fused through a progressive feature aggregation strategy. The aggregation strategy adopts a hierarchical residual connection method, which adjusts the number of channels of shallow features through convolution and adds them to deep features element by element to form a multi-level feature representation.

3. The method according to claim 2, characterized in that, The cross-modal attention module includes parallel visible light feature branches and infrared feature branches. Each branch adopts a multi-layer Transformer encoder structure. Each Transformer encoder layer contains a multi-head self-attention sub-layer and a feedforward neural sub-layer. Each sub-layer is followed by layer normalization and residual connection. The cross-modal attention module further includes a cross-modal interaction layer, which includes a hierarchical alignment module and a bidirectional cross-attention mechanism; The hierarchical alignment module uses a local self-attention network to align visible light features and infrared features in the spatial dimension. The local self-attention network adopts a sliding window mechanism to calculate self-attention within each window and compensates for viewpoint deviation through learnable positional bias terms. The bidirectional cross-attention mechanism includes: a first cross-attention sublayer using visible light features as query terms and infrared features as keys and values, and obtaining visible light features through layer normalization and a feedforward network after multi-head cross-attention calculation; and a second cross-attention sublayer using infrared features as query terms and visible light features as keys and values, and obtaining infrared features through layer normalization and a feedforward network after multi-head cross-attention calculation. The output of the cross-modal interaction layer is processed by a feedforward neural network and an activation function, and then cross-modal fusion features are generated through a feature weighted fusion strategy. The feature weighted fusion strategy uses learnable weight parameters to weight and sum the visible light features and infrared features.

4. The method according to claim 1, characterized in that, The decoding module adopts a hierarchical feature pyramid network structure, including: The top-down upsampling path upsamples the high-level semantic features output by the encoding module and performs skip connections layer by layer with the corresponding level of encoded feature maps. The skip connections are performed by convolution aligning the number of channels and then adding them element by element. The bottom-up feature aggregation path inputs the feature map after skip connections to the convolutional block for feature refinement through lateral connections. The convolutional block adopts depthwise separable convolution, first performing depthwise convolution, then pointwise convolution, and each convolutional layer is followed by batch normalization and activation function. The output layer maps multi-scale feature maps into a fused image of the same size as the input image through convolution.

5. The method according to claim 1, characterized in that, The defect classification head includes a region proposal network and a classification sub-network; The region proposal network generates anchor boxes on the multi-scale feature maps of the fused image and generates candidate defect regions through foreground-background binary classification and bounding box regression. The classification subnetwork performs feature extraction on the candidate defect region, extracts a feature map of fixed size, and classifies the defects in the candidate region through a fully connected layer and a classifier; The defect types include one or more of the following: dirt, cracks, broken skirt, and damp mandrel.

6. The method according to claim 1, characterized in that, The defect segmentation head uses a fully convolutional network with an encoder-decoder structure. The encoder part of the defect segmentation head shares the feature extraction layer output of the multi-task detection model and captures multi-scale contextual information through the hollow spatial pyramid pooling module. The hollow spatial pyramid pooling module contains multiple parallel branches, each branch uses convolution with different hole rates, and the output of each branch is convolved with the original feature map after adjusting the number of channels. The decoder part of the defect segmentation head restores the spatial resolution of the feature map through upsampling and skip connections. Upsampling adopts transposed convolution. After each upsampling layer, skip connections are made with the feature map of the corresponding layer of the encoder, and depthwise separable convolution is used. The output layer of the defect segmentation head generates a binary segmentation mask through an activation function.

7. The method according to claim 1, characterized in that, The weighted loss function used in the end-to-end joint training and optimization consists of a weighted sum of fused image quality loss, defect detection loss, defect segmentation loss, and spectral gradient constraint loss. The quality loss of the fused image is composed of a weighted sum of pixel-level loss and perceptual loss, wherein the perceptual loss is calculated using intermediate layer convolutional features extracted by the pre-trained network. The spectral gradient constraint loss is calculated based on the temperature gradient corresponding to the pixel in the fused image. The temperature gradient is obtained from the infrared image. By calculating the gradient map of the infrared image, constraints are applied to the corresponding pixel positions in the fused image so that the gradient direction of the fused image in the high-temperature region is consistent with that of the infrared image. The spectral gradient constraint loss uses a similarity loss function to calculate the difference between the gradient of the fused image and the gradient of the infrared image. The defect detection loss is composed of the weighted sum of the classification loss and regression loss of the region proposal network and the cross-entropy loss of the classification subnetwork, wherein the classification loss of the region proposal network adopts the focus loss. The defect segmentation loss is a weighted sum of overlap loss and focus loss; The weight coefficients of the loss function are dynamically adjusted balance coefficients, which are adjusted according to the convergence speed of each loss term during the training process. The weight coefficient of each loss term is related to its corresponding learnable noise parameter.

8. The method according to claim 1, characterized in that, The step of constructing the insulator two-light image dataset further includes an image registration sub-step: SIFT feature points are extracted from visible light and infrared images respectively, and coarse matching is performed based on the nearest neighbor distance ratio. A progressive consistent sampling algorithm is used to eliminate mismatched point pairs and retain high-confidence in-point matching pairs. Calculate the homography transformation matrix based on interior point matching pairs; Perform perspective transformation on the visible light image to precisely align it with the infrared image at the pixel level; The registered image pairs are cropped to remove invalid areas caused by the transformation at the edges.

9. The method according to claim 1, characterized in that, The step of outputting the detection result further includes: The defect segmentation mask is input into the Canny edge detection algorithm to extract the closed contour curve of the defect; The Douglas-Puk algorithm is used to vectorize and simplify the contour curve, generating a polygonal contour representation with optimized storage space. The vertex coordinates of the polygonal contour are spatially aligned with the coordinates of the defect location box to establish the correspondence between pixel coordinates and image physical coordinates. Based on the detection results of consecutive frame sequences, a time series analysis is performed on the contour changes of the same defect to predict the diffusion trend and evolution rate of the defect. The defect type, location box coordinates, polygon contour vector, and diffusion trend prediction results are integrated into a structured inspection report and output.

10. An embedded identification device for insulator defect detection that integrates infrared and visible light images, used to implement the method according to any one of claims 1 to 9, characterized in that, include: A data construction unit is used to construct a dual-light image dataset of insulators, the dataset including registered visible light images and infrared images; The image fusion unit is equipped with a spectral gradient-aware cross-modal attention fusion network. The network includes an encoding module, a cross-modal attention module, a spectral gradient constraint module, and a decoding module, which are used to extract features, perform cross-modal interaction, constrain and reconstruct spectral gradients between the visible light image and the infrared image to generate a fused image. The multi-task detection unit is equipped with a multi-task detection model based on end-to-end joint optimization. The multi-task detection model includes a defect classification head and a defect segmentation head, which are used to input the fused image and simultaneously output the defect localization box and the defect segmentation mask. The joint training and optimization unit is used to perform end-to-end joint training and optimization of the spectral gradient sensing cross-modal attention fusion network and the multi-task detection model. The spectral gradient sensing cross-modal attention fusion network and the multi-task detection model share a feature extraction layer, forming a unified end-to-end neural network structure. Joint training is performed by minimizing the weighted sum of fusion image quality loss, defect detection loss, defect segmentation loss and spectral gradient constraint loss. The detection result output unit is used to output the detection result, which includes the defect type, the coordinates of the positioning box, and the defect segmentation contour.

Images