Power system insulator defect detection method and device based on bimodal data linkage, computer equipment, readable storage medium and program product

Abstract

The invention discloses a power system insulator defect detection method and device, computer equipment, a readable storage medium and a program product. The technical problem that an existing insulator defect detection method is poor in detection effect is solved. The method comprises the steps that S1, a visible light image and an infrared image of an insulator are obtained and input into an insulator bimodal defect detection network, and the insulator bimodal defect detection network comprises a bimodal parallel feature extraction module, a multi-level feature fusion extraction module and a defect detection head; s2, performing feature extraction on the visible light image and the infrared image by adopting a bimodal parallel feature extraction module, and outputting a visible light feature map and an infrared feature map; s3, performing feature fusion and extraction on the visible light feature map and the infrared feature map by adopting a multi-level feature fusion extraction module, and outputting a bimodal feature map; and S4, carrying out insulator defect detection on the bimodal characteristic pattern by adopting a defect detection head, and outputting an insulator defect detection result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of power systems, and particularly relates to a power system insulator defect detection method and device based on dual-mode data linkage, a computer device, a readable storage medium and a program product. BACKGROUND

[0002] Insulators are key core equipment for high-voltage power transmission of power systems, which provide reliable mechanical support for power transmission lines, ensure the stability of the line structure, and accurately isolate the current conduction between the live body and the grounding structure to avoid the risk of electric leakage. Once the insulator has defects such as cracks, aging, surface contamination, and zero value, it may initially cause partial discharge of the line, increase in power loss, and form potential safety hazards. If not handled in time, it may subsequently break through the insulating layer, leading to line short circuit, and even induce major safety accidents such as substation shutdown and large-scale power outage in cities, directly threatening the overall operation safety of the power system. Therefore, timely detection and accurate identification of insulator defects in power systems are key links to maintain stable operation of the power transmission network. Hereinafter, the insulator of the power system is referred to as the insulator.

[0003] Traditional insulator defect detection is mainly based on manual inspection, which has the following significant limitations: manual inspection is low in efficiency and high in cost, and needs to be operated in a high-altitude high-voltage environment with prominent safety risks; the equipment detection is easily disturbed by weather and light, and has poor precision stability; it is also difficult to identify hidden defects such as small cracks and early aging, resulting in a high rate of missed detection.

[0004] In recent years, image processing technology based on deep learning has gradually become the mainstream technical direction, which can automatically learn defect features from massive images, effectively adapt to complex background and noise interference scenes, realize detection process automation and reduce manual burden, and accurately identify small hidden defects, thereby further greatly improving detection efficiency and precision.

[0005] Among them, visible light-infrared dual-mode fusion detection shows outstanding advantages: the two can efficiently complement structural details and thermal anomaly information - visible light can assist infrared to exclude pseudo-thermal source interference, infrared can accurately identify internal defects that are difficult to detect by the naked eye, and can comprehensively cover shape and temperature defects, and has stronger robustness in complex scenes, thereby improving the efficiency of insulator defect identification.

[0006] Traditional insulator defect detection methods mainly rely on manual inspection or image processing models combined with software and hardware, which have the following limitations: (1) Existing methods for processing visible light images cannot simultaneously accurately capture the details and global features of multi-scale defects (small such as small cracks and large such as umbrella skirt large area damage) of insulators, have a high rate of missed detection of multi-size defects, easily lose spatial information of small defects due to ordinary down-sampling, cannot fully cover large defects due to limited receptive field, and have large fluctuations in detection accuracy in complex scenes; (2) The existing method for processing infrared images has the problems that traditional convolution is difficult to fit the shape of irregular heat-emitting areas, is easily disturbed by redundant heat information in the background, the boundary of thermal anomalies is extracted ambiguously, weak temperature difference defect features are easily submerged, effective thermal features are not strengthened, the discriminability and integrity of infrared defects are insufficient, and the detection accuracy and robustness are poor; (3) The existing dual-modal fusion method has the problems of extensive fusion mechanism, not deeply mining the complementarity of visible light texture and infrared thermal information, feature redundancy and low signal-to-noise ratio, being unable to design differential heterogeneous branches for texture, thermal anomaly and complex multi-source defects, the advantages of dual modal not being fully played, and the detection accuracy being greatly fluctuated in complex scenes; meanwhile, the model has a large number of parameters and is difficult to be deployed on edge devices such as unmanned aerial vehicles. SUMMARY

[0007] The application discloses a power system insulator defect detection method and device based on dual-modal data linkage, computer equipment, readable storage medium and program product, aiming at solving the problems of insufficient pertinence of visible light and infrared feature extraction, low dual-modal feature fusion efficiency and large model parameter quantity in the existing dual-modal insulator detection, to break through the limitations of poor feature complementarity in complex scenes, difficult deployment on edge devices, and difficult to balance detection real-time performance and accuracy.

[0008] Term explanation: Conv1D: a convolution operation for one-dimensional data, the convolution kernel only slides in a single spatial dimension, captures local dependencies in the sequence through a local window, can preserve the time sequence or linear continuity of the data, and is often used to extract local feature patterns in the sequence.

[0009] Conv3D: one of the most commonly used operations in two-dimensional convolution, the convolution kernel size is 3x3, and slides in the width and height two spatial dimensions of the feature map. Compared with small size convolution kernel, it can capture more rich local spatial correlation; compared with large size convolution kernel, it has advantages in balancing receptive field and computational efficiency, and is the core operation for extracting middle-level texture and structure features.

[0010] Conv5D: also a commonly used two-dimensional convolution operation, the convolution kernel size is 5x5, the receptive field is larger than that of the 3x3 convolution kernel, and more extensive spatial context information can be captured. Although the parameter quantity is slightly higher than that of the small size convolution kernel, it can effectively cover large-scale features and global correlations, and is suitable for scenes that need to integrate large range spatial information.

[0011] Concat: A commonly used feature fusion operation in deep learning, it integrates information by concatenating multiple feature maps of the same scale along the channel dimension. After concatenation, the width and height of the feature map remain unchanged, and the number of channels is the sum of the number of channels of each input feature map. This operation can preserve the integrity of features from different paths or levels, enhance the model's ability to express complex information, and is widely used in multimodal fusion and cross-layer feature connections.

[0012] Dynamic Convolution (DC) is a convolution technique that dynamically adjusts the parameters of the convolution kernel, overcoming the limitations of fixed parameters in traditional static convolution kernels. It dynamically generates convolution kernel weights based on the statistical characteristics of the input features through attention mechanisms or weight fusion strategies: first, it learns a set of basic convolution kernels, and then adaptively combines these kernels for each input sample, making the convolution operation more focused on key feature regions and improving its adaptability to diverse data.

[0013] Dilated Convolution: By introducing a dilation rate-controlled spacing between holes into the standard convolution kernel, the receptive field is expanded without increasing the number of parameters or computational cost. For example, a 3×3 dilated convolution with a dilation rate of 2 has a receptive field equivalent to a 5×5 standard convolution, while maintaining the same computational cost. It is suitable for capturing a wide range of contextual information on high-resolution feature maps, avoiding the loss of details caused by downsampling.

[0014] Depthwise separable convolution (DWConv): Each input channel is processed independently using a single-channel convolution kernel to extract spatial features within a single channel; pointwise convolution is responsible for fusing cross-channel information. Compared to standard convolution, DWConv significantly reduces the number of parameters and computational cost, down to 1 / N of the original, where N is the number of input channels, enabling efficient extraction of local features in a lightweight model.

[0015] Pointwise convolution PConv: The convolution kernel size is 1×1, and the operation is performed only on the channel dimension, without changing the width and height of the feature map. The main function of this operation is to adjust the number of channels, compress redundant channels or expand the channel dimension, and at the same time realize the information exchange between different channels. It is a basic operation for feature refinement and dimension adaptation.

[0016] Deformable Convolution: By learning an additional set of offsets on top of the fixed grid sampling points of standard convolution, the convolution kernel can dynamically adjust its sampling position based on the input features. This operation overcomes the limitations of the traditional fixed sampling mode of convolution, and can adaptively focus on key regions of irregularly shaped targets, improving the accuracy of capturing non-rigid features.

[0017] Max Pooling: Selects the maximum value in a local spatial window as the output, which is used to retain the most significant features in the local region, while reducing the spatial dimension of the feature map, reducing the amount of computation, and enhancing the robustness of the model to small displacements.

[0018] Average pooling (AvgPool): Calculates the average value as the output within a local spatial window to preserve the overall statistical features of the local region. Compared to max pooling, it focuses more on the smoothness of features and is suitable for scenarios that require the integration of global information of a region.

[0019] CBS submodule: It is a high-efficiency basic convolutional module that extracts local features through 3×3 convolutional kernels, stabilizes the distribution through BatchNorm normalization, and then introduces non-linearity to enhance expressive power through Swish activation function. It is widely used in visual backbone networks, taking into account both lightweight and feature enhancement.

[0020] Upsampling: This operation expands the width and height of the feature map through interpolation or deconvolution to restore the spatial resolution lost due to downsampling. It is mainly used for cross-scale feature fusion to ensure that the spatial location information of small-scale detailed features is not lost and to achieve spatial alignment of features at different levels.

[0021] Channel shuffling: This operation rearranges channels after grouped convolution to break the independence between different channel groups. By randomly mixing channels from different groups, it promotes the interaction of cross-group information along the channel dimension, avoids feature isolation, and enhances the effectiveness of feature fusion.

[0022] SeNet: A feature enhancement module based on channel attention, comprising two core steps: Squeeze and Excitation. Squeeze compresses the spatial information of each channel into a single-valued descriptor through global average pooling, representing the global response of the channel; Excitation learns channel weights through fully connected layers, adaptively enhancing channels that are critical to the task, suppressing redundant background channels, and improving the discriminative power of features.

[0023] CBAM: A two-dimensional attention module that combines channel attention and spatial attention. First, channel attention focuses on key feature channels to distinguish between effective features and redundant channels. Then, spatial attention locates important spatial regions in the feature map. This dual filtering strengthens effective information and enhances the convolutional neural network's attention to key features.

[0024] The A2C2f submodule is an improved feature extraction module proposed in the YOLOv12 algorithm. It combines area-attention and residual connections, primarily to improve the efficiency and accuracy of feature extraction. The defect detection head is responsible for parsing the fused feature map and generating the final prediction result. Its output is a three-dimensional tensor, and the prediction information for each spatial location includes the class probability of the predicted box, the coordinates of the bounding box, and the confidence of the object's presence. These are key parameters for model localization and classification.

[0025] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting defects in power system insulators based on dual-modal data linkage, comprising: S1. Acquire visible light and infrared images of the insulator, and input the visible light and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. S2. Using the dual-modal parallel feature extraction module, feature extraction is performed on the visible light image and the infrared image, and visible light feature map and infrared feature map are output; S3. Using the multi-level feature fusion extraction module, the visible light feature map and the infrared feature map are fused and extracted to output a dual-modal feature map; S4. Using the defect detection head, perform insulator defect detection on the dual-modal feature map and output the insulator defect detection result.

[0026] Preferably, the execution steps of the dual-modal parallel feature extraction module include: S2.1 Input the visible light image into the CBS submodule, extract features from the image to obtain the first visible light feature map X1; then, input X1 into the first multi-scale parallel dilated convolution submodule to obtain the second visible light feature map X2. S2.2 Input X2 into the CBS submodule for feature optimization to obtain the third visible light feature map X3; then, input X3 into the second multi-scale parallel dilated convolution submodule to obtain the fourth visible light feature map X4. S2.3. Input X4 into the CBS submodule to continuously optimize the feature representation quality and obtain the fifth visible light feature map X5; input X5 into the CBS submodule to further extract and optimize features and obtain the sixth visible light feature map X6; finally, input X6 into the third multi-scale parallel dilated convolution submodule to obtain the seventh visible light feature map X7. S2.4 Input the infrared image into the CBS submodule, extract the features of the image, and obtain the first infrared feature map Y1; then, input Y1 into the first pooling enhanced deformable attention submodule to obtain the second infrared feature map Y2; S2.5 Input Y2 into the CBS submodule to further optimize the features and obtain the third infrared feature map Y3; then, input Y3 into the second pooling enhanced deformable attention submodule to obtain the fourth infrared feature map Y4; S2.6 Input Y4 into the CBS submodule to continuously optimize the feature representation quality and obtain the fifth infrared feature map Y5; input Y5 into the CBS submodule to further extract and optimize features and obtain the sixth infrared feature map Y6; finally, input Y6 into the third pooling enhanced deformable attention submodule to obtain the seventh infrared feature map Y7.

[0027] Preferably, the execution steps of the multi-scale parallel dilated convolution submodule include: For ease of description, the input of the multi-scale parallel dilated convolution submodule is denoted as the visible light feature map A0; The visible light feature map A0 is input into the CBS submodule for feature extraction to obtain the first-scale fused visible light feature map A1; then, A1 is processed in three paths. (1) Path 1: Perform Conv1D operation on A1 to obtain the second-scale fused visible light feature map A2; perform dilated conv operation on A2 to obtain the third-scale fused visible light feature map A3; input A3 into the CBS submodule for feature optimization to obtain the fourth-scale fused visible light feature map A4; (2) Path 2: Perform Conv3D operation on A1 to obtain the fifth scale fused visible light feature map A5; perform dilated conv operation on A5 to obtain the sixth scale fused visible light feature map A6; input A6 into the CBS submodule for feature optimization to obtain the seventh scale fused visible light feature map A7; (3) Path 3: Perform Conv5D operation on A1 to obtain the eighth scale fused visible light feature map A8; perform dilated conv operation on A8 to obtain the ninth scale fused visible light feature map A9; input A9 into the CBS submodule for feature optimization to obtain the tenth scale fused visible light feature map A10; Next, A4, A7, and A10 are concatted to obtain the eleventh-scale fused visible light feature map A11; A11 is input into the CBS submodule to continuously optimize the feature representation quality, resulting in the twelfth-scale fused visible light feature map A12. A12 is then processed using the Sigmoid activation function to obtain A12'. A12' is then multiplied element-wise with A12 to obtain the thirteenth-scale fused visible light feature map A13. Finally, A13 is input into the CBS submodule for feature optimization, resulting in the fourteenth-scale fused visible light feature map A14.

[0028] Preferably, the execution steps of the pooling-enhanced deformable attention submodule include: For ease of description, the input of the pooling-enhanced deformable attention submodule is denoted as the infrared feature map B0; The infrared feature map B0 is input into the CBS submodule to extract features and obtain the first pooled fused infrared feature map B1. Then, B1 is processed in three paths. (1) Path 1: Perform MaxPool operation on B1 to obtain the second pooled fused infrared feature map B2; perform Conv1D operation on B2 to obtain the third pooled fused infrared feature map B3; input B3 into the CBS submodule for feature optimization to obtain the fourth pooled fused infrared feature map B4. (2) Path 2: Input B1 into the CBS submodule to further extract features and obtain the fifth pooling fusion infrared feature map B5; use the Sigmoid activation function to process B5 to obtain B5', and then perform element-wise multiplication of B5' and B5 to obtain the sixth pooling fusion infrared feature map B6. (3) Path 3: Perform average pooling (AvgPool) operation on B1 to obtain the seventh pooled fused infrared feature map B7; perform Conv1D operation on B7 to obtain the eighth pooled fused infrared feature map B8; input B8 into the CBS submodule to improve the feature quality and obtain the ninth pooled fused infrared feature map B9. Concatenate B4, B6, and B9 to obtain the tenth pooled fused infrared feature map B10; perform a deformable convolution operation on B10 to obtain the eleventh pooled fused infrared feature map B11. Input B11 into the CBAM submodule to obtain the twelfth pooled fusion infrared feature map B12; finally, input B12 into the CBS submodule to complete feature extraction and obtain the thirteenth pooled fusion infrared feature map B13.

[0029] Preferably, the execution steps of the multi-level feature fusion extraction module include: S3.1. Perform a concat operation on the second visible light feature map X2 and the second infrared feature map Y2 to obtain the first bimodal feature map Z1; input Z1 into the first bimodal lightweight fusion submodule to obtain the second bimodal feature map Z2; S3.2. Perform a concat operation on the fourth visible light feature map X4 and the fourth infrared feature map Y4 to obtain the third bimodal feature map Z3; input Z3 into the second bimodal lightweight fusion submodule to obtain the fourth bimodal feature map Z4; S3.3. Perform a concat operation on the seventh visible light feature map X7 and the seventh infrared feature map Y7 to obtain the fifth bimodal feature map Z5; input Z5 into the third bimodal lightweight fusion submodule to obtain the sixth bimodal feature map Z6; S3.4 Input Z6 into the A2C2f submodule to obtain the seventh bimodal feature map Z7. Perform an upsampling operation on Z7 to obtain the eighth bimodal feature map Z8. S3.5. Concat Z8 and Z4 to obtain the ninth bimodal feature map Z9. Input Z9 into the A2C2f submodule to obtain the tenth bimodal feature map Z10. S3.6 Input Z10 into the CBS submodule, optimize the feature quality to obtain the eleventh bimodal feature map Z11, and perform an upsampling operation on Z11 to obtain the twelfth bimodal feature map Z12; S3.7. Concat Z12 and Z2 to obtain the thirteenth bimodal feature map Z13; input Z13 into the fourth bimodal lightweight fusion submodule to obtain the fourteenth bimodal feature map Z14. S3.8 Input Z14 into the CBS submodule for feature optimization to obtain the fifteenth bimodal feature map Z15; Concat Z15 with Z10 to obtain the sixteenth bimodal feature map Z16. S3.9 Input Z16 into the fifth bimodal lightweight fusion submodule to obtain the seventeenth bimodal feature map Z17; input Z17 into the CBS submodule for feature optimization to obtain the eighteenth bimodal feature map Z18; S3.10. Perform a concat operation on Z18 and Z7 to obtain the nineteenth bimodal feature map Z19. Input Z19 into the sixth bimodal lightweight fusion submodule to obtain the twentieth bimodal feature map Z20.

[0030] Preferably, the execution steps of the dual-modal lightweight fusion submodule include: For ease of description, the input of the bimodal lightweight fusion submodule is denoted as the bimodal feature map C0; The bimodal feature map C0 is input into the CBS submodule for feature processing to obtain the first lightweight fusion feature map C1. C1 is then input into the SeNet submodule to obtain the second lightweight fusion feature map C2. Next, a channel shuffle operation is performed on C2 to obtain the third lightweight fused feature map C3; then, C3 is processed in three paths. (1) Path 1: Perform a deformable convolution operation on C3 to obtain the fourth lightweight fusion feature map C4, perform a depthwise separable convolution operation on C4 to obtain the fifth lightweight fusion feature map C5, and then perform a pointwise convolution operation on C5 to obtain the sixth lightweight fusion feature map C6. (2) Path 2: Perform Conv3D operation on C3 to obtain the seventh lightweight fusion feature map C7, perform depthwise separable convolution (DWConv) operation on C7 to obtain the eighth lightweight fusion feature map C8; then perform pointwise convolution (PConv) operation on C8 to obtain the ninth lightweight fusion feature map C9. (3) Path 3: Perform dynamic convolution (Dynamic Conv) operation on C3 to obtain the tenth lightweight fusion feature map C10, perform depthwise separable convolution (DWConv) operation on C10 to obtain the eleventh lightweight fusion feature map C11; then perform pointwise convolution (PConv) operation on C11 to obtain the twelfth lightweight fusion feature map C12. Concat C6, C9, and C12 to obtain the thirteenth lightweight fusion feature map C13; finally, input C13 into the CBS submodule to complete feature extraction and fusion, and output the fourteenth lightweight fusion feature map C14.

[0031] This invention also provides a power system insulator defect detection device based on dual-modal data linkage, characterized in that it includes: The visible light image and infrared image acquisition module is used to acquire visible light images and infrared images, and input the visible light images and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. An insulator defect feature extraction module is used to extract insulator defect features from the visible light image and the infrared image using the dual-modal parallel feature extraction module, and output visible light feature map and infrared feature map; The dual-modal feature map generation module is used to fuse and extract features from the visible light feature map and the infrared feature map using the multi-level feature fusion extraction module, and output a dual-modal feature map. An insulator defect detection module is used to perform insulator defect detection on the dual-modal feature map using the defect detection head, and output the insulator defect detection results.

[0032] The present invention also provides a computer device, characterized in that it includes: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any of the preceding claims.

[0033] The present invention also provides a computer-readable storage medium, characterized in that it includes: a computer program stored thereon, wherein when the computer program is executed, it implements the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any of the preceding claims.

[0034] The present invention also provides a computer program product, characterized in that it includes: the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any of the preceding claims.

[0035] This invention can fully leverage the complementarity between visible light texture details and infrared thermal information, enhancing the feature representation capabilities under complex scenarios such as lighting changes and thermal interference. At the same time, it reduces the number of model parameters and computational load with lightweight modules, adapting to the real-time inspection needs of devices such as drones and edge terminals, and providing more efficient and accurate technical support for intelligent inspection of power transmission networks. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a power system insulator defect detection method based on dual-modal data linkage, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the dual-mode defect detection network structure for insulators. Figure 3 This is a schematic diagram of a multi-scale parallel dilated convolutional submodule structure. Figure 4 This is a schematic diagram of the pooling-enhanced deformable attention submodule structure; Figure 5 This is a schematic diagram of the multi-level feature fusion extraction module structure; Figure 6 This is a schematic diagram of the dual-modal lightweight fusion submodule structure. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Example

[0040] like Figure 1 As shown, this embodiment of the invention provides a method for detecting defects in power system insulators based on dual-modal data linkage, including: S1. Acquire visible light and infrared images of the insulator, and input the visible light and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. Furthermore, this invention proposes a Dual-Modal Fusion Insulator Defect Detection Network (DMF-InsNet), overcoming existing technological bottlenecks: After inputting visible light and infrared images, the dual-modal parallel feature extraction module accurately extracts multi-scale structural features from the visible light image through a multi-scale parallel dilated convolution submodule, while capturing irregular thermal information from the infrared image using a pooling-enhanced deformable attention submodule; subsequently, in the multi-level feature fusion extraction module, lightweight and deeply complementary fusion of visible light and infrared features is performed through a dual-modal lightweight fusion submodule; finally, the results are input into the defect detection head to obtain the insulator defect detection results, enabling insulator defect detection in complex scenarios. The main structure of DMF-InsNet is as follows: Figure 2 As shown: S2. Using the dual-modal parallel feature extraction module, feature extraction is performed on the visible light image and the infrared image, and visible light feature map and infrared feature map are output; S3. Using the multi-level feature fusion extraction module, the visible light feature map and the infrared feature map are fused and extracted to output a dual-modal feature map; S4. Using the defect detection head, perform insulator defect detection on the dual-modal feature map and output the insulator defect detection result.

[0041] In one embodiment of the present invention, step S2 includes the following execution steps of the dual-modal parallel feature extraction module: S2.1 Input the visible light image into the CBS submodule to extract and optimize features, and obtain the first visible light feature map X1; then, input X1 into the first MPDC submodule, and use multi-branch convolution operation to simultaneously capture small-scale features such as tiny cracks and corona discharge points of the insulator, as well as large-scale features such as the overall shape of the insulator skirt, and obtain the second visible light feature map X2. S2.2 Input X2 into the CBS submodule to further optimize the feature representation. While filtering out redundant background information, the effective defect features of the insulator are enhanced to obtain the third visible light feature map X3. Then, input X3 into the second MPDC submodule to perform multi-scale feature extraction on the insulator defects again. The defect patterns such as the extension texture of cracks and the morphological changes of skirt damage in medium-sized features are mined to obtain the fourth visible light feature map X4. S2.3 Input X4 into the CBS submodule to continuously optimize the feature representation quality and obtain the fifth visible light feature map X5; input X5 into the CBS submodule to further extract and optimize features and obtain the sixth visible light feature map X6; input X6 into the third MPDC submodule to extract multi-scale features of insulator defects, especially for capturing subtle defect features of insulators and obtain the seventh visible light feature map X7. S2.4. Input the infrared image into the CBS submodule to extract basic thermal features, such as the temperature distribution contour of the insulator and the preliminary shape of the hot zone, to obtain the first infrared feature map Y1. Then, input Y1 into the first PEDA submodule. With the local focusing capability of pooling, the shape adaptation capability of deformable convolution and the feature enhancement capability of attention, it accurately focuses on the local high temperature area of ​​the insulator, filters redundant thermal information in the background, adapts to the shape of irregular heating area, enhances the effective thermal features of the insulator and suppresses noise, and makes the infrared weak temperature difference signal more prominent, to obtain the second infrared feature map Y2. S2.5 Input Y2 into the CBS submodule to further optimize the thermal characteristics of insulator defects and obtain the third infrared feature map Y3; then, input Y3 into the second PEDA submodule to further enhance the thermal characteristics of insulator defects and accurately mine the defect patterns in the thermal characteristics to obtain the fourth infrared feature map Y4. S2.6 Input Y4 into the CBS submodule to continuously optimize the thermal feature characterization quality and obtain the fifth infrared feature image Y5; Input Y5 into the CBS submodule again to further extract thermal features and obtain the sixth infrared feature image Y6; Finally, Input Y6 into the third PEDA submodule to enhance the thermal features of insulator defects and improve the ability to capture the thermal features of insulator defects and obtain the seventh infrared feature image Y7.

[0042] It should be noted that the dual-modal parallel feature extraction module extracts features from both visible light and infrared images, encompassing both visual information such as insulator texture details and thermal signals such as thermal anomaly distribution. This avoids the limitations of single-modal insulator feature extraction, enabling comprehensive defect identification. (1) By coordinating the CBS submodule and the MPDC submodule, the multi-scale defect features of insulators from micro to macro in visible light images are fully explored: ensuring that small defect details are not lost and fully covering the anomalies of large-scale structures, providing accurate and rich visible light feature support for subsequent dual-modal fusion and defect detection, effectively improving the accuracy of insulator defect detection and robustness in complex scenarios.

[0043] (2) By working together with the CBS submodule and the PEDA submodule, the multi-dimensional thermal features of insulator defects in infrared images are fully explored, providing accurate and robust infrared thermal feature support for insulator defect detection, and effectively improving the accuracy and reliability of insulator thermal defect detection in complex scenarios.

[0044] The dual-modal parallel feature extraction module adopts a parallel structure, which can improve computational efficiency while ensuring the quality of feature extraction, and balance the requirements of detection accuracy and real-time performance, effectively adapting to the complex scenarios of insulator defect detection.

[0045] Further, in step S2.1, a multi-scale parallel dilated convolution (MPDC) submodule is constructed. The structure of the multi-scale parallel dilated convolution submodule is as follows: Figure 3 As shown, its execution flow includes: For ease of description, the input of the MPDC submodule is denoted as the visible light feature map A0; First, the visible light feature map A0 is input into the CBS submodule for feature extraction, resulting in the first-scale fused visible light feature map A1. Then, A1 is processed along three paths: (1) Path 1: Perform Conv1D operation on A1 to capture small-scale features of insulator defects, ensuring that small-scale features are not lost, and obtain the second-scale fused visible light feature map A2; perform dilated conv operation on A2 to expand the receptive field without increasing the number of parameters, obtain the contextual information of small-scale features, and obtain the third-scale fused visible light feature map A3; input A3 into the CBS submodule to filter out noise and enhance effective small-scale features, and obtain the fourth-scale fused visible light feature map A4; (2) Path 2: Perform Conv3D operation on A1 to capture the medium-scale features of insulator defects and obtain the fifth-scale fused visible light feature map A5. Perform dilated conv operation on A5 to expand the receptive field to obtain the correlation information of medium-scale features and obtain the sixth-scale fused visible light feature map A6. Input A6 into the CBS submodule to enhance the consistency and effectiveness of medium-scale features and obtain the seventh-scale fused visible light feature map A7. (3) Path 3: Perform Conv5D operation on A1 to capture the large-scale features of insulator defects and obtain the eighth-scale fused visible light feature map A8. Perform dilated conv operation on A8 to further expand the receptive field and fully grasp the large-scale features to obtain the ninth-scale fused visible light feature map A9. Input A9 into the CBS submodule to improve the quality of the large-scale features and obtain the tenth-scale fused visible light feature map A10. Next, A4, A7, and A10 are concatted to fuse small, medium, and large-scale features, aggregating multi-scale information in the channel dimension to enhance the richness and completeness of insulator defect features, resulting in the eleventh-scale fused visible light feature map A11. A11 is input into the CBS submodule to integrate the spliced ​​multi-scale features of insulator defects, further improving the quality of the multi-scale features and obtaining the twelfth-scale fused visible light feature map A12. A12 is then processed using the Sigmoid activation function to obtain A12'. A12' and A12 are then multiplied element-wise, and the weights are adaptively adjusted according to the feature content to accurately enhance the key features of insulator defects and suppress redundant background, making the effective features more prominent, resulting in the thirteenth-scale fused visible light feature map A13. Finally, A13 is input into the CBS submodule for feature processing to obtain the fourteenth-scale fused visible light feature map A14.

[0046] It should be noted that the multi-scale parallel dilated convolution submodule relies on the collaborative design of multi-dimensional convolution, dilated convolution, and attention mechanisms to comprehensively mine the global correlations of small-scale, medium-scale, and large-scale features of the input features: the combination of multi-dimensional convolution and dilated convolution can cover the details and contextual information of features at different scales, ensuring the richness of insulator defect features; the attention mechanism can specifically strengthen key feature information, suppress redundant interference, and significantly improve the accuracy and effect of feature extraction.

[0047] Further, in step S2.4, a Pooled Enhanced Deformable Attention (PEDA) submodule is constructed. The structure of the Pooled Enhanced Deformable Attention submodule is as follows: Figure 4 As shown, its execution flow includes: For ease of description, the input of the PEDA submodule is denoted as infrared feature map B0; First, the infrared feature map B0 is input into the CBS submodule to initially extract the thermal features of insulator defects, resulting in the first pooled fused infrared feature map B1. Then, B1 is processed in three paths. (1) Path 1: Perform MaxPool operation on B1 to highlight the most significant local thermal signal of insulator defects, strengthen key thermal areas and suppress background thermal interference to obtain the second pooled fused infrared feature map B2; perform Conv1D operation on B2 to capture the fine local thermal features of the insulator to obtain the third pooled fused infrared feature map B3; input B3 into the CBS submodule to improve the purity and representativeness of small-scale thermal features to obtain the fourth pooled fused infrared feature map B4; (2) Path 2: Input B1 into the CBS submodule to further extract the basic thermal features of insulator defects and obtain the fifth pooled fusion infrared feature map B5; use the Sigmoid activation function to process B5 to obtain B5', and then perform element-wise multiplication with B5' to adaptively enhance the effective thermal features of the insulator, suppress redundant information, focus on key thermal areas, and obtain the sixth pooled fusion infrared feature map B6. (3) Path 3: Perform average pooling (AvgPool) operation on B1 to integrate local thermal information of insulator defects, smooth the thermal signal, obtain the overall thermal trend of the region, and obtain the seventh pooled fused infrared feature map B7; perform Conv1D operation on B7 to capture local thermal features in another dimension, and obtain the eighth pooled fused infrared feature map B8; input B8 into the CBS submodule to improve the quality of regional thermal features and obtain the ninth pooled fused infrared feature map B9; Then, B4, B6 and B9 are concatted to aggregate different thermal features of insulator defects in the channel dimension, enhancing feature richness and obtaining the tenth pooled fused infrared feature map B10; Deformable convolution is performed on B10 to dynamically adapt to the irregular shape of thermal anomalies, accurately capture thermal feature deformation, and improve adaptability to non-rigid thermal anomalies, resulting in the eleventh pooled fused infrared feature map B11. Next, B11 is input into the CBAM submodule to enhance thermal feature attention from both channel and spatial dimensions, focusing on the effective thermal region of insulator defects and filtering out invalid background, resulting in the twelfth pooled fusion infrared feature map B12; finally, B12 is input into the CBS submodule to complete feature fusion extraction and output the thirteenth pooled fusion infrared feature map B13.

[0048] It should be noted that the pooling-enhanced deformable attention submodule, through the synergy of multi-branch pooling, deformable convolution, and attention mechanisms, not only achieves complete preservation of local details of insulator infrared thermal features, comprehensive grasp of global thermal distribution trends, and effective enhancement of key thermal regions, but also flexibly adapts to the non-rigid forms of thermal defects and accurately removes background interference information. This comprehensively improves the precision, deformation robustness, and discrimination purity of infrared thermal features, providing high-quality and robust feature support for the infrared branch of dual-mode insulator defect detection.

[0049] As one embodiment of the present invention, in step S3, the structural schematic diagram of the multi-level feature fusion extraction module is as follows: Figure 5 As shown, its execution steps include: S3.1. Perform a concatenation operation on the second visible light feature map X2 and the second infrared feature map Y2 to obtain the first bimodal feature map Z1. Input Z1 into the first DMLF submodule, and enhance the complementarity of the bimodal insulator defect features by using the cross-modal attention mechanism and convolution operation. Accurately locate the infrared thermal features using the visible light texture, and at the same time use the infrared thermal features to highlight the abnormal edges in the visible light to obtain the second bimodal feature map Z2. S3.2. Perform a concat operation on the fourth visible light feature map X4 and the fourth infrared feature map Y4 to obtain the third bimodal feature map Z3. Input Z3 into the second DMLF submodule to further explore the correlation of bimodal insulator defect features, obtain the correspondence between the texture morphology and thermal features at the insulator skirt damage point, and obtain the fourth bimodal feature map Z4. S3.3. Perform a concat operation on the seventh visible light feature map X7 and the seventh infrared feature map Y7 to obtain the fifth bimodal feature map Z5; input Z5 into the third DMLF submodule to enhance the discriminativeness of the bimodal insulator defect features, obtain the overall morphology of the insulator defect and the coupling features of thermal anomaly aggregation, and obtain the sixth bimodal feature map Z6. S3.4 Input Z6 into the A2C2f submodule, dynamically adjust the number of channels and feature scale according to the feature content, compress redundant channels, enhance key feature dimensions, and improve the efficiency of insulator defect feature expression to obtain the seventh bimodal feature map Z7; perform upsampling (UpSample) operation on Z7 to restore the spatial resolution of the features, make up for the details lost in the deep features of insulator defects, and make the position information of small-scale features more accurate to obtain the eighth bimodal feature map Z8; S3.5. Perform a concat operation on Z8 and Z4 to fuse the enhanced features with the bimodal features, resulting in the ninth bimodal feature map Z9. Input Z9 into the A2C2f submodule to optimize the cross-scale fused features, making the bimodal insulator defect information of different scales more coordinated, resulting in the tenth bimodal feature map Z10. S3.6 Input Z10 into the CBS submodule, extract optimized features and filter out noise to obtain the eleventh bimodal feature map Z11; perform upsampling (UpSample) operation on Z11 to improve spatial resolution to match the scale of insulator defect features to obtain the twelfth bimodal feature map Z12. S3.7. Perform a concat operation on Z12 and Z2 to fuse the bimodal features and cross-scale enhancement features, resulting in the thirteenth bimodal feature map Z13. Input Z13 into the fourth DMLF submodule to strengthen the correlation of cross-scale bimodal insulator defect features, obtain the texture details, thermal anomalies and multi-scale context features of small-scale features, and obtain the fourteenth bimodal feature map Z14. S3.8 Input Z14 into the CBS submodule for feature processing to obtain the fifteenth bimodal feature map Z15; Concat Z15 with Z10 to obtain the sixteenth bimodal feature map Z16. S3.9 Input Z16 into the fifth DMLF submodule to further mine the fine-grained correlation of dual-mode insulator defect features and obtain the seventeenth dual-mode feature map Z17; input Z17 into the CBS submodule for feature processing and obtain the eighteenth dual-mode feature map Z18. S3.10. Perform a concat operation on Z18 and Z7 to fuse the dual-mode insulator defect features and the cross-scale insulator defect enhancement features, resulting in the nineteenth dual-mode feature map Z19. Input Z19 into the sixth DMLF submodule to complete the final dual-mode deep fusion and output the twentieth dual-mode feature map Z20.

[0050] It should be noted that the multi-level feature fusion extraction module performs multi-scale deep fusion of visible light texture and infrared thermal features, enabling insulator defect features to possess both morphological recognition and thermal property discrimination capabilities, greatly enriching the information dimensions of insulator defect identification. By dynamically adjusting the channels and scale of features and restoring spatial resolution, it not only solves the interaction barriers between features at different levels but also fully preserves the fine details of small-scale defects, improving defect localization accuracy. At the same time, it can effectively control computational complexity, achieving a balance between detection efficiency and accuracy. With the integration of multi-scale and multi-modal insulator defect features, the robustness and reliability of insulator defect classification and localization in complex environments are greatly enhanced.

[0051] Further, in step S3.1, a Dual-Modal Lightweight Fusion (DMLF) submodule is constructed. The structure of the Dual-Modal Lightweight Fusion submodule is as follows: Figure 6 As shown, its execution flow includes: For ease of description, the input of the DMLF submodule is denoted as the bimodal feature map C0; The bimodal feature map C0 is input into the CBS submodule to optimize the basic features of insulator defects, resulting in the first lightweight fusion feature map C1; C1 is input into the SeNet submodule to adaptively enhance the channel features in the bimodality through the channel attention mechanism, resulting in the second lightweight fusion feature map C2. Next, a channel shuffle operation is performed on C2 to break down the mode barriers between channels and promote the mixed interaction of visible light features and infrared features of the insulator in the channel dimension, resulting in the third lightweight fusion feature map C3. Subsequently, C3 is processed through three paths; (1) Path 1: Perform Deformable Conv operation on C3 to dynamically adapt to the irregular shape of insulator defects, improve the feature capture capability of irregular defects, and obtain the fourth lightweight fusion feature map C4. Perform Depth Separable Conv operation on C4 to extract the spatial features of insulator defects and obtain the fifth lightweight fusion feature map C5. Then perform pointwise conv operation on C5 and adjust the number of channels to obtain the sixth lightweight fusion feature map C6. (2) Path 2: Perform Conv3D operation on C3 to mine the correlation of dual-modal features in multiple scale dimensions and obtain the seventh lightweight fusion feature map C7. Perform depthwise separable convolution (DWConv) operation on C7 to extract the spatial features of insulator defects and obtain the eighth lightweight fusion feature map C8. Then perform pointwise convolution (PConv) operation on C8 to obtain the ninth lightweight fusion feature map C9 and enhance the correlation of features. (3) Path 3: Perform dynamic convolution (Dynamic Conv) operation on C3, adaptively adjust the weight of the convolution kernel according to the feature content, accurately enhance the key information of insulator defects, and obtain the tenth lightweight fusion feature map C10; perform depthwise separable convolution (DWConv) operation on C10 to extract the spatial features of insulator defects, and obtain the eleventh lightweight fusion feature map C11; then perform pointwise convolution (PConv) operation on C11 to obtain the twelfth lightweight fusion feature map C12; By concatting C6, C9, and C12, the multi-morphological and multi-dimensional dual-mode insulator defect features captured by different paths are fused to obtain the thirteenth lightweight fusion feature map C13. Finally, C13 is input into the CBS submodule to complete feature fusion and extraction, and the fourteenth lightweight fusion feature map C14 is output.

[0052] It should be noted that the dual-modal lightweight fusion submodule, while maintaining computational efficiency, deeply integrates the advantages of dual-modal features, namely visible light and infrared thermal features. This module first breaks down information barriers between modes through channel attention mechanisms and channel shuffling operations. Then, it extracts multi-dimensional features of insulator defects in parallel using multi-path depthwise separable convolutions. Combined with deformable convolutions to adapt to the irregular shapes of insulator defects and dynamic convolutions to adaptively enhance key defect information, it possesses strong adaptability to changes in defect morphology and strengthens key dual-modal information, providing feature support for lightweight models in insulator defect detection tasks.

[0053] As one embodiment of the present invention, in step S4, a defect detection head is used to perform insulator defect detection on the dual-modal feature map and output the insulator defect detection result; wherein the insulator defect detection result includes information such as predicted bounding box coordinates, defect category label and confidence score.

[0054] For example, visible light and infrared images of insulators are acquired by simultaneously collecting visible light and infrared images of insulators using an inspection device equipped with dual cameras; The dual-modal parallel feature extraction module is used to extract features from the visible light image and the infrared image, generating visible light feature maps and infrared feature maps: A visible light image of size 640×640×3 is input into the CBS submodule to obtain a first visible light feature map X1 of size 320×320×64; then, X1 is input into the first MPDC submodule to obtain a second visible light feature map X2 of size 320×320×128. Inputting X2 into the CBS submodule yields a third visible light feature map X3 with a size of 160×160×128; then, inputting X3 into the second MPDC submodule yields a fourth visible light feature map X4 with a size of 160×160×256. Inputting X4 into the CBS submodule yields the fifth visible light feature map X5, which has a size of 80×80×256. Inputting X5 into the CBS submodule yields the sixth visible light feature map X6, which has a size of 40×40×256. Finally, inputting X6 into the third MPDC submodule yields the seventh visible light feature map X7, which has a size of 40×40×512.

[0055] An infrared image of size 640×640×1 is input into the CBS submodule to obtain a first infrared feature map Y1 of size 320×320×64; then, Y1 is input into the first PEDA submodule to obtain a second infrared feature map Y2 of size 320×320×128. Then, Y2 is input into the CBS submodule to obtain the third infrared feature map Y3 with a size of 160×160×128; then, Y3 is input into the second PEDA submodule to obtain the fourth infrared feature map Y4 with a size of 160×160×256. Next, Y4 is input into the CBS submodule to obtain the fifth infrared feature map Y5 with a size of 80×80×256; Y5 is input into the CBS submodule to obtain the sixth infrared feature map Y6 with a size of 40×40×256; finally, Y6 is input into the third PEDA submodule to obtain the seventh infrared feature map Y7 with a size of 40×40×512.

[0056] The construction and operation process of the multi-scale parallel dilated convolution (MPDC) submodule is as follows: For ease of description, the input of the MPDC submodule is denoted as the visible light feature map A0; First, input A0, which is 320×320×64, into the CBS submodule to obtain the first-scale fused visible light feature map A1, which is 320×320×64. Subsequently, A1 is processed through three paths; (1) Path 1: Perform Conv1D operation on A1 to obtain a second-scale fused visible light feature map A2 with a size of 320×320×64; perform Dilated Conv operation on A2 to obtain a third-scale fused visible light feature map A3 with a size of 320×320×64; input A3 into the CBS submodule to obtain a fourth-scale fused visible light feature map A4 with a size of 320×320×64; (2) Path 2: Perform Conv3D operation on A1 to obtain a fifth-scale fused visible light feature map A5 with a size of 320×320×64; perform Dilated Conv operation on A5 to obtain a sixth-scale fused visible light feature map A6 with a size of 320×320×64; input A6 into the CBS submodule to obtain a seventh-scale fused visible light feature map A7 with a size of 320×320×64; (3) Path 3: Perform Conv5D operation on A1 to obtain the eighth-scale fused visible light feature map A8 with a size of 320×320×64; perform Dilated Conv operation on A8 to obtain the ninth-scale fused visible light feature map A9 with a size of 320×320×64; input A9 into the CBS submodule to obtain the tenth-scale fused visible light feature map A10 with a size of 320×320×64; Next, A4, A7, and A10 are concatted to obtain the eleventh-scale fused visible light feature map A11 with a size of 320×320×192. Inputting A11 into the CBS submodule yields a 12th-scale fused visible light feature map A12 with a size of 320×320×192. Subsequently, A12 is processed using the Sigmoid activation function to obtain A12'. Then, A12' is multiplied element-wise with A12 to obtain a 13th-scale fused visible light feature map A13 with a size of 320×320×192. Finally, A13 is input into the CBS submodule to obtain a 14th-scale fused visible light feature map A14 with a size of 320×320×128.

[0057] The construction and operation process of the Pooled Enhanced Deformable Attention (PEDA) submodule is as follows: For ease of description, the input of the PEDA submodule is denoted as infrared feature map B0; Inputting B0, which is 320×320×64 in size, into the CBS submodule yields the first pooled fused infrared feature map B1, which is 320×320×64 in size. Then, B1 is processed in three paths. (1) Path 1: Perform MaxPool operation on B1 to obtain the second pooled fusion infrared feature map B2 with a size of 320×320×64; perform Conv1D operation on B2 to obtain the third pooled fusion infrared feature map B3 with a size of 320×320×64; then input B3 into the CBS submodule to obtain the fourth pooled fusion infrared feature map B4 with a size of 320×320×64; (2) Path 2: Input B1 into the CBS submodule to obtain the fifth pooling fusion infrared feature map B5 with a size of 320×320×64; process B5 using the Sigmoid activation function to obtain B5', and then perform element-wise multiplication of B5' and B5 to obtain the sixth pooling fusion infrared feature map B6 with a size of 320×320×64; (3) Path 3: Perform AvgPool operation on B1 to obtain the seventh pooling fusion infrared feature map B7 with a size of 320×320×64; perform Conv1D operation on B7 to obtain the eighth pooling fusion infrared feature map B8 with a size of 320×320×64; input B8 into the CBS submodule to obtain the ninth pooling fusion infrared feature map B9 with a size of 320×320×64; Perform a Concat operation on B4, B6, and B9 to obtain the tenth pooled fusion infrared feature map B10 with a size of 320×320×192; perform a Deformable Conv operation on B10 to obtain the eleventh pooled fusion infrared feature map B11 with a size of 320×320×192. Inputting B11 into the CBAM submodule yields the twelfth pooled fusion infrared feature map B12, which has a size of 320×320×192; inputting B12 into the CBS submodule yields the thirteenth pooled fusion infrared feature map B13, which has a size of 320×320×128.

[0058] The multi-level feature fusion extraction module is used to fuse and extract features from the visible light feature map and the infrared feature map, and output a dual-modal feature map. Perform a concat operation on X2 and Y2 to obtain a first bimodal feature map Z1 with a size of 320×320×256; input Z1 into the first DMLF submodule to obtain a second bimodal feature map Z2 with a size of 320×320×128; Perform a concat operation on X4 and Y4 to obtain a third bimodal feature map Z3 with a size of 160×160×512; input Z3 into the second DMLF submodule to obtain a fourth bimodal feature map Z4 with a size of 160×160×256. Perform a concat operation on X7 and Y7 to obtain the fifth bimodal feature map Z5 with a size of 40×40×1024; input Z5 into the third DMLF submodule to obtain the sixth bimodal feature map Z6 with a size of 40×40×512. Inputting Z6 into the A2C2f submodule yields the seventh bimodal feature map Z7 with a size of 80×80×256; performing the UpSample operation on Z7 yields the eighth bimodal feature map Z8 with a size of 160×160×256. Perform a concat operation on Z8 and Z4 to obtain the ninth bimodal feature map Z9 with a size of 160×160×512; input Z9 into the A2C2f submodule to obtain the tenth bimodal feature map Z10 with a size of 160×160×256. Inputting Z10 into the CBS submodule yields the eleventh bimodal feature map Z11 with a size of 160×160×256; performing the UpSample operation on Z11 yields the twelfth bimodal feature map Z12 with a size of 320×320×256. Perform a concat operation on Z12 and Z2 to obtain the thirteenth bimodal feature map Z13 with a size of 320×320×384; input Z13 into the fourth DMLF submodule to obtain the fourteenth bimodal feature map Z14 with a size of 320×320×192; Inputting Z14 into the CBS submodule yields the fifteenth bimodal feature map Z15 with a size of 320×320×192; performing a Concat operation between Z15 and Z10 yields the sixteenth bimodal feature map Z16 with a size of 320×320×448. Inputting Z16 into the fifth DMLF submodule yields the seventeenth bimodal feature map Z17 with a size of 320×320×224; inputting Z17 into the CBS submodule yields the eighteenth bimodal feature map Z18 with a size of 320×320×224. Perform a concat operation on Z18 and Z7 to obtain the nineteenth bimodal feature map Z19 with a size of 320×320×480; input Z19 into the sixth DMLF submodule to obtain the twentieth bimodal feature map Z20 with a size of 320×320×240. The construction and operation process of the Dual-Modal Lightweight Fusion (DMLF) submodule is as follows: For ease of description, the input of the DMLF submodule is denoted as the bimodal feature map C0; Inputting C0, which has a size of 320×320×256, into the CBS submodule yields a first lightweight fusion feature map C1, which has a size of 320×320×256; inputting C1 into the SeNet submodule yields a second lightweight fusion feature map C2, which has a size of 320×320×256. Next, a Channel Shuffle operation is performed on C2 to obtain a third lightweight fusion feature map C3 with a size of 320×320×256; then, C3 is processed in three paths. Path 1: Perform a Deformable Conv operation on C3 to obtain a fourth lightweight fusion feature map C4 with a size of 320×320×256; perform a DWConv operation on C4 to obtain a fifth lightweight fusion feature map C5 with a size of 320×320×85; then perform a PConv operation on C5 to obtain a sixth lightweight fusion feature map C6 with a size of 320×320×85. Path 2: Perform Conv3D operation on C3 to obtain the seventh lightweight fusion feature map C7 with a size of 320×320×256; perform DWConv operation on C7 to obtain the eighth lightweight fusion feature map C8 with a size of 320×320×85; then perform PConv operation on C8 to obtain the ninth lightweight fusion feature map C9 with a size of 320×320×85. Path 3: Perform a Dynamic Conv operation on C3 to obtain the tenth lightweight fusion feature map C10 with a size of 320×320×256; perform a DWConv operation on C10 to obtain the eleventh lightweight fusion feature map C11 with a size of 320×320×86; then perform a PConv operation on C11 to obtain the twelfth lightweight fusion feature map C12 with a size of 320×320×86. Perform a Concat operation on C6, C9, and C12 to obtain the thirteenth lightweight fusion feature map C13 with a size of 320×320×256. Finally, C13 is input into the CBS submodule to obtain the thirteenth lightweight fusion feature map C13 with a size of 320×320×128.

[0059] The defect detection head is used to detect insulator defects on the dual-modal feature map, and the insulator defect detection results are output. These results include predicted bounding box coordinates, defect category labels, and confidence scores.

[0060] Example 2: This invention also provides a power system insulator defect detection device based on dual-modal data linkage, comprising: The visible light image and infrared image acquisition module is used to acquire visible light images and infrared images, and input the visible light images and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. An insulator defect feature extraction module is used to extract insulator defect features from the visible light image and the infrared image using the dual-modal parallel feature extraction module, and output visible light feature map and infrared feature map; The dual-modal feature map generation module is used to fuse and extract features from the visible light feature map and the infrared feature map using the multi-level feature fusion extraction module, and output a dual-modal feature map. An insulator defect detection module is used to perform insulator defect detection on the dual-modal feature map using the defect detection head, and output the insulator defect detection results.

[0061] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any of the above embodiments.

[0062] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any of the above embodiments.

[0063] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the deep learning-based method for detecting foreign objects attached to utility poles as described in any of the above embodiments.

[0064] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting defects in power system insulators based on dual-modal data linkage, characterized in that, include: S1. Acquire visible light and infrared images of the insulator, and input the visible light and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. S2. Using the dual-modal parallel feature extraction module, feature extraction is performed on the visible light image and the infrared image, and visible light feature map and infrared feature map are output; S3. Using the multi-level feature fusion extraction module, the visible light feature map and the infrared feature map are fused and extracted to output a dual-modal feature map; S4. Using the defect detection head, perform insulator defect detection on the dual-modal feature map and output the insulator defect detection result.

2. The power system insulator defect detection method based on dual-modal data linkage as described in claim 1, characterized in that, The execution steps of the dual-modal parallel feature extraction module include: S2.1 Input the visible light image into the CBS submodule, extract features from the image to obtain the first visible light feature map X1; then, input X1 into the first multi-scale parallel dilated convolution submodule to obtain the second visible light feature map X2. S2.2 Input X2 into the CBS submodule for feature optimization to obtain the third visible light feature map X3; then, input X3 into the second multi-scale parallel dilated convolution submodule to obtain the fourth visible light feature map X4. S2.

3. Input X4 into the CBS submodule to continuously optimize the feature representation quality and obtain the fifth visible light feature map X5; input X5 into the CBS submodule to further extract and optimize features and obtain the sixth visible light feature map X6; finally, input X6 into the third multi-scale parallel dilated convolution submodule to obtain the seventh visible light feature map X7. S2.4 Input the infrared image into the CBS submodule, extract the features of the image, and obtain the first infrared feature map Y1; then, input Y1 into the first pooling enhanced deformable attention submodule to obtain the second infrared feature map Y2; S2.5 Input Y2 into the CBS submodule to further optimize the features and obtain the third infrared feature map Y3; then, input Y3 into the second pooling enhanced deformable attention submodule to obtain the fourth infrared feature map Y4; S2.6 Input Y4 into the CBS submodule to continuously optimize the feature representation quality and obtain the fifth infrared feature map Y5; input Y5 into the CBS submodule to further extract and optimize features and obtain the sixth infrared feature map Y6; finally, input Y6 into the third pooling enhanced deformable attention submodule to obtain the seventh infrared feature map Y7.

3. The power system insulator defect detection method based on dual-modal data linkage as described in claim 2, characterized in that, The execution steps of the multi-scale parallel dilated convolution submodule include: For ease of description, the input of the multi-scale parallel dilated convolution submodule is denoted as the visible light feature map A0; The visible light feature map A0 is input into the CBS submodule for feature extraction to obtain the first-scale fused visible light feature map A1; then, A1 is processed in three paths. (1) Path 1: Perform Conv1D operation on A1 to obtain the second-scale fused visible light feature map A2; perform dilated conv operation on A2 to obtain the third-scale fused visible light feature map A3; input A3 into the CBS submodule for feature optimization to obtain the fourth-scale fused visible light feature map A4; (2) Path 2: Perform Conv3D operation on A1 to obtain the fifth scale fused visible light feature map A5; perform dilated conv operation on A5 to obtain the sixth scale fused visible light feature map A6; input A6 into the CBS submodule for feature optimization to obtain the seventh scale fused visible light feature map A7; (3) Path 3: Perform Conv5D operation on A1 to obtain the eighth scale fused visible light feature map A8; perform dilated conv operation on A8 to obtain the ninth scale fused visible light feature map A9; input A9 into the CBS submodule for feature optimization to obtain the tenth scale fused visible light feature map A10; Next, A4, A7, and A10 are concatted to obtain the eleventh-scale fused visible light feature map A11; A11 is input into the CBS submodule to continuously optimize the feature representation quality, resulting in the twelfth-scale fused visible light feature map A12. A12 is then processed using the Sigmoid activation function to obtain A12'. A12' is then multiplied element-wise with A12 to obtain the thirteenth-scale fused visible light feature map A13. Finally, A13 is input into the CBS submodule for feature optimization, resulting in the fourteenth-scale fused visible light feature map A14.

4. The power system insulator defect detection method based on dual-modal data linkage as described in claim 2, characterized in that, The execution steps of the pooling-enhanced deformable attention submodule include: For ease of description, the input of the pooling-enhanced deformable attention submodule is denoted as the infrared feature map B0; The infrared feature map B0 is input into the CBS submodule to extract features and obtain the first pooled fused infrared feature map B1. Then, B1 is processed in three paths. (1) Path 1: Perform MaxPool operation on B1 to obtain the second pooled fused infrared feature map B2; perform Conv1D operation on B2 to obtain the third pooled fused infrared feature map B3; input B3 into the CBS submodule for feature optimization to obtain the fourth pooled fused infrared feature map B4. (2) Path 2: Input B1 into the CBS submodule to further extract features and obtain the fifth pooling fusion infrared feature map B5; use the Sigmoid activation function to process B5 to obtain B5', and then perform element-wise multiplication of B5' and B5 to obtain the sixth pooling fusion infrared feature map B6. (3) Path 3: Perform average pooling (AvgPool) operation on B1 to obtain the seventh pooled fused infrared feature map B7; perform Conv1D operation on B7 to obtain the eighth pooled fused infrared feature map B8; input B8 into the CBS submodule to improve the feature quality and obtain the ninth pooled fused infrared feature map B9. Concatenate B4, B6, and B9 to obtain the tenth pooled fused infrared feature map B10; perform a deformable convolution operation on B10 to obtain the eleventh pooled fused infrared feature map B11. Input B11 into the CBAM submodule to obtain the twelfth pooled fusion infrared feature map B12; finally, input B12 into the CBS submodule to complete feature extraction and obtain the thirteenth pooled fusion infrared feature map B13.

5. The power system insulator defect detection method based on dual-modal data linkage as described in claim 1, characterized in that, The execution steps of the multi-level feature fusion extraction module include: S3.

1. Perform a concat operation on the second visible light feature map X2 and the second infrared feature map Y2 to obtain the first bimodal feature map Z1; input Z1 into the first bimodal lightweight fusion submodule to obtain the second bimodal feature map Z2; S3.

2. Perform a concat operation on the fourth visible light feature map X4 and the fourth infrared feature map Y4 to obtain the third bimodal feature map Z3; input Z3 into the second bimodal lightweight fusion submodule to obtain the fourth bimodal feature map Z4; S3.

3. Perform a concat operation on the seventh visible light feature map X7 and the seventh infrared feature map Y7 to obtain the fifth bimodal feature map Z5; input Z5 into the third bimodal lightweight fusion submodule to obtain the sixth bimodal feature map Z6; S3.4 Input Z6 into the A2C2f submodule to obtain the seventh bimodal feature map Z7. Perform an upsampling operation on Z7 to obtain the eighth bimodal feature map Z8. S3.

5. Concat Z8 and Z4 to obtain the ninth bimodal feature map Z9. Input Z9 into the A2C2f submodule to obtain the tenth bimodal feature map Z10. S3.6 Input Z10 into the CBS submodule, optimize the feature quality to obtain the eleventh bimodal feature map Z11, and perform an upsampling operation on Z11 to obtain the twelfth bimodal feature map Z12; S3.

7. Concat Z12 and Z2 to obtain the thirteenth bimodal feature map Z13; input Z13 into the fourth bimodal lightweight fusion submodule to obtain the fourteenth bimodal feature map Z14. S3.8 Input Z14 into the CBS submodule for feature optimization to obtain the fifteenth bimodal feature map Z15; Concat Z15 with Z10 to obtain the sixteenth bimodal feature map Z16. S3.9 Input Z16 into the fifth bimodal lightweight fusion submodule to obtain the seventeenth bimodal feature map Z17; input Z17 into the CBS submodule for feature optimization to obtain the eighteenth bimodal feature map Z18; S3.

10. Perform a concat operation on Z18 and Z7 to obtain the nineteenth bimodal feature map Z19. Input Z19 into the sixth bimodal lightweight fusion submodule to obtain the twentieth bimodal feature map Z20.

6. The power system insulator defect detection method based on dual-modal data linkage as described in claim 5, characterized in that, The execution steps of the dual-modal lightweight fusion submodule include: For ease of description, the input of the bimodal lightweight fusion submodule is denoted as the bimodal feature map C0; The bimodal feature map C0 is input into the CBS submodule for feature processing to obtain the first lightweight fusion feature map C1. C1 is then input into the SeNet submodule to obtain the second lightweight fusion feature map C2. Next, a channel shuffle operation is performed on C2 to obtain the third lightweight fused feature map C3; then, C3 is processed in three paths. (1) Path 1: Perform a deformable convolution operation on C3 to obtain the fourth lightweight fusion feature map C4, perform a depthwise separable convolution operation on C4 to obtain the fifth lightweight fusion feature map C5, and then perform a pointwise convolution operation on C5 to obtain the sixth lightweight fusion feature map C6. (2) Path 2: Perform Conv3D operation on C3 to obtain the seventh lightweight fusion feature map C7, perform depthwise separable convolution (DWConv) operation on C7 to obtain the eighth lightweight fusion feature map C8; then perform pointwise convolution (PConv) operation on C8 to obtain the ninth lightweight fusion feature map C9. (3) Path 3: Perform dynamic convolution (Dynamic Conv) operation on C3 to obtain the tenth lightweight fusion feature map C10, perform depthwise separable convolution (DWConv) operation on C10 to obtain the eleventh lightweight fusion feature map C11; then perform pointwise convolution (PConv) operation on C11 to obtain the twelfth lightweight fusion feature map C12. Concat C6, C9, and C12 to obtain the thirteenth lightweight fusion feature map C13; finally, input C13 into the CBS submodule to complete feature extraction and fusion, and output the fourteenth lightweight fusion feature map C14.

7. A power system insulator defect detection device based on dual-modal data linkage, characterized in that, include: The visible light image and infrared image acquisition module is used to acquire visible light images and infrared images, and input the visible light images and infrared images into the insulator dual-mode defect detection network. The insulator dual-mode defect detection network includes a dual-mode parallel feature extraction module, a multi-level feature fusion extraction module, and a defect detection head. An insulator defect feature extraction module is used to extract insulator defect features from the visible light image and the infrared image using the dual-modal parallel feature extraction module, and output visible light feature map and infrared feature map; The dual-modal feature map generation module is used to fuse and extract features from the visible light feature map and the infrared feature map using the multi-level feature fusion extraction module, and output a dual-modal feature map. An insulator defect detection module is used to perform insulator defect detection on the dual-modal feature map using the defect detection head, and output the insulator defect detection results.

8. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the power system insulator defect detection method based on dual-modal data linkage as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed, implements the power system insulator defect detection method based on dual-modal data linkage as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the power system insulator defect detection method based on dual-modal data linkage as described in any one of claims 1-6.

Images