A multi-modal power equipment detection method based on infrared guidance and contrast enhancement

Abstract

The application provides a multimodal power equipment detection method based on infrared guidance and contrast enhancement, visible light images and infrared images after preprocessing are input into a double-branch target detection network based on YOLO11 to detect the faults of power equipment; the double-branch target detection network based on YOLO11 comprises a double-branch main network, a neck network and a detection head which are connected in sequence; the double-branch main network comprises a visible light image feature extraction branch and an infrared image feature extraction branch which are connected in parallel; the neck network comprises a bottom-up path and a top-down path, an IGF asymmetric guidance edge perception fusion module is introduced into the bottom-up path, a CFGM cross-scale feature gate regulation module is introduced into the top-down path, and an M2BE multi-scale semantic and boundary enhancement module is introduced into the detection head. The application makes full use of modal information, improves the cross-scale feature fusion effect and the recognition degree of similar category features.

Description

Technical Field

[0001] This invention relates to the technical field of power industry automation, and in particular to a method for detecting multimodal power equipment. Background Technology

[0002] Current power system equipment inspection mainly relies on deep learning technology, using convolutional neural networks and target detectors to automatically identify power scenes. Existing technologies can be mainly divided into the following three categories: 1) Detection methods based on single-modal visible light (RGB): Existing technologies mostly use improved Faster R-CNN, ResNet-18, and YOLO series models (such as YOLOX, YOLOv4, YOLOv5s). These methods enhance the feature extraction capabilities of insulators, surge arresters, and disconnectors by introducing attention mechanisms, hybrid attention and feature balancing strategies, and AFF-BiFPN feature fusion structures designed for occlusion. However, this type of technology is overly dependent on lighting conditions and cannot capture key thermal information reflecting the equipment status. 2) Detection methods based on single-modal infrared (IR): For thermal defect identification, existing technologies utilize enhanced feature fusion SSD models, equal quantization FCOS algorithms, or rotating frame detection frameworks (such as R3Det) to locate key components in infrared images. Although infrared images can reflect thermal anomalies, their low resolution and severe texture loss make accurate localization difficult in complex backgrounds, and they are prone to localization deviations. 3) Preliminary multimodal fusion detection methods: To integrate the advantages of the two types of images, existing technologies attempt to guide cross-layer fusion through dual-branch feature extraction networks or integrate image fusion and detection into a unified framework using shared feature extraction networks. The above-mentioned existing technologies still have significant shortcomings in practical applications: 1) Insufficient utilization of modal complementarity: Existing multimodal fusion strategies mostly rely on simple weighted summation or local fusion, failing to deeply explore the complementary semantic information between cross-modalities, resulting in limited detection performance in low-contrast or complex backgrounds. 2) Limited generalization ability and multi-class recognition: Most studies focus on single-type equipment or specific defects (such as only for insulator bursting), lacking comprehensive models that can simultaneously recognize multiple types of equipment such as transformers, surge arresters, and disconnect switches, and have low recognition accuracy for small targets.

[0003] Currently, power equipment detection technology is still generally limited to single-modal or single-category recognition. Existing fusion strategies are too simple and lack global correlation, resulting in problems such as insufficient utilization of modal information, cross-scale feature fusion, and insufficient recognition of similar category features. They are difficult to solve the problems of coexistence of multiple types of equipment, redundancy of cross-modal information, and missed detection of small targets in complex power scenarios. Summary of the Invention

[0004] To address the technical challenges faced by existing detectors in processing multimodal data from power equipment, such as underutilization of modal information, cross-scale feature fusion, and insufficient recognition of similar category features, this invention proposes a multimodal power equipment detection method based on infrared guidance and contrast enhancement. Based on the improved YOLO11 model, using infrared and visible light images as dual inputs, an asymmetric guided edge-aware fusion (IGF) module is introduced at three scales in the backbone. The infrared and visible light features at corresponding scales are input into the IGF for fusion, achieving deep complementarity and efficient utilization of infrared-visible multimodal information, thus solving the problem of underutilization of modal information. At the network neck, a cross-scale feature gating (CFGM) module is designed to replace the traditional stitching method. Through an adaptive gating mechanism, more efficient feature collaboration is achieved, addressing the problem of poor cross-scale feature fusion performance. Finally, a multi-head, multi-block enhancement module (M²BE) is introduced before each detection head. This module combines the auxiliary loss of contrastive learning to enhance the discriminability of target features without increasing inference overhead, thus solving the problem of insufficient recognition of similar category features.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] S1: Acquire visible light and infrared images of the substation's power equipment and perform preprocessing;

[0007] S2: Input the preprocessed visible light image and infrared image into a YOLO11-based dual-branch target detection network to perform fault detection of power equipment;

[0008] The YOLO11-based dual-branch target detection network includes a dual-branch backbone network, a neck network, and a detection head connected in sequence. The dual-branch backbone network includes parallel visible light image feature extraction branches and infrared image feature extraction branches. The neck network includes a bottom-up path and a top-down path. An IGF asymmetric guided edge perception fusion module is introduced in the bottom-up path to achieve infrared guided fusion, modal interaction enhancement, and edge texture optimization. A CFGM cross-scale feature gating adjustment module is introduced in the top-down path to achieve cross-scale feature fusion. An M²BE multi-scale semantic and boundary enhancement module is introduced in the detection head to enhance the discriminability of target features.

[0009] Furthermore, the bottom-up path from the bottom network to the top network includes a first asymmetric guided edge perception fusion module, a second asymmetric guided edge perception fusion module, and a third asymmetric guided edge perception fusion module. The first asymmetric guided edge perception fusion module, the second asymmetric guided edge perception fusion module, and the third asymmetric guided edge perception fusion module take the output features of the visible light image feature extraction branch and the infrared image feature extraction branch at different scales as inputs, respectively. The third asymmetric guided edge perception fusion module is connected to the SPPF module and the C2PSA module in sequence.

[0010] The top-down path, from the top-level network to the bottom-level network, sequentially includes a first Upsample module, a first cross-scale feature gating adjustment module, a first C3K2 module, a second Upsample module, a second cross-scale feature gating adjustment module, a second C3K2 module, a first Conv module, a third cross-scale feature gating adjustment module, a third C3K2 module, a second Conv module, a fourth cross-scale feature gating adjustment module, and a fourth C3K2 module. The output of the C2PSA module is connected to the first Upsample module and the fourth cross-scale feature gating adjustment module, respectively. The output of the first asymmetric guided edge perception fusion module is connected to the second cross-scale feature gating adjustment module, the output of the second asymmetric guided edge perception fusion module is connected to the first cross-scale feature gating adjustment module, and the output of the first C3K2 module is also connected to the third cross-scale feature gating adjustment module.

[0011] In the detection head, the output of the second C3K2 module is connected to the first multi-scale semantic and boundary enhancement module, and the output of the first multi-scale semantic and boundary enhancement module is connected to the first Detect module; the output of the third C3K2 module is connected to the second multi-scale semantic and boundary enhancement module, and the output of the second multi-scale semantic and boundary enhancement module is connected to the first Detect module; the output of the fourth C3K2 module is connected to the third multi-scale semantic and boundary enhancement module, and the output of the third multi-scale semantic and boundary enhancement module is connected to the third Detect module.

[0012] Furthermore, the asymmetric-guided edge-aware fusion module includes sequentially executed:

[0013] The AIG asymmetric infrared guidance module is used to take visible light image features and infrared image features of the same scale, as well as the downsampling results output by the previous asymmetric infrared guidance module, as input. With infrared mode as the main mode, it guides visible light features through thermal signals to achieve preliminary cross-modal fusion of visible light image features and infrared image features, and obtain preliminary cross-modal fusion features.

[0014] The CMI modal interaction module is used to take preliminary cross-modal fusion features as input and, through intermodal interaction operations, enable visible light and infrared light to achieve complementary enhancement at the semantic and structural levels, thereby obtaining modal interaction enhancement features.

[0015] The ETE edge and texture extraction module takes modal interaction enhancement features as input, explicitly extracts gradient information in the horizontal and vertical directions through the Sobel operator, and combines convolution to extract local texture structure, enhance edge and texture information, and obtain the final gradient texture enhancement features.

[0016] Furthermore, using the infrared mode as the primary modality, visible light features are guided by thermal signals to achieve preliminary cross-modal fusion of visible light and infrared image features, obtaining preliminary cross-modal fusion features, including:

[0017] The features output by the asymmetric infrared guidance module of the previous layer are downsampled and mapped to the same size as the input of the edge perception fusion module of the current layer's asymmetric guidance, resulting in cross-scale supplementary features. And the visible light features input from the current layer's asymmetric-guided edge-aware fusion module. and infrared features These serve as inputs for the cross-scale supplementary branch, the visible light branch, and the infrared branch, respectively.

[0018] By supplementing features across scales Visible light characteristics and infrared features X and Y axis attention calculations were performed separately to obtain the X and Y axis attention enhancement features of the cross-scale supplementary branch, visible light branch and infrared branch respectively;

[0019] By coaxially fusing the X and Y axis attention enhancement features of the cross-scale supplementary branch with the X and Y axis attention enhancement features of the visible light branch and the infrared branch respectively, and then performing cross-axial fusion, the cross-scale-visible light height-width fused features are obtained. and cross-scale-infrared high-width fusion features ;

[0020] High-width fusion features High-width fusion features Perform convolution, batch normalization, and ReLU activation operations sequentially to obtain the first activation representation and the second activation representation.

[0021] The first and second activation representations are split into height and width branches along the channel dimension, respectively, and the first height feature map is generated by two independent 1×1 convolutions. First width feature map and the second height feature map Second width feature map ;

[0022] Visible light characteristics and complementary features across scales Element-by-element addition and merging with the first height feature map First width feature map Element-wise multiplication yields the output characteristics of the visible light branch. Infrared features and complementary features across scales Element-by-element addition and merging with the second height feature map Second width feature map The output characteristics of the infrared branch are obtained by element-wise multiplication. .

[0023] Output characteristics of the infrared branch After further processing through pooling, convolution, batch normalization (BN), and ReLU operations, the input is fed into the attention mask generator to obtain the channel attention weights. and filter attention weights Channel attention weights Output characteristics of the infrared branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the infrared branch is obtained after multiplication. Channel attention weights Output characteristics of visible light branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the visible light branch is obtained after multiplication. Attention feature map of infrared branch Attention feature map of visible light branch This serves as the initial cross-modal fusion feature.

[0024] Furthermore, cross-axial fusion is performed by coaxially fusing the X and Y axis attention enhancement features of the cross-scale supplementary branch with the X and Y axis attention enhancement features of the visible light branch and the infrared branch, respectively. This includes: fusing the X axis attention enhancement features of the cross-scale supplementary branch... X-axis attention enhancement features compared to the visible light branch X-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the horizontal joint representation of the cross-scale supplementary branch and the visible light branch fused together. Horizontal joint representation of cross-scale supplementary branch and infrared branch fusion ; Enhance Y-axis attention features for cross-scale supplementary branches Y-axis attention enhancement features compared to the visible light branch Y-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the joint vertical representation of the fusion of the cross-scale supplementary branch and the visible light branch. Vertical joint representation of cross-scale supplementary branch and infrared branch fusion ; Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and visible light branches are obtained by stitching along the channel. Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and infrared branches are obtained by stitching along the channel. .

[0025] Furthermore, through intermodal interaction operations, visible light and infrared light achieve complementary enhancement at the semantic and structural levels, obtaining modal interaction enhancement features, including:

[0026] Attention feature map of infrared branch Attention feature map of visible light branch The attention feature map of the visible light branch is used as input. Attention feature map of infrared branch Perform 1×1 convolution operations separately to obtain visible light feature projection maps used for modulating infrared features. and infrared light feature projection map used to guide visible light features ;

[0027] Projecting visible light features Attention feature map with infrared branch The first cross-enhancing feature is obtained by element-wise multiplication. Projecting infrared light features Attention feature map of visible light branch Element-wise multiplication yields the second cross-enhancement feature. The first and second cross-enhancement features are concatenated along the channel dimension to obtain the fused modal interaction enhancement features. .

[0028] Furthermore, the Sobel operator is used to explicitly extract gradient information in the horizontal and vertical directions, and convolution is combined to extract local texture structure, enhance edge and texture information, and obtain the final gradient texture enhancement features, including:

[0029] Enhanced features through fused modal interactions As input, a Sobelx convolution kernel is used. With Sobely convolution kernel Extract modal interaction enhancement features separately horizontal gradient with vertical gradient ∗ represents the convolution operation; it calculates the gradient magnitude. , where ϵ is the minimum value; for modal interaction enhancement features Feature map and gradient magnitude after 3×3 convolution After channel concatenation, 1×1 convolution is used for channel fusion and compression to obtain the final gradient texture enhancement feature. .

[0030] Furthermore, the processing procedure of the CFGM cross-scale feature gating adjustment module is as follows:

[0031] In the channel dimension, the C1 channels of the first input feature map and the C2 channels of the second input feature map are concatenated to generate intermediate features. For intermediate features The attention weight vector is obtained by applying the Softmax function for activation. The attention weight vector Decomposed into first attention weights based on the number of input channels. Second attention weight Utilizing the first attention weight Second attention weight For feature maps respectively and feature map By performing channel-by-channel weighting, the first weighted feature is obtained. and the first weighted feature For the first weighted feature Second weighted features Apply DropPath to randomly mask certain channels and then apply DropPath to the remaining channels. Scaling involves element-wise concatenation of two features that have undergone the DropPath operation to obtain the output of the cross-scale feature gating adjustment module. .

[0032] Furthermore, the implementation method of the multi-scale semantics and boundary enhancement module is as follows:

[0033] The input feature map is processed through two parallel branches: channel attention and spatial attention. Perform local enhancement to obtain channel attention feature maps. Spatial attention feature map Channel attention The importance of channels is modeled sequentially using global average pooling and 1×1 convolution, followed by spatial attention. We employ depthwise separable convolutions to reduce computational complexity while preserving local spatial modeling capabilities; and we integrate channel attention feature maps. Spatial attention feature map Element-wise multiplication yields locally enhanced features ;

[0034] Local enhancement features The channel dimension is divided into k independent attention heads, each head corresponding to a sub-feature representation. Within each head, a convolution operation is performed to generate key and value feature maps. Subsequently, the key and value are divided into Nb non-overlapping spatial sub-blocks to obtain block-level key / value feature maps.

[0035] Remove the Q-projection branch and directly calculate the intra-block normalized weights based on the block-level key features. , And use this weight to sum up all feature points within the block. Then, information is aggregated for each block. Based on block-level features, low-rank constraints are introduced, and global information is reconstructed through two layers of linear mapping. The global compensation features are obtained. , fusion feature map Features of compensation The final output of the multi-scale semantic and boundary enhancement module is obtained. .

[0036] Furthermore, the training process of the YOLO11-based dual-branch object detection network includes two main parts: the object detection path and the auxiliary contrastive learning path. The training process is as follows:

[0037] t1. Obtain the multimodal target detection dataset for power equipment, and after preprocessing, divide it into training set, validation set and test set;

[0038] t2. On the main road target detection path:

[0039] Using the training set as input, the main path of the YOLO11-based dual-branch object detection network outputs features after forward propagation. The bounding box location and class probability predicted by the detection head are used to calculate the basic detection loss of the model. Including bounding box regression loss Classification loss and distributed focus loss ;

[0040] t3. In the auxiliary comparative learning path:

[0041] t31, Features Flattened into positional sequences N is the total number of spatial locations in the feature map, and is determined by the projection head. Mapping to contrastive embedding space: , The dimension of the real number space is the dimension of the contrast-embedded space;

[0042] t32. Let the total number of positions within a batch be... , Batch size, embedding as , tag as For each anchor point The positive sample set is constructed as follows: , anchor point The set of positive samples contains all other samples of the same category as i. For sample labels;

[0043] t33, Calculate anchor points and Cosine similarity between positions: in, For temperature parameters;

[0044] t34, The contrast loss is obtained as follows:

[0045] ;

[0046] t35. Calculate the total loss: , To compensate for the loss weights, the model parameters are updated through backpropagation.

[0047] The beneficial effects of this invention are:

[0048] By introducing an Infrared Guided Fusion (IGF) module, this invention constructs an asymmetric cross-modal interaction mechanism, utilizing the stable thermal response of infrared modes under complex lighting conditions to extract joint spatial and channel guidance information. This solves the problem of "averaging" complementary information caused by simple splicing or independent weighting in existing technologies, as well as the technical shortcomings of insufficient visible light discrimination capability in low-light and occluded scenes. Compared to the missed detections or fragmented results that are prone to occur in single-modal methods, IGF specifically enhances visible light features in the significant infrared region, generating more complete and accurate target localization results.

[0049] By introducing a cross-scale feature gating module (CFGM), a dynamic interaction pathway combining bottom-up and top-down approaches is constructed at the network neck. This addresses the challenges of insufficient interaction between high-order semantics and low-order details in existing multi-scale fusion methods, and the difficulty in distinguishing effective features from redundant information using fixed weighting methods. CFGM utilizes a learnable gating mechanism to modulate cross-scale features, effectively aggregating previously scattered feature responses and mitigating the "attention dilution" phenomenon caused by scale variations. Building upon the initial contours provided by IGF, CFGM further enhances the model's multi-scale perception capability for power equipment (such as tiny pins and large towers), making the localization results more robust.

[0050] By introducing a multi-head, multi-block semantic enhancement module (M²BE) and combining it with auxiliary contrastive supervised learning, a unified modeling of local saliency and global context is achieved. This addresses the problems of high computational cost in global modeling of high-level features, weak ability to distinguish fine-grained structures, and semantic confusion between similar categories of devices (such as insulators of different models). The M²BE module integrates fragmented responses into clearly defined target regions through channel / spatial parallel attention and low-rank global compensation. Contrastive learning enhances the compactness of intra-class features and expands the inter-class margin, significantly improving the boundary discrimination and semantic consistency between adjacent targets. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a framework diagram of the YOLO11-based dual-branch target detection network of the present invention.

[0053] Figure 2 This is a structural diagram of the Infrared-Guided Fusion (IGF) of the present invention.

[0054] Figure 3 This is a structural diagram of the (Asymmetric Infrared-Guided block, AIG) module of the present invention.

[0055] Figure 4 This is a structural diagram of the Cross-Scale Feature Gating Module (CFGM) of the present invention.

[0056] Figure 5This is a structural diagram of the Multi-Head Block Attention (MHBA) of the present invention.

[0057] Figure 6 The figure shows the comparison results of this invention with other models.

[0058] Figure 7 The figure shows the ablation experiment results of this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] This invention is based on a dual-path feature extraction structure, using infrared and visible light images as inputs. The core modules include: an asymmetric guided edge-aware fusion module (IGF), which leverages the sensitivity of infrared thermal imaging to high-temperature targets to guide the extraction of visible light features; a cross-scale feature gating adjustment module (CFGM), used to achieve information exchange between feature maps of different resolutions; and a multi-scale semantic and boundary enhancement module (M²BE), used to improve discriminative power through supervised contrastive learning and boundary refinement. The structure is as follows: Figure 1 As shown, infrared and visible light images are first fed into the YOLO11 backbone for feature extraction. Unlike the original YOLO11, this invention introduces asymmetric guided edge-aware fusion modules (IGF) at three scales of the backbone, fusing infrared and visible light features from the corresponding scales into the IGF. The IGF fuses multimodal data through an infrared-dominated asymmetric guided mechanism while explicitly preserving edge structure information. The fused features from the first two scales are directly fed into the prediction branch, while the fusion result from the last scale serves as the input to the SPPF. At the network neck, this invention designs a cross-scale feature gating module (CFGM) to replace the traditional stitching method, achieving more efficient feature collaboration through an adaptive gating mechanism. Finally, a multi-head, multi-block semantic enhancement module (M²BE) is introduced before each detection head. This module combines contrastive learning-assisted loss to enhance the discriminability of target features without increasing inference overhead.

[0061] A multimodal power equipment detection method based on infrared guidance and contrast enhancement, the specific implementation steps of which include:

[0062] S1: Acquire visible light and infrared images of the substation's power equipment and perform preprocessing.

[0063] S11: Multimodal data acquisition and spatial alignment:

[0064] Visible light and infrared thermal images of the area to be detected are simultaneously acquired using a coaxial optical system. This coaxial optical system utilizes a beam splitter to ensure that visible and infrared light share the same optical axis, guaranteeing pixel-level spatial alignment between the acquired RGB image (three-channel color image) and IR image (single-channel grayscale thermal image) at the hardware level. The acquired images are then read into matrix arrays of size H×W.

[0065] S12: Proportional fill scaling:

[0066] To maintain the aspect ratio of the original images and avoid target distortion, Letterbox-based scaling is used to uniformly adjust both images to the model's expected input size (e.g., 640×640).

[0067] Calculate scaling ratio: Scale proportionally based on the ratio of the long side of the input size to the original size;

[0068] Symmetrical fill: Symmetrical fill is applied to both sides of the scaled shorter side. The RGB image channels are filled with neutral gray values ​​[128, 128, 128], and the IR image channels are filled with a scalar value of 128.

[0069] Parameter recording: Synchronously records the current scaling factor and the coordinate offset generated by the filling, which is used to reverse map the detection result (Bounding Box) back to the original image coordinate system after inference.

[0070] S13: Pixel value normalization and feature calibration:

[0071] Numerical quantization is performed on the scaled image to eliminate the effects of illumination fluctuations and differences in device imaging.

[0072] Normalization: Divide the pixel values ​​of the RGB and IR images by 255.0 respectively, and map their values ​​to the [0,1] interval;

[0073] Mean and standard deviation correction (optional supplement): If the model training includes a standardization step, then the tensor is processed by Input=(Value -Mean) / Std to further accelerate model convergence and improve inference stability.

[0074] S14: Multimodal Tensor Construction and Device Distribution:

[0075] The preprocessed multimodal features are re-dimensioned to construct an input format that meets the requirements of deep learning models:

[0076] Channel splicing: The normalized RGB three-channel tensor and the IR single-channel tensor are spliced ​​horizontally along the channel dimension to synthesize a four-channel composite input tensor with the shape (1,4,640,640).

[0077] Data format conversion: The composite tensor is uniformly converted into float32 single-precision floating-point format and loaded from main memory (CPU) into the video memory of the inference computing device (such as GPU or NPU) to complete the input preparation for forward propagation.

[0078] S2: Input the preprocessed visible light image and infrared image into a YOLO11-based dual-branch target detection network to perform fault detection of power equipment;

[0079] The YOLO11-based dual-branch target detection network, such as Figure 1 As shown, the system includes a dual-branch backbone network, a neck network, and a detection head connected in sequence. The dual-branch backbone network includes parallel visible light image feature extraction branches and infrared image feature extraction branches. The neck network includes a bottom-up path and a top-down path. An asymmetric guided edge-aware fusion module (IGF) is introduced in the bottom-up path to achieve infrared-guided fusion, modal interaction enhancement, and edge texture optimization. A cross-scale feature gating adjustment module (CFGM) is introduced in the top-down path to achieve cross-scale feature fusion. A multi-scale semantic and boundary enhancement module (M²BE) is introduced in the detection head to enhance the discriminability of target features.

[0080] In this embodiment, both the visible light image feature extraction branch and the infrared image feature extraction branch use the original YOLO11 backbone network, such as... Figure 1 As shown.

[0081] The visible light branch extracts features at three scales from large to small, and the infrared branch does the same. For small scales, the input to the IGF is obtained by concatenating small-scale infrared and visible light features; for medium scales, the input to the IGF is obtained by concatenating medium-scale infrared and visible light features and the output of the small-scale IGF module; for large scales, the input to the IGF is obtained by concatenating large-scale infrared and visible light features and the output of the medium-scale IGF module. In the small-scale IGF module, the input is first processed by dividing the concatenated features into visible light and infrared light features in sequence; for the AIG module, a judgment procedure is first executed. If there is no input from the previous scale, the AIG module only processes infrared and visible light features; in the medium-scale IGF module, it receives infrared and visible light features from the medium scale, as well as the features fused from the small-scale IGF module, and needs to be downsampled to the same resolution as the medium-scale features. At this time, the input to the AIG is infrared and visible light features and the fused features from the large scale. The processing flow for large scales is the same as for medium scales.

[0082] In the embodiments of this application, such as Figure 1 As shown, the bottom-up path from the bottom network to the top network includes a first asymmetric guided edge perception fusion module (IGF), a second asymmetric guided edge perception fusion module (IGF), and a third asymmetric guided edge perception fusion module (IGF). The first asymmetric guided edge perception fusion module (IGF), the second asymmetric guided edge perception fusion module (IGF), and the third asymmetric guided edge perception fusion module (IGF) take the splicing results of the large, medium, and small scale output features of the visible light image feature extraction branch and the infrared image feature extraction branch, respectively, as input. The third asymmetric guided edge perception fusion module (IGF) is connected to the SPPF module and the C2PSA module in sequence.

[0083] In this embodiment, the asymmetric-guided edge perception fusion module (IGF) does not use the traditional symmetric addition. Instead, it uses the infrared branch as a priori mask and extracts thermal saliency weights from the infrared feature map through spatial guidance to identify the core heat-generating area of ​​the power equipment. Through feature injection, the weights are applied to the visible light branch, enabling the model to accurately locate the structural details of the target and eliminate background interference in low light or cluttered background environments.

[0084] Specifically, to effectively fuse visible light and infrared light features and enhance structural information, this invention designs an asymmetric guided edge-aware fusion module (IGF), such as... Figure 2As shown, the following modules are executed sequentially: the Asymmetric Infrared-Guided block (AIG), the Cross-Modal Interaction Block (CMI), and the Edge-Texture Enhancement block (ETE), which respectively implement infrared-guided fusion, cross-modal interaction enhancement, and edge texture optimization. Figure 2 As shown, the Asymmetric Infrared Guiding Module (AIG) is primarily infrared-mode guided, using thermal signals to guide visible light features, achieving initial cross-modal fusion. The AIG highlights the semantic information of infrared targets and provides a reliable fusion foundation for subsequent feature interactions. The fused features are then input into the Modal Interaction Module (CMI), where intermodal operations enable complementary enhancement of visible and infrared light at both semantic and structural levels. The CMI utilizes RGB color and texture information to highlight key thermal targets in infrared images while also enhancing occluded or poorly lit areas in visible light images. Finally, the output features enter the Edge and Texture Extraction Module (ETE), which explicitly extracts horizontal and vertical gradient information using the Sobel operator and combines convolutional branches to extract local texture structures, further enhancing edge and texture information. This module effectively optimizes the performance of the fused features in terms of structural clarity and local detail.

[0085] Regarding the Asymmetric Infrared Guidance Module (AIG), such as Figure 3 As shown. Most studies on visible and infrared light fusion treat the two as equivalent, typically involving simple feature stitching. However, the inventors of this application have discovered that the infrared mode is more important in power equipment inspection tasks, especially in low-light scenarios. To fully utilize this advantageous mode, this invention designs an infrared-guided fusion method, such as... Figure 3 The asymmetric infrared guidance module (AIG) shown.

[0086] The Asymmetric Infrared Guiding Module (AIG) takes visible light and infrared image features of the same scale, along with the downsampling results from the previous layer's Asymmetric Infrared Guiding Module (AIG), as input. Primarily driven by the infrared mode, it guides the visible light features through thermal signals, achieving preliminary cross-modal fusion of visible light and infrared image features and obtaining initial cross-modal fused features. This module can highlight the semantic information of infrared targets and provide a reliable fusion foundation for subsequent feature interaction.

[0087] Specifically, using the infrared mode as the primary modality, visible light features are guided by thermal signals to achieve preliminary cross-modal fusion of visible light and infrared image features, obtaining preliminary cross-modal fusion features, including:

[0088] Taking the processing of the second asymmetric guided edge perception fusion module (IGF) as an example, since the features output by the first asymmetric infrared guided module (AIG) contain global contextual information under a larger receptive field, they can be used as supplementary features across scales. Therefore, the present invention first uses the features output from the first asymmetric infrared guidance module (AIG) The image is mapped to the same size as the input of the second asymmetric guided edge-aware fusion module (IGF) through a 1×1 convolution (i.e., downsampling), and is compared with the input of the second asymmetric guided edge-aware fusion module (IGF) (i.e., mesoscale visible light features). and infrared features These are used as inputs for the cross-scale supplementary branch, the visible light branch, and the infrared branch, respectively. When there are no cross-scale supplementary features from the previous layer, the infrared and visible light features are directly fused.

[0089] Supplementary features across scales Both the visible light feature φ and the infrared feature φ are subjected to X-attention calculations and Y-attention calculations, respectively, to obtain the X-axis attention-enhanced features with cross-scale supplementary branches. and Y-axis attention enhancement features X-axis attention enhancement features of visible light branch and Y-axis attention enhancement features X-axis attention enhancement features of the infrared branch and Y-axis attention enhancement features This invention introduces X-axis attention and Y-axis attention to model the spatial structure of features in the horizontal and vertical directions respectively, thereby capturing the correlation between spatial location and channel.

[0090] The calculation methods for X-axis attention and Y-axis attention are as follows:

[0091] ;

[0092] ;

[0093] in, , , . and They represent the width respectively and height The position represents the feature vector of i or j. Compared to self-attention, X / Y attention effectively explores the relationship between positional and channel information.

[0094] X-axis attention enhancement features for cross-scale supplementary branches X-axis attention enhancement features compared to the visible light branch X-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the horizontal joint representation of the cross-scale supplementary branch and the visible light branch fused together. Horizontal joint representation of cross-scale supplementary branch and infrared branch fusion ; Enhance Y-axis attention features for cross-scale supplementary branches Y-axis attention enhancement features compared to the visible light branch Y-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the joint vertical representation of the fusion of the cross-scale supplementary branch and the visible light branch. Vertical joint representation of cross-scale supplementary branch and infrared branch fusion ; Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and visible light branches are obtained by stitching along the channel. Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and infrared branches are obtained by stitching along the channel. ;

[0095] High-width fusion features High-width fusion features Convolution, batch normalization (BN), and ReLU activation operations are performed sequentially to generate activation representations of the fused spatial structure and channel saliency, including high-width fused features. The first activation representation and the high-width fused features are obtained. The resulting second activation representation;

[0096] The first and second activation representations are split into height and width branches along the channel dimension, respectively, and generated by two independent 1×1 convolutions. The first height feature map is obtained from the first activation representation. First width feature map And the second height feature map is obtained from the second activation representation. Second width feature map ;

[0097] Visible light characteristics and complementary features across scales Element-by-element addition and merging with the first height feature map First width feature map Element-wise multiplication yields the output characteristics of the visible light branch. Infrared features and complementary features across scales Element-by-element addition and merging with the second height feature map Second width feature map The output characteristics of the infrared branch are obtained by element-wise multiplication. .

[0098] Output characteristics of the infrared branch After further processing through pooling, convolution, batch normalization (BN), and ReLU operations, the input is fed into the attention mask generator to obtain the channel attention weights. and filter attention weights Channel attention weights Output characteristics of the infrared branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the infrared branch is obtained after multiplication. Similarly, channel attention weights Output characteristics of visible light branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the visible light branch is obtained after multiplication. Attention feature map of infrared branch Attention feature map of visible light branch This serves as the initial cross-modal fusion feature.

[0099] Attention feature map of infrared branch For example, the calculation formula can be expressed as: Here, the present invention defines a method that incorporates infrared attention weights. Parameterized mapping of depthwise separable convolutions This mapping operates on visible light and infrared features to achieve cross-modal guidance based on infrared semantics.

[0100] Channel attention The purpose is to characterize the correlation between each infrared feature channel and the semantics of the thermal target. Because infrared images have a natural advantage in terms of thermal radiation intensity and target region saliency, different channels respond differently to thermal structure information. By learning this difference, This approach enhances the weights of channels containing key thermal cues while suppressing channels irrelevant or redundant to the target, making the fusion process more selective and targeted along the channel dimension. When this attention is applied to visible and infrared features, the model can proactively emphasize feature channels with strong thermal semantic responses during the fusion stage, thereby achieving a significant enhancement of cross-modal information. In contrast, filter attention... The aim is to dynamically adjust the response of convolutional filters in the visible and infrared branches based on global thermal semantics. Different filters often encode different spatial structures and semantic patterns, such as detailed textures, edge orientations, or large-scale object responses. By selectively enhancing or suppressing these filters, It can adjust the participation level of each convolutional kernel in the fusion process, allowing the model to prioritize structural responses that are consistent with the spatial distribution of infrared targets and are more favorable for detection. (Filter attention) This not only improves the discernibility of illuminated areas in the visible light branch, but also allows the infrared branch to better preserve spatial details while maintaining thermal semantics.

[0101] Regarding the Modal Interaction Module (CMI), such as Figure 2 As shown, the Modal Interaction Module (CMI) takes preliminary cross-modal fusion features as input and, through intermodal interaction operations, enables visible and infrared light to achieve complementary enhancement at the semantic and structural levels, thereby obtaining modal interaction enhancement features. This module can utilize RGB color and texture information to highlight key thermal targets in infrared images, while also enhancing occluded or poorly lit areas in visible light images.

[0102] Specifically, through intermodal interaction operations, visible light and infrared light achieve complementary enhancement at the semantic and structural levels, thereby obtaining modal interaction enhancement features, including:

[0103] Attention feature map of infrared branch Attention feature map of visible light branch The attention feature map of the visible light branch is used as input. Attention feature map of infrared branch Perform 1×1 convolution operations separately to obtain visible light feature projection maps used for modulating infrared features. and infrared light feature projection map used to guide visible light features ;

[0104] Projecting visible light features Attention feature map with infrared branch The first cross-enhancing feature is obtained by element-wise multiplication. This allows infrared features to be enhanced at locations with strong visible light response, thus projecting the infrared light features into a map. Attention feature map of visible light branch Element-wise multiplication yields the second cross-enhancement feature. The first and second cross-enhancement features are concatenated along the channel dimension to obtain the fused modal interaction enhancement features. In this way, the model can use RGB color and texture information to highlight key thermal target areas in IR images, while also using IR thermal radiation information to enhance occluded or poorly lit targets in RGB images.

[0105] Regarding the Edge and Texture Extraction (ETE) module, such as Figure 2 As shown in the diagram, an edge and texture extraction module (ETE) was designed to compensate for potential structural information loss during feature extraction. The ETE takes modal interaction enhancement features as input, explicitly extracts horizontal and vertical gradient information using the Sobel operator, and combines this with convolution to extract local texture structures, further enhancing edge and texture information to obtain the final gradient texture enhancement features. This module effectively optimizes the performance of the fused features in terms of structural clarity and local detail.

[0106] Specifically, the enhanced features are based on the fused modal interactions. As input, Sobelx and Sobely convolutional kernels are used to extract modal interaction enhancement features, respectively. horizontal gradient with vertical gradient ∗ represents the convolution operation; then, the gradient magnitude is calculated. To form a feature map that represents the edge intensity, a minimum value ϵ, usually taken as 10⁻⁶, is added during calculation for numerical stability to avoid numerical instability or zero denominator; the feature is enhanced by modal interaction. A 3×3 convolution is performed to extract stable texture and structural information within the local neighborhood. To compensate for the noise generated by the Sobel gradient in complex texture regions, an additional 3×3 convolution branch is introduced. This branch extracts stable texture and structural information within the local neighborhood through standard convolution, thus suppressing gradient noise while preserving details. The feature map after the subsequent 3×3 convolution is then compared with the feature map. After channel concatenation, 1×1 convolution is used for channel fusion and compression to obtain the final gradient texture enhancement feature. .

[0107] The Sobelx convolution kernel and the Sobely convolution kernel are defined as follows:

[0108] ;

[0109] in, Sobelx convolution kernel, It is a Sobely convolution kernel.

[0110] In this embodiment of the application, the top-down path is as follows: Figure 1 As shown, the network from top to bottom includes, in sequence, a first Upsample module, a first cross-scale feature gating adjustment module (CFGM), a first C3K2 module, a second Upsample module, a second cross-scale feature gating adjustment module (CFGM), a second C3K2 module, a first Conv module, a third cross-scale feature gating adjustment module (CFGM), a third C3K2 module, a second Conv module, a fourth cross-scale feature gating adjustment module (CFGM), and a fourth C3K2 module. The output of the C2PSA module is connected to the first Upsample module and the fourth cross-scale feature gating adjustment module (CFGM), respectively. The output of the first asymmetric guided edge perception fusion module (IGF) is connected to the second cross-scale feature gating adjustment module (CFGM), the output of the second asymmetric guided edge perception fusion module (IGF) is connected to the first cross-scale feature gating adjustment module (CFGM), and the output of the first C3K2 module is also connected to the third cross-scale feature gating adjustment module (CFGM).

[0111] High-order and low-order feature maps carry semantic information at different levels. Effectively aggregating representations from multi-scale feature maps is crucial for enhancing the semantic discrimination ability of difficult-to-distinguish targets. The constructed top-down and bottom-up bidirectional paths enable the full integration of high-level semantic information and low-level detailed information.

[0112] First, the bottom-up path begins with high-resolution, large-scale feature maps (the lowest layer of the network), which are initially fed into the IGF module for fusion. The IGF output is then distributed to two branches: on one hand, downsampling is used to pass detailed information to higher-level IGF modules; on the other hand, it is fused with higher-level features obtained through upsampling via the Cross-Scale Feature Gated Adjustment (CFGM) module, integrating information from different levels. This approach is repeated on mesoscale and small-scale features, effectively incorporating the rich spatial details of the large-scale feature maps into features at all levels.

[0113] Secondly, the top-down path begins with a small-scale feature map (the top layer), whose IGF module output is first fed into the SPPF and C2PSA modules for processing. Subsequently, it is upsampled to a medium scale and fused with the medium-scale features obtained from the bottom-up path (i.e., the CFGM output). This is then processed by the C3K2 module to form enhanced medium-scale features. Finally, it is upsampled again to a large scale, fused with the large-scale enhanced features obtained from the bottom-up path, and processed by the C3K2 module to obtain high-resolution and semantically enhanced large-scale features.

[0114] In this embodiment, to address the issue of large scale differences in power equipment, a Cross-Scale Feature Gated Adjustment Module (CFGM) is proposed to optimize the feature fusion process at the network neckline. The CFGM replaces the traditional splicing method, achieving more efficient feature collaboration through an adaptive gating mechanism. This adaptive gating mechanism introduces learnable parameters (gated units) to adaptively adjust the information flow at different scales based on the size of the input features. For example, when detecting small-scale insulators, the gating mechanism automatically increases the weight of shallow, high-resolution features. The implementation method of the Cross-Scale Feature Gated Adjustment Module (CFGM) is as follows:

[0115] Taking the first cross-scale feature gating adjustment module (CFGM) as an example, such as Figure 4 As shown, the output (small scale) of the third asymmetric guided edge-aware fusion module (IGF) is sequentially fed into the SPPF and C2PSA modules for processing, and then upsampled to the medium scale by the first Upsample module, which is represented as a feature map. The output feature map of the second asymmetric guided edge-aware fusion module (IGF) is represented as follows: , These correspond to small-scale, medium-scale, and large-scale, respectively. In this example... In terms of channel dimension The C1 channels and The C2 channels are spliced ​​to generate intermediate features. For intermediate features The attention weight vector is obtained by applying the Softmax function for activation. The attention weight vector Decomposed into first attention weights based on the number of input channels. Second attention weight Utilizing the first attention weight Second attention weight For feature maps respectively and feature map By performing channel-by-channel weighting, the first weighted feature is obtained. and the first weighted feature For the first weighted feature Second weighted features Apply DropPath to randomly mask certain channels and then apply DropPath to the remaining channels. Scaling (p = 0.1) suppresses overfitting and improves generalization. The two features processed by DropPath are then fused element-wise to obtain the output of the first cross-scale feature gating module (CFGM):

[0116] .

[0117] Compared to traditional BiFPN, CFGM improves upon the multi-scale feature fusion strategy. BiFPN typically achieves multi-scale information fusion through weighted summation of cross-layer features (using fixed or simple learnable scalar weights). CFGM introduces a set of learnable weights to adaptively evaluate the importance of each input feature channel and multiplies it element-wise with the original features, thereby fully mining and utilizing the discriminative information between different channels to achieve more accurate feature representation.

[0118] In this embodiment of the application, in the detection head, the output of the second C3K2 module is connected to the first multi-scale semantic and boundary enhancement module (M²BE), and the output of the first multi-scale semantic and boundary enhancement module (M²BE) is connected to the first Detect module; the output of the third C3K2 module is connected to the second multi-scale semantic and boundary enhancement module (M²BE), and the output of the second multi-scale semantic and boundary enhancement module (M²BE) is connected to the first Detect module; the output of the fourth C3K2 module is connected to the third multi-scale semantic and boundary enhancement module (M²BE), and the output of the third multi-scale semantic and boundary enhancement module (M²BE) is connected to the third Detect module.

[0119] Regarding the Multi-Scale Semantics and Boundary Enhancement Module (M²BE), such as Figure 5As shown, after passing through the IGF and CFGM modules, the feature map has fully integrated multimodal information and contains rich local contextual structures. Compared to the Backbone stage, modeling the global relationship between the target and the background in the high-level feature space is more effective at this stage. On the one hand, global contextual information can characterize dependencies across spatial locations, thereby suppressing redundant backgrounds and enhancing the discriminative power of the target region. On the other hand, due to the high similarity in appearance of some power equipment categories (such as voltage transformers and current transformers), similar samples are easily scattered in the feature space, and dissimilar samples are not sufficiently spaced, thus limiting detection and recognition performance. Based on this, this invention proposes the Multi-Head Multi-Block Enhancement Module (M²BE), which further improves the discriminative power of features through structured global modeling and comparative constraints during the training stage without changing the detection backbone structure and inference path.

[0120] Specifically, the Multi-Scale Semantic and Boundary Enhancement Module (M²BE) utilizes a multi-head attention mechanism to capture the global topology of the device, enhancing the understanding of the overall attributes of complex devices. Combined with the auxiliary loss of contrastive learning, it enhances the discriminability of target features without increasing inference overhead. Supervised Contrastive Learning introduces a contrastive loss function during training. By "bringing closer" similar targets and "distanceing" dissimilar targets in the feature space (e.g., forcibly separating the features of current transformers from those of voltage transformers), it improves discrimination accuracy at the feature layer. During training, the output of M²BE consists of features plus the supervised contrastive loss. The supervised contrastive loss is calculated from the features output by M²BE using the supervised contrastive loss function. The M²BE output features are also simultaneously fed into the basic detection loss calculation function; the total loss consists of the basic loss and the supervised contrastive loss. During inference, the supervised contrastive loss is removed, retaining only the basic loss.

[0121] The implementation method of the multi-scale semantic and boundary enhancement module (M²BE) is as follows:

[0122] Taking the first multi-scale semantic and boundary enhancement module (M²BE) as an example, such as Figure 5 As shown, the input is the output feature map of the second C3K2 module, represented as a feature map. The feature map is processed through two parallel branches: channel attention and spatial attention. Perform local enhancement to obtain channel attention feature maps. Spatial attention feature map Channel attention The importance of channels is modeled sequentially using global average pooling and 1×1 convolution, followed by spatial attention. We employ depthwise separable convolutions to reduce computational complexity while preserving local spatial modeling capabilities; and we integrate channel attention feature maps. Spatial attention feature map Element-wise multiplication yields locally enhanced features The system highlights significant regions and suppresses irrelevant backgrounds in both pixel and channel dimensions, and then feeds them into the Multi-Head BlockAttention (MHBA) module for efficient global relationship modeling.

[0123] Guided by channel and spatial attention, the MHBA module prioritizes discriminative regions when modeling long-range dependencies, thereby reducing the interference of background noise and complex textures on global semantic modeling. Within the MHBA module, local enhancement features are incorporated. The system is divided into k independent attention heads along the channel dimension, each corresponding to a sub-feature representation. Within each head, key and value feature maps are generated through convolution operations. Subsequently, the key and value maps are divided into Nb non-overlapping spatial sub-blocks, resulting in block-level key / value feature maps. If the size of the key / value feature map for each head is H×W, then the sub-block size... This block-based strategy restricts attention computation to local regions, significantly reducing computational complexity while improving the ability to model fine-grained structures.

[0124] Inspired by the "softmax weighting based on activation values" in the SoftPool method, this paper simplifies the traditional self-attention mechanism. In Transformer, Q and K represent finding global tokens for relationship modeling. However, in MHBA, each token comes from the same spatial block, and the semantic distributions of Q and K are highly consistent, meaning the similarity of QKᵀ contains relatively little information. The activation intensity of the key features is sufficient to effectively characterize the importance of different positions. Therefore, MHBA removes the Query projection branch and directly calculates intra-block normalized weights based on the key features. Specifically, the Q projection branch is removed, and intra-block normalized weights are directly calculated based on the block-level key features. , And use this weight to sum up all feature points within the block. Then, information is aggregated for each block. This process enables modules to capture adaptive contextual dependencies within local regions, thereby enhancing their ability to model complex structures. To further model global dependencies across blocks and reduce redundant computation, this invention introduces low-rank constraints on top of block-level features, reconstructing global information through two layers of linear mapping to obtain globally compensated features. , fusion feature map Features of compensation The final output of the Multi-Scale Semantic and Boundary Enhancement Module (M²BE) is obtained. The following formula:

[0125] ;

[0126] ;

[0127] ;

[0128] in, Indicates the first Output features of the head, , Represents two linear layers, where For rank-constrained dimensions, The number of input channels for the first-level linear mapping. The number of output channels for the second linear mapping layer, and the number of output channels for the first linear layer. Used to divide dimensions Mapping to low-rank dimension The second linear layer Used to reduce low-rank dimensions Mapped to .

[0129] In this embodiment of the application, the training process of the YOLO11-based dual-branch object detection network is as follows:

[0130] Dataset Acquisition: The dataset used is a self-built multimodal target detection dataset for power equipment, derived from actual power facility inspection processes. During acquisition, visible light cameras and infrared thermal imaging devices were used to simultaneously image the same scene, acquiring paired visible light and infrared image data. This dataset contains 1764 pairs of spatially aligned visible light and infrared images. The acquired data covers various equipment categories, targets of different scales, and complex background conditions, including normal lighting, low light, and occlusion scenarios. All image samples were manually selected and precisely labeled, including the target category and its spatial location, for subsequent model training and testing.

[0131] The dataset is divided into training, validation, and test sets in a fixed ratio. Specifically, the data partitioning tool provided by scikit-learn (sklearn) is used to randomly partition the dataset using images as the basic sample unit. A stratified sampling strategy is employed to ensure that the distribution ratio of samples from each class is approximately consistent across the training, validation, and test sets, thereby avoiding class imbalance that could bias the model training and evaluation results. Simultaneously, the partitioning process ensures that there is no sample overlap between different subsets. Ultimately, the dataset can be partitioned into training, validation, and test sets in a 7:1:2 ratio. Alternatively, it can be partitioned into training and test sets in an 8:2 ratio to accommodate different experimental setups.

[0132] The training consists of two main parts: the main object detection path and the auxiliary contrastive learning path, as detailed below:

[0133] Main path object detection: Taking the training set as input, the main path outputs features after forward propagation through a YOLO11-based dual-branch object detection network. The bounding box position and class probability predicted by the detection head;

[0134] The basic detection loss of the computational model includes bounding box regression loss. Classification loss and distributed focus loss Distributed focus loss is used for fine-grained regression of bounding boxes and corresponds to the true labels, i.e. .

[0135] Auxiliary contrastive learning path: The auxiliary contrastive learning path of M²BE will output features from the main path. The supervised expression is calculated by mapping the projection head to the contrast embedding space and combining it with the corresponding labels. The calculation method is as follows:

[0136] First, the features Flattened into positional sequences N is the total number of spatial locations in the feature map, and is determined by the projection head. Mapping to contrastive embedding space:

[0137] ;

[0138] in, It is the dimension of the real number space, that is, the dimension of the contrast embedding space.

[0139] Furthermore, let the total number of positions within the batch be... , Batch size, embedding as , tag as For each anchor point The positive sample set is constructed as follows:

[0140] ;

[0141] in, anchor point The set of positive samples contains all other locations that are the same as category i. For location labels.

[0142] Further, calculate the anchor point. and Cosine similarity between positions:

[0143] ;

[0144] in, This refers to the temperature parameter.

[0145] Furthermore, the SupCon loss can be written as:

[0146] ;

[0147] in, Let represent the log-likelihood of anchor point i with respect to positive sample p, which measures the probability that positive samples are identified as similar.

[0148] Therefore, the total training loss is , The value is set to 0.05, and backpropagation is performed by calling `loss_total.backward()`. PyTorch computes all gradients that depend on the total loss based on the computation graph. Then, the model parameters are updated through backpropagation.

[0149] The training process consisted of 250 epochs, with the network performance evaluated using a validation set after each epoch. An early stopping mechanism was implemented, monitoring the bounding box regression loss, classification loss, distributed focus loss, mAP, and mAP50 on the validation set and recording their changes over 10 consecutive epochs. If no significant improvement was observed, the model was considered close to its optimal state, and training automatically terminated to prevent overfitting in subsequent iterations. After training, the final model weights retained were those corresponding to the optimal validation metrics. The parameter settings used during training are shown in Table 1.

[0150] The trained model was validated using a test set. To validate the model on the test set, images from the test set were input into the trained YOLO11 two-branch object detection network. The network output the predicted object location, class probability, and confidence score. Next, results with a confidence score below 0.3 in the predicted object boxes were removed, and non-maximum suppression was applied to the remaining overlapping predicted boxes to remove duplicate detections. Finally, the processed predicted boxes were matched with the ground truth annotations in the test set. The number of correct detections, false negatives, and misdetections for each class were counted, and mean precision (mAP), recall, and accuracy were calculated to obtain the model's actual performance on the test set, thus determining the final qualified model.

[0151] To verify the model's performance, a series of experiments were conducted on a self-built dataset. The model was also compared with other YOLO models and some advanced object detection algorithms, as shown in Table 2. Experimental results show that the method in this application achieves an mAP50 of 88.8% and an mAP of 71.7% on the RGB+IR multimodal dataset, significantly higher than the baseline models Faster R-CNN, YOLOv5, YOLOv8, and YOLOv11. Furthermore, it outperforms the single-modal model FFCA-YOLO and the multimodal model SuperYOLO, both designed for small object detection, fully demonstrating the performance advantages of the method in multimodal object detection tasks. Figure 6 As shown in (a), compared to YOLO11s and FFCA-YOLO, the method in this application enhances key channels and suppresses irrelevant channels in CFGM, making multi-scale feature representation more focused on the target region. This significantly improves the detection accuracy of the model in dense multi-target and target occlusion scenarios, and reduces the probability of missed detections and false detections. Figure 6 As shown in (b), YOLO11s and FFCA-YOLO suffer from severe false negatives in low-light conditions. The method in this application, however, primarily stems from the effective enhancement of the infrared-guided attention mechanism in the IGF module to the response of thermally significant regions, thereby strengthening the feature representation capability of the target region and significantly mitigating false negatives. Figure 6 As shown in (c), YOLO11s showed a false negative, while FFCA-YOLO showed a false positive. This application's method, by introducing the M²BE module, can effectively distinguish the boundary features of insulator targets in scenarios where targets overlap, thus significantly improving the ability to differentiate overlapping insulator targets.

[0152] This invention not only achieves the best performance balance in various types of power equipment inspection tasks, but also ensures the real-time requirements in engineering applications such as real-time inspection by UAVs due to the rational design of the dual-stream architecture and high-efficiency components, thus having extremely high practical application value.

[0153] like Figure 7The ablation experiments show that adding the IGF module itself significantly improves detection performance, increasing mAP50 and mAP by approximately 4.0% and 7.0%, respectively. Compared to single-modal input, IGF generates more complete and accurate target localization results, while single-modal methods often suffer from missed detections or fragmented detection results. This performance improvement mainly stems from the effective enhancement of the response of the infrared-guided attention mechanism to thermally salient regions, thereby strengthening the feature representation capability of the target region. Building upon the IGF feature fusion, the introduction of the CFGM module further enhances detection performance. IGF provides a preliminary target outline, while CFGM aggregates the originally scattered feature responses, effectively alleviating the attention dilution problem caused by scale inconsistency. Through adaptive adjustment of cross-scale information flow, CFGM enhances the model's multi-scale perception capability, resulting in more robust target localization results.

[0154] Ultimately, the model achieved optimal performance by incorporating the M²BE module along with IGF and CFGM, resulting in an mAP50 improvement to 88.8%. M²BE effectively integrates fragmented responses into continuous and clearly defined target regions, significantly enhancing the boundary discrimination ability between adjacent targets. This achievement is attributed to the introduction of auxiliary contrastive supervision, which further optimizes fine-grained structural representation and semantic consistency by enhancing intra-class feature compactness and expanding inter-class feature spacing.

[0155] Table 1 Training Parameter Settings

[0156]

[0157] Table 2 Comparison of the model with other methods

[0158]

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal power equipment detection method based on infrared guidance and contrast enhancement, characterized in that, The steps are as follows: S1: Acquire visible light and infrared images of the substation's power equipment and perform preprocessing; S2: Input the preprocessed visible light image and infrared image into a YOLO11-based dual-branch target detection network to perform fault detection of power equipment; The YOLO11-based dual-branch target detection network includes a dual-branch backbone network, a neck network, and a detection head connected in sequence. The dual-branch backbone network includes parallel visible light image feature extraction branches and infrared image feature extraction branches. The neck network includes a bottom-up path and a top-down path. An IGF asymmetric guided edge perception fusion module is introduced in the bottom-up path to achieve infrared guided fusion, modal interaction enhancement, and edge texture optimization. A CFGM cross-scale feature gating adjustment module is introduced in the top-down path to achieve cross-scale feature fusion. An M²BE multi-scale semantic and boundary enhancement module is introduced in the detection head to enhance the discriminability of target features.

2. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 2, characterized in that, The bottom-up path, from the bottom network to the top network, includes a first asymmetric guided edge perception fusion module, a second asymmetric guided edge perception fusion module, and a third asymmetric guided edge perception fusion module. The first asymmetric guided edge perception fusion module, the second asymmetric guided edge perception fusion module, and the third asymmetric guided edge perception fusion module take the output features of the visible light image feature extraction branch and the infrared image feature extraction branch at different scales as inputs, respectively. The third asymmetric guided edge perception fusion module is connected to the SPPF module and the C2PSA module in sequence. The top-down path, from the top-level network to the bottom-level network, sequentially includes a first Upsample module, a first cross-scale feature gating adjustment module, a first C3K2 module, a second Upsample module, a second cross-scale feature gating adjustment module, a second C3K2 module, a first Conv module, a third cross-scale feature gating adjustment module, a third C3K2 module, a second Conv module, a fourth cross-scale feature gating adjustment module, and a fourth C3K2 module. The output of the C2PSA module is connected to the first Upsample module and the fourth cross-scale feature gating adjustment module, respectively. The output of the first asymmetric guided edge perception fusion module is connected to the second cross-scale feature gating adjustment module, the output of the second asymmetric guided edge perception fusion module is connected to the first cross-scale feature gating adjustment module, and the output of the first C3K2 module is also connected to the third cross-scale feature gating adjustment module. In the detection head, the output of the second C3K2 module is connected to the first multi-scale semantic and boundary enhancement module, and the output of the first multi-scale semantic and boundary enhancement module is connected to the first Detect module; the output of the third C3K2 module is connected to the second multi-scale semantic and boundary enhancement module, and the output of the second multi-scale semantic and boundary enhancement module is connected to the first Detect module; the output of the fourth C3K2 module is connected to the third multi-scale semantic and boundary enhancement module, and the output of the third multi-scale semantic and boundary enhancement module is connected to the third Detect module.

3. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 2, characterized in that, The asymmetric-guided edge-aware fusion module includes sequentially executed modules: The AIG asymmetric infrared guidance module is used to take visible light image features and infrared image features of the same scale, as well as the downsampling results output by the previous asymmetric infrared guidance module, as input. With infrared mode as the main mode, it guides visible light features through thermal signals to achieve preliminary cross-modal fusion of visible light image features and infrared image features, and obtain preliminary cross-modal fusion features. The CMI modal interaction module is used to take preliminary cross-modal fusion features as input and, through intermodal interaction operations, enable visible light and infrared light to achieve complementary enhancement at the semantic and structural levels, thereby obtaining modal interaction enhancement features. The ETE edge and texture extraction module takes modal interaction enhancement features as input, explicitly extracts gradient information in the horizontal and vertical directions through the Sobel operator, and combines convolution to extract local texture structure, enhance edge and texture information, and obtain the final gradient texture enhancement features.

4. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 3, characterized in that, Using infrared mode as the primary modality, visible light features are guided by thermal signals to achieve preliminary cross-modal fusion of visible light and infrared image features, obtaining preliminary cross-modal fusion features, including: The features output by the asymmetric infrared guidance module of the previous layer are downsampled and mapped to the same size as the input of the edge perception fusion module of the current layer's asymmetric guidance, resulting in cross-scale supplementary features. And the visible light features input from the current layer's asymmetric-guided edge-aware fusion module. and infrared features These serve as inputs for the cross-scale supplementary branch, the visible light branch, and the infrared branch, respectively. By supplementing features across scales Visible light characteristics and infrared features X and Y axis attention calculations were performed separately to obtain the X and Y axis attention enhancement features of the cross-scale supplementary branch, visible light branch and infrared branch respectively; By coaxially fusing the X and Y axis attention enhancement features of the cross-scale supplementary branch with the X and Y axis attention enhancement features of the visible light branch and the infrared branch respectively, and then performing cross-axial fusion, the cross-scale-visible light height-width fused features are obtained. and cross-scale-infrared high-width fusion features ; High-width fusion features High-width fusion features Perform convolution, batch normalization, and ReLU activation operations sequentially to obtain the first activation representation and the second activation representation. The first and second activation representations are split into height and width branches along the channel dimension, respectively, and the first height feature map is generated by two independent 1×1 convolutions. First width feature map and the second height feature map Second width feature map ; Visible light characteristics and complementary features across scales Element-by-element addition and merging with the first height feature map First width feature map Element-wise multiplication yields the output characteristics of the visible light branch. Infrared features and complementary features across scales Element-by-element addition and merging with the second height feature map Second width feature map The output characteristics of the infrared branch are obtained by element-wise multiplication. . Output characteristics of the infrared branch After further processing through pooling, convolution, batch normalization (BN), and ReLU operations, the input is fed into the attention mask generator to obtain the channel attention weights. and filter attention weights Channel attention weights Output characteristics of the infrared branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the infrared branch is obtained after multiplication. Channel attention weights Output characteristics of visible light branch After element-wise multiplication, the result is subjected to a depthwise separable convolution, and then combined with the filter attention weights. The attention feature map of the visible light branch is obtained after multiplication. Attention feature map of infrared branch Attention feature map of visible light branch This serves as the initial cross-modal fusion feature.

5. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 4, characterized in that, Cross-axial fusion is performed by coaxially fusing the X and Y axis attention enhancement features of the cross-scale supplementary branch with the X and Y axis attention enhancement features of the visible light branch and the infrared branch, respectively. This includes: fusing the X axis attention enhancement features of the cross-scale supplementary branch... X-axis attention enhancement features compared to the visible light branch X-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the horizontal joint representation of the cross-scale supplementary branch and the visible light branch fused together. Horizontal joint representation of cross-scale supplementary branch and infrared branch fusion ; Enhance Y-axis attention features for cross-scale supplementary branches Y-axis attention enhancement features compared to the visible light branch Y-axis attention enhancement features of the infrared branch By splicing along the channel, we obtain the joint vertical representation of the fusion of the cross-scale supplementary branch and the visible light branch. Vertical joint representation of cross-scale supplementary branch and infrared branch fusion ; Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and visible light branches are obtained by stitching along the channel. Joint representation of the horizontal direction Combined representation with vertical direction High-width fusion features of cross-scale supplementary branches and infrared branches are obtained by stitching along the channel. .

6. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 5, characterized in that, Through intermodal interaction operations, visible light and infrared light achieve complementary enhancement at the semantic and structural levels, thereby obtaining modal interaction enhancement features, including: Attention feature map of infrared branch Attention feature map of visible light branch The attention feature map of the visible light branch is used as input. Attention feature map of infrared branch Perform 1×1 convolution operations separately to obtain visible light feature projection maps used for modulating infrared features. and infrared light feature projection map used to guide visible light features ; Projecting visible light features Attention feature map with infrared branch The first cross-enhancing feature is obtained by element-wise multiplication. Projecting infrared light features Attention feature map of visible light branch Element-wise multiplication yields the second cross-enhancement feature. The first and second cross-enhancement features are concatenated along the channel dimension to obtain the fused modal interaction enhancement features. .

7. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 6, characterized in that, The Sobel operator is used to explicitly extract gradient information in the horizontal and vertical directions, and convolution is combined to extract local texture structure, enhance edge and texture information, and obtain the final gradient texture enhancement features, including: Enhanced features through fused modal interactions As input, a Sobelx convolution kernel is used. With Sobely convolution kernel Extract modal interaction enhancement features separately horizontal gradient with vertical gradient ∗ represents the convolution operation; it calculates the gradient magnitude. , where ϵ is the minimum value; for modal interaction enhancement features Feature map and gradient magnitude after 3×3 convolution After channel concatenation, 1×1 convolution is used for channel fusion and compression to obtain the final gradient texture enhancement feature. .

8. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to any one of claims 2-7, characterized in that, The processing procedure of the CFGM cross-scale feature gating adjustment module is as follows: In the channel dimension, the C1 channels of the first input feature map and the C2 channels of the second input feature map are concatenated to generate intermediate features. For intermediate features The attention weight vector is obtained by applying the Softmax function for activation. The attention weight vector Decomposed into first attention weights based on the number of input channels. Second attention weight Utilizing the first attention weight Second attention weight For feature maps respectively and feature map By performing channel-by-channel weighting, the first weighted feature is obtained. and the first weighted feature For the first weighted feature Second weighted features Apply DropPath to randomly mask certain channels and then apply DropPath to the remaining channels. Scaling involves element-wise concatenation of two features that have undergone the DropPath operation to obtain the output of the cross-scale feature gating adjustment module. .

9. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 8, characterized in that, The implementation method of the multi-scale semantics and boundary enhancement module is as follows: The input feature map is processed through two parallel branches: channel attention and spatial attention. Perform local enhancement to obtain channel attention feature maps. Spatial attention feature map Channel attention The importance of channels is modeled sequentially using global average pooling and 1×1 convolution, followed by spatial attention. We employ depthwise separable convolutions to reduce computational complexity while preserving local spatial modeling capabilities; and we integrate channel attention feature maps. Spatial attention feature map Element-wise multiplication yields locally enhanced features ; Local enhancement features The channel dimension is divided into k independent attention heads, each head corresponding to a sub-feature representation. Within each head, a convolution operation is performed to generate key and value feature maps. Subsequently, the key and value are divided into Nb non-overlapping spatial sub-blocks to obtain block-level key / value feature maps. Remove the Q-projection branch and directly calculate the intra-block normalized weights based on the block-level key features. , And use this weight to sum up all feature points within the block. Then, information is aggregated for each block. Based on block-level features, low-rank constraints are introduced, and global information is reconstructed through two layers of linear mapping. The global compensation features are obtained. , fusion feature map Features of compensation The final output of the multi-scale semantic and boundary enhancement module is obtained. .

10. The multimodal power equipment detection method based on infrared guidance and contrast enhancement according to claim 9, characterized in that, The training process of the YOLO11-based dual-branch object detection network includes two main parts: the object detection path and the auxiliary contrastive learning path. The training process is as follows: t1. Obtain the multimodal target detection dataset for power equipment, and after preprocessing, divide it into training set, validation set and test set; t2. On the main road target detection path: Using the training set as input, the main path of the YOLO11-based dual-branch object detection network outputs features after forward propagation. The bounding box location and class probability predicted by the detection head are used to calculate the basic detection loss of the model. Including bounding box regression loss Classification loss and distributed focus loss ; t3. In the auxiliary comparative learning path: t31, Features Flattened into positional sequences N is the total number of spatial locations in the feature map, and is determined by the projection head. Mapping to contrastive embedding space: , The dimension of the real number space is the dimension of the contrast-embedded space; t32. Let the total number of positions within a batch be... , Batch size, embedding as , tag as For each anchor point The positive sample set is constructed as follows: , anchor point The set of positive samples contains all other samples of the same category as i. For sample labels; t33, Calculate anchor points and Cosine similarity between positions: in, For temperature parameters; t34, The contrast loss is obtained as follows: ； t35. Calculate the total loss: , To compensate for the loss weights, the model parameters are updated through backpropagation.

Images