An unmanned aerial vehicle power station inspection fault detection method and system based on multi-modal deep feature fusion

Abstract

The application discloses a kind of unmanned plane power station inspection fault detection methods based on improved YOLOv12 and multimodal deep feature fusion, it is first by to visible light image, infrared thermal imaging image and laser radar point cloud image are carried out image enhancement, pseudo color mapping etc. Preprocessing operation, improve the quality of input data and the intermodal fusibility;Second, construct spatial alignment and channel stacking mechanism, and introduce MMFM module in backbone feature extraction module, realize the saliency guidance and dynamic fusion of multi-modal feature, to enhance the detection robustness of model in low illumination, shielding and weak thermal signal Complex environment;Third, integrate MADSE module in backbone feature extraction module, construct multi-scale dynamic perception structure, effectively improve the synchronous identification ability to micro-fault and macro structure;While combining MGFP module, fine modeling the complementary relationship and interference suppression ability between modes, improve the expression effect of fusion feature;Finally, construct multi-label learning framework.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and intelligent operation and maintenance technology of power equipment. More specifically, it relates to a method and system for fault detection of power plant inspection by unmanned aerial vehicles based on multimodal deep feature fusion. Background Technology

[0002] With the rapid development of smart grids, digital energy, and intelligent operation and maintenance, intelligent perception and automated inspection of power plant equipment status have become one of the core directions for improving operation and maintenance efficiency and ensuring power supply security. In recent years, drones, due to their advantages such as high-altitude observation, flexibility, and remote control, have been widely used in power system inspection tasks, becoming an important inspection platform after manual and vehicle-mounted systems. Combined with the various types of sensors carried by drones, such as visible light cameras, infrared thermal imagers, and lidar, a multimodal perception data source has been formed, providing rich information support for vision-based power fault detection.

[0003] Currently, the mainstream power equipment fault detection methods include three types: (1) detection methods based on traditional image processing methods, such as edge extraction, region growing, and hot spot recognition. These methods use traditional image processing techniques such as image preprocessing, edge detection, or hot spot region segmentation to extract and analyze structural or temperature anomalies in power equipment images, thereby achieving fault identification. (2) methods based on single-modal deep learning models, such as YOLO and Faster Region-based Convolutional Neural Network (FasterR-CNN). These methods typically use visible light images (Red-Green-Blue image, RGB Image) to train target detection models and have a certain accuracy under standard lighting conditions. (3) methods that combine infrared thermal imaging images for thermal anomaly detection. These methods mainly extract high-temperature regions through threshold setting and region segmentation to assist in judging component faults.

[0004] However, the aforementioned existing methods all suffer from the following significant technical drawbacks:

[0005] (1) Detection methods based on traditional image processing rely on fixed rules (such as edge thresholds and hot spot region determination), have a simple processing flow and lack adaptability, making it difficult to accurately extract target areas in complex environments (such as low illumination, cluttered backgrounds, or partial occlusion). At the same time, the lack of multimodal fusion means that it is impossible to consider both equipment structural information and temperature characteristics at the same time, resulting in low accuracy of fault identification;

[0006] (2) Methods based on single-modal deep learning models typically use only RGB images for training, lacking the utilization of infrared or depth modalities, resulting in a significant decrease in detection performance in scenarios such as nighttime inspections, when thermal signal anomalies are not obvious, or when targets are occluded. At the same time, due to the fixed model structure and limited receptive field, it is impossible to simultaneously meet the recognition needs of large-scale equipment components and small fault areas;

[0007] (3) Methods for detecting thermal anomalies by combining infrared thermal imaging images often rely on simple threshold segmentation or region growing, lack deep semantic analysis capabilities, and are easily affected by thermal signal noise, resulting in false detections or missed detections. At the same time, when the thermal image is blurred, has a high dynamic range, or has infrared sensor deviation, there is a lack of supporting image enhancement and intermodal compensation mechanisms, resulting in poor stability of the detection results;

[0008] (4) The above three methods have poor universality and scalability, making it difficult to detect and classify multiple types of faults in the same model at the same time. They lack the ability to score the severity of faults and cannot meet the actual needs of "one network with multiple functions and graded response" in power inspection. Summary of the Invention

[0009] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for fault detection in power plant inspection using unmanned aerial vehicles (UAVs) based on multimodal deep feature fusion. Its purpose is to fully utilize the complementary information from visible light images, infrared thermal imaging images, and lidar point cloud images, combined with multi-scale dynamic feature modeling and scene perception mechanisms, to significantly improve the detection accuracy, stability, and real-time response capability of fault targets in complex power plant environments. This solves the technical problems of existing detection methods based on traditional image processing methods, such as a single processing flow, lack of adaptability, difficulty in accurately extracting target areas in complex environments, and low fault identification accuracy; the technical problems of existing methods based on single-modal deep learning models, such as significantly reduced detection performance in scenarios like nighttime inspections, inconspicuous thermal signal anomalies, or target occlusion, and inability to simultaneously address the identification needs of large-scale equipment components and small fault areas; the technical problems of existing methods combining infrared thermal imaging images for thermal anomaly detection, which are susceptible to thermal signal noise, leading to false detections or missed detections; the technical problems of poor stability of detection results due to the lack of supporting image enhancement and intermodal compensation mechanisms; and the technical problems of poor versatility and scalability of the three existing methods.

[0010] To achieve the above objectives, according to one aspect of the present invention, a method for fault detection in UAV power plant inspection based on improved YOLOv12 and multimodal deep feature fusion is provided, comprising:

[0011] (1) Obtain multimodal power plant image data from the power plant site using UAV, and preprocess the multimodal power plant image data to obtain preprocessed multimodal power plant image data;

[0012] (2) Input the preprocessed multimodal power plant image data obtained in step (1) into the pre-trained YOLOv12 improved model to obtain the final detection result.

[0013] Preferably, step (1) includes the following sub-steps:

[0014] (1-1) The UAV acquires multimodal power station data from the power station site and adds timestamps to the multimodal power station data to obtain multimodal power station data with added timestamps;

[0015] (1-2) Perform time-series alignment processing on the multimodal power plant data obtained in step (1-1) after adding timestamps to obtain multimodal power plant image data after one processing.

[0016] (1-3) Obtain image modal data from the multimodal power plant data after primary processing obtained in step (1-2), and perform geometric correction processing on the obtained image modal data using a combination of feature matching and projection transformation to obtain multimodal power plant image data after secondary processing.

[0017] (1-4) Spatial projection and alignment processing are performed on the multimodal power plant image data after secondary processing obtained in step (1-3) to obtain multimodal power plant image data after tertiary processing.

[0018] (1-5) Stack the channels of the multimodal power plant image after secondary processing obtained in step (1-3) and the multimodal power plant image after tertiary processing obtained in step (1-4) to obtain multimodal power plant image data that fuses illumination, heat and spatial information.

[0019] (1-6) The multimodal power plant image data obtained in step (1-5) after fusing illumination, heat and spatial information is processed using a modal adaptive enhancement and optimization method to obtain multimodal power plant image data after four processing steps.

[0020] (1-7) The multimodal power plant image data obtained in step (1-6) after four processing steps are processed using data augmentation techniques to obtain preprocessed multimodal power plant image data.

[0021] Preferably, step (1-3) specifically involves the following steps: for visible light images and infrared thermal imaging images, this step uses the Scale Invariant Feature Transform (SIFT) or the Oriented Scale Invariant Feature Transform (ORB) combined feature extraction method to perform geometric correction processing; for lidar point cloud images, the geometric correction processing is performed using a combination of Fast Point Feature Histogram (FPFH) and Random Sample Consensus Algorithm (RANSAC).

[0022] Steps (1-4) are as follows: First, the iterative nearest point (ICP) algorithm is used to perform 3D spatial alignment processing on the lidar point cloud image and the visible light image in the multimodal power plant image data obtained in step (1-3) after secondary processing, so as to obtain the processed lidar point cloud image and the visible light image; then, the view projection is performed based on the alignment processing result to generate a depth map; at the same time, the infrared thermal imaging image and the visible light image in the multimodal power plant image data obtained in step (1-3) after secondary processing are subjected to projection mapping and interpolation alignment processing to obtain the processed infrared thermal imaging image; finally, the processed visible light image, infrared thermal imaging image, lidar point cloud image and the generated depth map are integrated to obtain the spatially aligned multimodal power plant image dataset after three processing steps.

[0023] Steps (1-6) specifically involve: first, converting the grayscale image of the multimodal power plant image data obtained in step (1-5), which integrates illumination, heat, and spatial information, into a pseudo-visible light image using a pseudo-color mapping method, to obtain an infrared thermal imaging image that highlights the visual characteristics of high-temperature areas; then, applying CLAHE to the visible light image of the multimodal power plant image data obtained in step (1-5), which integrates illumination, heat, and spatial information, to enhance local contrast, while simultaneously introducing random Gamma correction, brightness perturbation, and image sharpening enhancement methods to achieve the desired effect. A visible light image with clearer image features is obtained. Subsequently, the depth map of the lidar point cloud image in the multimodal power plant image data obtained in steps (1-5), which integrates illumination, heat, and spatial information, is subjected to median filtering for noise reduction, and the pixel values ​​are normalized to obtain a normalized lidar point cloud image. Finally, the infrared thermal imaging image highlighting the visual features of the high-temperature area, the visible light image with clearer image features, and the normalized lidar point cloud image are integrated to obtain the multimodal power plant image data after four processing steps.

[0024] The data augmentation techniques in steps (1-7) include, but are not limited to, one or any combination of multiple of the following: Mosaic mosaic enhancement method, MixUp image fusion enhancement technique, random scale transformation, and color perturbation.

[0025] Preferably, the improved YOLOv12 model includes an input module, a backbone feature extraction module, a feature fusion module, and a detection head module;

[0026] The input module takes a multimodal power plant image with a size of 640×640×3 as input, performs normalization processing on the multimodal power plant image, and outputs a normalized multimodal power plant image.

[0027] The specific structure of the backbone feature extraction module is as follows:

[0028] The first layer is the preliminary feature extraction layer. Its input is the normalized multimodal power plant image obtained by the input module. This preliminary feature extraction layer first inputs the normalized multimodal power plant image into a standard convolutional layer to obtain low-level features in the multimodal power plant image and outputs a feature map with a dimension of 640×640×64. Then, the GhostConv module further processes the feature map, outputting a feature map with a dimension of 640×640×64. Subsequently, the residual block Res performs feature enhancement processing on the 640×640×64 feature map, outputting a feature map with a dimension of 640×640×64. Finally, the 640×640×64 feature map is downsampled, and the final output is a feature map with a dimension of 320×320×128.

[0029] The second layer is a multimodal feature fusion and further feature extraction layer. Its input is the 320×320×128 feature map obtained from the first layer. This multimodal feature fusion and further feature extraction layer first inputs the 320×320×128 feature map output from the first layer into a convolutional layer to output a 320×320×128 feature map. Then, the GhostConv module processes this feature map to output a 320×320×128 feature map. Finally, residual blocks are used... The 320×320×128 feature map is processed (further enhancing features; residual connections ensure that important features are not lost during training) to output a 320×320×128 feature map. Then, the MMFM (Multimodal Attention Fusion) module is used to process the 320×320×128 feature map to output a 320×320×128 feature map. Finally, the feature map is downsampled to output a 160×160×256 feature map.

[0030] The third layer is a deep feature extraction layer. Its input is the 160×160×256 feature map obtained from the second layer. This deep feature extraction layer inputs the 160×160×256 feature map into a convolutional layer to output a 160×160×256 feature map. Next, the GhostConv module is used to further extract features from this feature map to output a 160×160×256 feature map. Subsequently, the feature map is processed through residual blocks to output a 160×160×256 feature map. Then, the MGFP module is used to process the feature map to output a 80×80×512 feature map. Finally, a downsampling operation is performed to output a 80×80×512 feature map.

[0031] The fourth layer is a high-level feature extraction layer. Its input is a feature map with an output dimension of 80×80×512 from the third layer. This high-level feature extraction layer first inputs this feature map into a convolutional layer to output a feature map with a dimension of 80×80×512. Then, the GhostConv module processes this feature map to output a feature map with a dimension of 80×80×512. Next, residual blocks are applied to process this feature map with a dimension of 80×80×512 to output a feature map with a dimension of 80×80×512. Finally, downsampling is performed on this feature map with a dimension of 80×80×512 to finally output a feature map with a dimension of 40×40×1024.

[0032] The fifth layer is the MADSE module, which takes as input the feature maps of each dimension output from the first to fourth layers of the backbone feature extraction module. It processes the input feature maps of each dimension using multi-path dilated convolution and modality saliency map generator, and finally outputs a feature map that integrates multiple information.

[0033] Preferably, the structural hierarchy of the MMFM module is as follows:

[0034] The first layer of the MMFM module is the Modality Adaptive Channel Attention (MCA) layer. Its input is a 320×320×128 feature map output from the residual block in the multimodal feature fusion and further feature extraction layer. This MCA layer uses a squeezing and SE attention mechanism that combines global average pooling, two fully connected layers, and a sigmoid activation function to weight the important channels of each modality in the feature map, so as to output three weighted feature maps with a dimension of 320×320×128.

[0035] The second layer of the MMFM module is the Modal Interaction Spatial Attention Layer (MISA). Its input is a 320×320×128 weighted feature map output from the MCA layer. The MISA layer concatenates these feature maps and generates a spatial attention map through a 3×3 convolution, ultimately resulting in a 320×320×1 spatial attention map.

[0036] The third layer of the MMFM module is the Attention Fusion Module (AGF) layer. Its input is the spatial attention map with a dimension of 320×320×1 output from the MISA layer. The AGF layer performs weighted fusion on the spatial attention map with a dimension of 320×320×1 and outputs a unified fused feature map with a dimension of 320×320×128.

[0037] The fourth layer of the MMFM module is a residual connection and normalization layer (RLN). Its input is a unified fusion feature map with a dimension of 320×320×128 output from the AGF layer. The RLN performs LayerNorm processing on the unified fusion feature map with a dimension of 320×320×128. Then, the LayerNorm processing result is added to the original modality fusion feature through residual connection, and finally outputs a feature map with a dimension of 320×320×128.

[0038] The specific structural hierarchy of the MGFP module is as follows:

[0039] The first layer of the MGFP module is the Modality Grouping Feature Decomposition (MGS) layer. Its input is a feature map with a dimension of 160×160×256 output from the residual block in the deep feature extraction layer. This MGS layer divides the multimodal features of the feature map according to the channel dimension, and each group is divided into multiple sub-branches, outputting a feature map with a dimension of 160×160×256.

[0040] The second layer of the MGFP module is the Pyramid Dilated Convolution Extraction (PDE) layer. Its input is the feature map with a dimension of 160×160×256 output from the MGS layer. The PDE layer groups the modal sub-features of the feature map and uses 3×3 convolutions with different dilation rates to model the grouping results in parallel to obtain a pyramid feature map with a dimension of 160×160×256 for each modality.

[0041] The third layer of the MGFP module is the Cross-Modal Context Fusion (CCF) layer. Its input is the pyramid feature map with a dimension of 160×160×256 output from the PDE layer for each modality. The CCF layer first concatenates the pyramid feature maps with a dimension of 160×160×256 for each modality, and then inputs them into a module consisting of a 3×3 convolutional layer + GELU activation to obtain a more abstract and higher-level feature map. Subsequently, an attention-guided cross-modal channel weighting mechanism is introduced to process the obtained more abstract and higher-level feature map, and finally outputs a fused feature map with a dimension of 160×160×256.

[0042] The fourth layer of the MGFP module is the Global Residual Fusion Layer (GRM). Its input is a fused feature map with a dimension of 160×160×256 output from the CCF layer. The GRM layer performs element-wise weighted fusion of this fused feature map with the 160×160×256 feature map output from the residual block in the deep feature extraction layer. Then, it uses a 1×1 convolution to balance the channel dimensions of the weighted fusion result, and then performs LayerNorm normalization processing. Finally, it outputs a feature map with a dimension of 160×160×256.

[0043] Preferably, the structural hierarchy of the MADSE module is as follows:

[0044] The first layer of the MADSE module is a multi-path dilated convolutional layer (DCP). Its input is the feature maps of various dimensions output from the first to fourth layers of the backbone feature extraction module. The DCP layer processes the input feature map through three different dilated convolutional paths to output the feature map after dilated convolution.

[0045] The second layer of the MADSE module is the Modality Saliency Map Generator (MSG) layer. Its input is the feature map processed by dilated convolution. The MSG layer performs salient feature activation on the feature map to output the saliency map for each input modality.

[0046] The third layer of the MADSE module is the Spatial Weighted Graph Fusion (SWF) layer. Its input is the feature map after dilated convolution output from the DCP layer and the saliency map of the input mode output from the MSG layer. The SWF layer processes the feature map and the saliency map through Softmax normalization to generate the spatial weighted graph.

[0047] The fourth layer of the MADSE module is the Feature Normalization and Residual Connection layer (FSR). Its input is the spatial weight map output by the SWF layer. The FSR layer first performs layer normalization on the spatial weight map, and then adds the layer normalized spatial weight map to the feature map input from the first layer of the MADSE module through residual connections to output a feature map.

[0048] Preferably, the inputs to the feature fusion module are the feature map with a dimension of 320×320×128 output from the first layer of the backbone feature extraction module, the feature map with a dimension of 160×160×256 output from the second layer of the backbone feature extraction module, the feature map with a dimension of 80×80×512 output from the third layer of the backbone feature extraction module, and the feature map with a dimension of 40×40×1024 output from the fourth layer of the backbone feature extraction module. The feature fusion module performs multi-scale feature fusion and enhancement processing on these four feature maps respectively to obtain feature maps P2, P3, P4, and P5 after fusion of the four dimensions.

[0049] The specific structural hierarchy of the feature fusion module is as follows:

[0050] The first layer is an upsampling and element-wise fusion layer. Its input is the feature maps output from the third and fourth layers of the backbone feature extraction module. This layer first performs 1×1 convolutional dimensionality reduction on the feature map output from the fourth layer of the backbone feature extraction module to obtain a feature map with a dimension of 40×40×512. Then, it performs upsampling on this feature map to output a feature map with a dimension of 80×80×512. Finally, it performs element-wise addition and fusion with the feature map output from the third layer of the backbone feature extraction module to output a feature map with a dimension of 80×80×512.

[0051] The second layer is a resampling and element-wise fusion layer. Its input is the feature map with a dimension of 80×80×512 output from the first layer and the feature map output from the second layer of the backbone feature extraction module. This layer first performs 1×1 convolution on the feature map with a dimension of 80×80×512 output from the first layer to reduce the dimension to obtain a feature map with a dimension of 80×80×256. Then, it performs upsampling on this feature map to output a feature map with a dimension of 160×160×256. Finally, it performs element-wise addition and fusion on the upsampled feature map and the feature map output from the second layer of the backbone feature extraction module to output a feature map with a dimension of 160×160×256.

[0052] The third layer is the final upsampling and element-wise fusion layer. Its input is the feature map with a dimension of 160×160×256 output from the second layer and the feature map output from the first layer of the backbone feature extraction module. This layer first performs 1×1 convolution on the feature map with a dimension of 160×160×256 output from the second layer to reduce the dimension to 160×160×128. Then, it performs upsampling on this feature map to output a feature map with a dimension of 320×320×128. Finally, it performs element-wise addition and fusion on the upsampled feature map and the feature map output from the first layer of the backbone feature extraction module to output a feature map with a dimension of 320×320×128.

[0053] The fourth layer is a downsampling and element-wise fusion layer. Its input is the 320×320×128 feature map output from the third layer and the 320×320 feature map output from the first layer of the backbone feature extraction module. This layer first performs a 3×3 convolution with a stride of 2 on the feature map output from the first layer of the backbone feature extraction module to obtain a feature map with a dimension of 160×160×128. Then, it fuses the obtained feature map with the 320×320×128 feature map output from the third layer to output a feature map with a dimension of 160×160×256. Finally, it performs a 3×3 convolution on the 160×160×256 feature map to output a feature map with a dimension of 160×160×256.

[0054] The fifth layer is a resampling and element-wise fusion layer. Its input is the feature map with a dimension of 160×160×256 output from the fourth layer and the feature map output from the fifth layer of the backbone feature extraction module. This layer first performs a 3×3 convolution on the feature map with a dimension of 160×160×256 output from the fourth layer to obtain a feature map with a dimension of 80×80×256. Then, the obtained feature map is fused with the feature map output from the fifth layer of the backbone feature extraction module to obtain a feature map with a dimension of 80×80×512. Finally, a 3×3 convolution is performed on the feature map with a dimension of 80×80×512 to output a feature map with a dimension of 80×80×512.

[0055] The sixth layer is the final downsampling and element-wise fusion layer. Its input is the feature map with a dimension of 80×80×512 output from the fifth layer and the feature map output from the fifth layer of the backbone feature extraction module. This layer first performs a 3×3 convolution on the feature map with a dimension of 80×80×512 output from the fifth layer to obtain a feature map with a dimension of 40×40×512. Then, the obtained feature map is fused with the feature map output from the fifth layer of the backbone feature extraction module to output a feature map with a dimension of 40×40×1024. Finally, the obtained feature map with a dimension of 40×40×1024 is performed a 3×3 convolution to output a feature map with a dimension of 40×40×1024.

[0056] The seventh layer is a channel-weighted and context-modeling layer. Its input is the feature maps of four dimensions output from the first to the sixth layers. This layer first uses a 3×3 convolution to extract local spatial features from the feature maps of the four dimensions to obtain feature maps of the four dimensions. Then, the feature maps of the four dimensions are flattened into a sequence to obtain feature maps of the four dimensions that retain local spatial feature information. After that, a lightweight multi-head self-attention mechanism (LTM) is used to process the flattened feature maps of the four dimensions to obtain feature maps of specific features in the four dimensions. Finally, the obtained feature maps of specific features in the four dimensions are input into a fully connected layer for normalization processing, and finally output four fused feature maps of the four dimensions: P2, P3, P4, and P5.

[0057] The detection head module takes as input the four-dimensional feature maps P2, P3, P4, and P5 output by the feature fusion module. The detection head module processes the feature map of each dimension using the corresponding detection head structure, and finally outputs a multimodal power plant image with defect category, bounding box coordinates, confidence score, and fault category.

[0058] Preferably, the improved YOLOv12 model is trained according to the following steps:

[0059] (3-1) Collect multimodal power station image data obtained by UAV during power station inspection, preprocess the multimodal power station image data to obtain preprocessed multimodal power station image data, and use manual annotation and automatic auxiliary tools to perform multi-label annotation on the equipment fault targets in the multimodal power station image data to obtain dataset S, and divide the dataset into training set and test set in an 8:2 ratio;

[0060] (3-2) Initialize the parameters of the improved YOLOv12 model to obtain the initialized improved YOLOv12 model;

[0061] (3-3) For each sample in the training set obtained in step (3-1), input the sample into the first layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map corresponding to the sample with a dimension of 320×320×128.

[0062] (3-4) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 320×320×128 corresponding to the sample obtained in step (3-3) is input into the second layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 160×160×256 corresponding to the sample.

[0063] (3-5) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 160×160×256 corresponding to the sample obtained in step (3-4) is input into the third layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 80×80×512 corresponding to the sample.

[0064] (3-6) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 80×80×512 corresponding to the sample obtained in step (3-5) is input into the fourth layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 40×40×1024 corresponding to the sample.

[0065] (3-7) For each sample in the training set obtained in step (3-1), the feature maps corresponding to the sample obtained in steps (3-3), (3-4), (3-5), and (3-6) are respectively input into the fifth layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature maps corresponding to the sample with dimensions of 320×320×128, 160×160×256, 80×80×512, and 40×40×1024.

[0066] (3-8) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 80×80×512 and 40×40×1024 obtained in steps (3-5) and (3-6) respectively are input into the first layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 80×80×512 corresponding to the sample.

[0067] (3-9) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 160×160×256 and 80×80×512 obtained in steps (3-4) and (3-8) respectively are input into the second layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 160×160×256 corresponding to the sample.

[0068] (3-10) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 320×320×128 and 160×160×256 obtained in steps (3-3) and (3-9) is input into the third layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 320×320×128 corresponding to the sample.

[0069] (3-11) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 320×320×128 and 320×320×128 obtained in steps (3-3) and (3-10) is input into the fourth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 160×160×256 corresponding to the sample;

[0070] (3-12) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 160×160×256 and 160×160×256 obtained in steps (3-7) and (3-11) are input into the fifth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 80×80×512 corresponding to the sample.

[0071] (3-13) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 80×80×512 and 80×80×512 obtained in steps (3-7) and (3-12) is input into the sixth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 40×40×1024 corresponding to the sample.

[0072] (3-14) For each sample in the training set obtained in step (3-1), input the feature map corresponding to the sample obtained in steps (3-8) to (3-13) into the seventh layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the fused feature map corresponding to the sample with dimensions of 320×320×128, 160×160×256, 80×80×512, and 40×40×1024.

[0073] (3-15) For each sample in the training set obtained in step (3-1), the feature map corresponding to the sample obtained in step (3-14) is input into the detection head module of the YOLOv12 improved model after initialization in step (2) to obtain the defect category, local bounding box coordinates and confidence score corresponding to the sample.

[0074] (3-16) For each sample in the training set obtained in step (3-1), based on the defect category corresponding to the sample obtained in step (3-15), the defect loss value corresponding to the sample is obtained by using the multi-label FocalLoss function. Based on the bounding box coordinates corresponding to the sample, the bounding box loss value corresponding to the sample is obtained by using the WIoUv2 function. Based on the confidence score corresponding to the sample, the confidence loss value corresponding to the sample is obtained by using the Softmax loss function. Based on the obtained defect loss value, bounding box loss value, and confidence loss value, the total loss corresponding to the sample is obtained.

[0075] (3-17) For each sample in the training set obtained in step (3-1), the total loss corresponding to the sample obtained in step (3-16) is used to iteratively train the YOLOv12 improved model using gradient descent until the YOLOv12 improved model reaches the preset number of iterations, and the optimal parameters of the YOLOv12 improved model at this time are obtained, thus obtaining the initially trained YOLOv12 improved model;

[0076] (3-18) Use the test set obtained in step (3-1) to test the YOLOv12 improved model initially trained in step (3-17) until the detection accuracy reaches the optimal level, so as to obtain the final trained YOLOv12 improved model.

[0077] Preferably, the initialization process in step (3-2) is as follows: the initial value of the weight parameter is a random value output by a truncated normal distribution with a standard deviation of 0.1, the initial value of the bias parameter is set to 0, the initial learning rate is lr=0.0003, a step-wise learning strategy is adopted, the step size is stepsize=200, and the weight gamma=0.1, that is, the learning rate is multiplied by 0.1 every 200 rounds;

[0078] The defect loss value is calculated using the following formula:

[0079]

[0080] Where C represents the total number of fault categories, This represents the true label of the i-th type of fault. This represents the predicted probability of the improved YOLOv12 model for the i-th type of fault. The importance coefficients represent the positive and negative samples of the i-th type of fault. Let represent the modulation factor, used to suppress the contribution of easily classified samples to the loss, and i∈[1,C];

[0081] The loss value of the positioning frame is calculated using the following formula:

[0082]

[0083] in, Indicates the prediction box. Represents a real bounding box; This represents the overlap index output by the WIoUv2 loss function, with a value range of [0,1]. The larger the value, the higher the overlap.

[0084] The confidence loss value for a sample is calculated using the following formula:

[0085]

[0086] in, This represents the true label of the j-th type of fault. Let represent the initial score of the improved YOLOv12 model for the j-th fault type, and j∈[1, C];

[0087] The total loss of the final model is expressed as:

[0088]

[0089] in , and These are the weighted coefficients for defect loss, location frame loss, and confidence loss, respectively, and the sum of the three is 1.

[0090] According to another aspect of the present invention, a fault detection system for unmanned aerial vehicle (UAV) power plant inspection based on improved YOLOv12 and multimodal deep feature fusion is provided, comprising:

[0091] The first module is used to acquire multimodal power plant image data from the power plant site using a drone, and to preprocess the multimodal power plant image data to obtain preprocessed multimodal power plant image data.

[0092] The second module is used to input the preprocessed multimodal power plant image data obtained from the first module into the pre-trained YOLOv12 improved model to obtain the final detection results.

[0093] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0094] (1) This invention uses steps (1-1) to (1-7), which enhance the feature information of visible light images, infrared thermal imaging images and point LiDAR point cloud images by using a combination of feature matching and projection transformation, and a series of multimodal preprocessing methods such as Contrast Limited Adaptive Histogram Equalization (CLAHE), to achieve multimodal image spatial alignment and channel stacking, and to fuse three types of information: illumination, heat and space. In addition, the MMFM module in step (2) is used to achieve saliency guidance and dynamic fusion between RGB, infrared thermal imaging images and LiDAR point cloud image features, which significantly enhances the recognition effect of the model in low illumination, occlusion and weak thermal signal scenes.

[0095] (2) The present invention uses the MADSE module in step (2), which enables the model to take into account both macroscopic equipment structure and micro fault areas through the dilated convolution group and modality-guided receptive field adjustment mechanism. At the same time, the detection head module in step (2) is used, which adds a P2 detection head to the detection head module to improve the ability to identify small targets such as microcracks and slight hot spots, thereby maintaining high-precision detection effect in scenarios where targets of various sizes coexist.

[0096] (3) In this invention, steps (1-5) are used to convert grayscale images into pseudo-visible light images by using pseudo-color mapping to obtain infrared thermal imaging images that highlight the visual features of high-temperature areas. At the same time, the MGFP module in step (2) is used to realize intra-group convolution and cross-attention fusion of modal features, which effectively improves the robustness to thermal signal noise. This strategy can dynamically fuse other modal features to compensate for missing features when modal information is incomplete, or when the infrared thermal imaging image is blurred or shifted, thereby ensuring the stability and accuracy of the detection results.

[0097] (4) Since the present invention adopts steps (3-1) to (3-2), by using a multi-label learning strategy, the model can not only identify the category and location of the fault target, but also output the severity score of the fault for subsequent alarm classification and maintenance priority ranking. It also supports unified detection of multiple types of faults such as hot spots, cracks, and damage, thereby improving the universality and scalability of the present invention. Attached Figure Description

[0098] Figure 1 This is a flowchart of the method for detecting faults in power plant inspections using unmanned aerial vehicles (UAVs) based on improved YOLOv12 and multimodal deep feature fusion, as described in this invention.

[0099] Figure 2 This is a schematic diagram of the working process of the improved YOLOv12 model of this invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0101] The basic idea of ​​this invention is to systematically improve the accuracy and adaptability of power equipment fault detection from two levels: multimodal information fusion and task modeling. First, by performing image enhancement and pseudo-color mapping preprocessing operations on visible light images, infrared thermal imaging images, and lidar point cloud images, the quality of input data and the fusionability between modalities are improved. Second, a spatial alignment and channel stacking mechanism is constructed, and an MMFM module is introduced into the backbone feature extraction module to achieve saliency guidance and dynamic fusion of multimodal features, thereby enhancing the model's robustness in complex environments such as low illumination, occlusion, and weak thermal signals. Third, a MADSE module is integrated into the backbone feature extraction module to construct a multi-scale dynamic perception structure, effectively improving the ability to simultaneously identify minute faults and macroscopic structures. Simultaneously, the MGFP module is combined to finely model the complementary relationships and interference suppression capabilities between modalities, improving the fusion feature expression effect. Finally, a multi-label learning framework is constructed, which not only enables joint detection of multiple types of faults but also introduces a fault severity scoring mechanism to meet the practical needs of "multi-energy in one network, risk classification" in power inspection tasks.

[0102] like Figure 1 As shown, this invention discloses a method for fault detection in UAV power plant inspection based on improved YOLOv12 and multimodal deep feature fusion, comprising:

[0103] (1) Obtain multimodal power plant image data from the power plant site using UAV, and preprocess the multimodal power plant image data to obtain preprocessed multimodal power plant image data;

[0104] Specifically, the UAV of the present invention is equipped with a high-resolution visible light camera, an infrared thermal imager, and a high-precision lidar system.

[0105] During the drone's inspection flight over the power plant area, the multimodal power plant image data collected includes visible light images, infrared thermal imaging images, and lidar point cloud images. The lidar point cloud data includes three-dimensional spatial coordinates (X,Y,Z) and reflectivity information, which are used for scene modeling and depth estimation.

[0106] This step includes the following sub-steps:

[0107] (1-1) The UAV acquires multimodal power station data from the power station site and adds timestamps to the multimodal power station data to obtain multimodal power station data with added timestamps;

[0108] Specifically, this step involves adding nanosecond-level GPS timestamps to the multimodal power plant data to ensure its synchronization consistency on the timeline, providing a precise timing basis for subsequent data fusion and alignment.

[0109] (1-2) Perform time-series alignment processing on the multimodal power plant data obtained in step (1-1) after adding timestamps to obtain multimodal power plant image data after one processing.

[0110] Specifically, this step uses a GPS timestamp-based time window matching algorithm (the algorithm flow can be found in steps one to three of the "Detailed Implementation" section of the Chinese invention application CN202111081296.0, entitled "Multi-sensor Time Synchronization Method Based on Location Information") to perform coarse-grained time synchronization processing on the multimodal power station image data (including visible light images, infrared thermal imaging images, and point lidar point cloud images, etc.) obtained in step (1-1) after adding timestamps, so as to obtain multimodal power station image data after one processing. The purpose is to achieve the correspondence and alignment of multimodal power station data within a unified time window.

[0111] (1-3) Obtain image modal data from the multimodal power plant data after primary processing obtained in step (1-2), and use the method of combining feature matching and projection transformation (see the improved GMS and weighted projection method proposed in the article "Image Registration Strategy Based on GMS Screening and Weighted Projection Transformation" published by Chen et al. in 2018) to perform geometric correction processing on the obtained image modal data to obtain the multimodal power plant image data after secondary processing.

[0112] Specifically, for visible light images and infrared thermal imaging images, this step uses Scale-Invariant Feature Transform (SIFT) or Oriented Features from Accelerated Segment Test and Rotated Binary Robust Independent Elementary Features (ORB) methods for geometric correction. For LiDAR point cloud images, a combination of Fast Point Feature Histograms (FPFH) and Random Sample Consensus (RANSAC) is used for geometric correction.

[0113] (1-4) Spatial projection and alignment processing are performed on the multimodal power station image data after secondary processing obtained in step (1-3) to obtain multimodal power station image data after tertiary processing.

[0114] Specifically, this step involves the following steps: First, the Iterative Closest Point (ICP) algorithm is used to perform 3D spatial alignment processing on the lidar point cloud image and the visible light image in the secondary processed multimodal power plant image data obtained in steps (1-3) to obtain the processed lidar point cloud image and visible light image. Then, based on the alignment processing result, a view projection is performed to generate a depth map. Simultaneously, the infrared thermal imaging image and the visible light image in the secondary processed multimodal power plant image data obtained in steps (1-3) are subjected to projection mapping and interpolation alignment processing to obtain the processed infrared thermal imaging image (ensuring that the infrared thermal imaging image is the same as the visible light image at the pixel level). Finally, the processed visible light image, infrared thermal imaging image, lidar point cloud image, and generated depth map are integrated to obtain the spatially aligned multimodal power plant image dataset after three processing steps.

[0115] (1-5) Stack the channels of the multimodal power plant image after secondary processing obtained in step (1-3) and the multimodal power plant image after tertiary processing obtained in step (1-4) to obtain multimodal power plant image data that fuses illumination, heat and spatial information.

[0116] This fusion method allows each sample to simultaneously contain illumination, heat, and spatial information, providing a rich feature base for subsequent model processing.

[0117] (1-6) The multimodal power plant image data obtained in step (1-5) after fusing illumination, heat and spatial information is processed using a modal adaptive enhancement and optimization method (the method is derived from the adaptive enhancement method and image processing method mentioned in the paper "Adaptive Enhancement Method for Multimode Remote Sensing Image Based on LiDAR") to obtain multimodal power plant image data after four processing steps (which can both retain modal specificity and enhance cross-modal complementarity).

[0118] Specifically, this step involves first converting the infrared thermal imaging image from the multimodal power plant image data obtained in steps (1-5), which integrates illumination, heat, and spatial information, into a pseudo-visible light image using a pseudo-color mapping method (such as pseudo-color mapping COLORMAP_JET). This process aims to obtain an infrared thermal imaging image that highlights the visual characteristics of high-temperature areas (facilitating model identification of fault signals such as hot spots and ablation). Then, the visible light image from the multimodal power plant image data obtained in steps (1-5), which integrates illumination, heat, and spatial information, is processed using CLAHE to enhance local contrast, while simultaneously introducing random Gamma correction. Enhancement methods such as brightness perturbation and image sharpening are used to obtain a visible light image with clearer image features. Subsequently, the depth map of the lidar point cloud image in the multimodal power plant image data obtained in steps (1-5), which integrates illumination, heat, and spatial information, is processed by median filtering for noise reduction, and the pixel values ​​are normalized to obtain a normalized lidar point cloud image. Finally, the infrared thermal imaging image highlighting the visual features of the high-temperature area, the visible light image with clearer image features, and the normalized lidar point cloud image are integrated to obtain the multimodal power plant image data after four processing steps.

[0119] (1-7) The multimodal power plant image data obtained in step (1-6) after four processing steps are processed using data augmentation techniques to obtain preprocessed multimodal power plant image data.

[0120] Specifically, the data augmentation techniques in this step include, but are not limited to, the Mosaic image stitching augmentation method (stitching all images in the four-processed multimodal power plant image data obtained in steps (1-6) into a large image to simulate richer scene combinations and target distributions, so as to obtain augmented samples with composite scene information), MixUp image fusion augmentation technology (weighting and mixing two images and their corresponding labels to generate transitional samples, so as to obtain intermediate training samples that enhance the model's ability to distinguish category boundaries), random scaling transformation (including random scaling, cropping, etc.), and color perturbation (including random adjustment of brightness, contrast, saturation and hue) or any combination of any of these.

[0121] The advantage of the above steps (1-1) to (1-7) is that it can obtain clearer and more feature-rich multimodal power plant image data, thereby enhancing the robustness of the algorithm model in complex environments.

[0122] (2) Input the preprocessed multimodal power plant image data obtained in step (1) into the pre-trained YOLOv12 improved model to obtain the final detection result;

[0123] Specifically, the final detection results include fault category, location information, and severity score.

[0124] like Figure 2 As shown, the improved YOLOv12 model in this invention includes an input module, a backbone feature extraction module, a feature fusion module, and a detection head module.

[0125] The input module takes a multimodal power plant image (including visible light image, infrared thermal imaging image and lidar point cloud image) with a size of 640×640×3 as input. The input module performs normalization processing on the multimodal power plant image (that is, normalizes its pixel values ​​to the range of [0,1]) and outputs the normalized multimodal power plant image.

[0126] The backbone feature extraction module of this invention is based on the original CSPDarkNet backbone structure of YOLOv12. This invention introduces the GhostConv module to replace some standard convolutional layers to reduce computational complexity and the number of model parameters. To enhance the model's adaptability to targets of different sizes, this invention introduces a multi-scale dynamic feature extraction module (MADSE) after the output of each layer from C2 to C5.

[0127] The specific structure of the backbone feature extraction module is as follows:

[0128] The first layer is the preliminary feature extraction layer. Its input is the normalized multimodal power plant image obtained from the input module. This preliminary feature extraction layer first inputs the normalized multimodal power plant image into a standard convolutional layer (3×3 kernel size, 64 output channels) to obtain low-level features (such as edges, textures, etc.) in the multimodal power plant image, and outputs a feature map with dimensions of 640×640×64. Then, the feature map is further processed by the GhostConv module (3×3 convolutional kernel, still with 64 output channels, and using sparse convolution to generate redundant features, thereby reducing computation while maintaining high feature representation capability), outputting a feature map with dimensions of 640×640×64. The first feature map has a dimension of 640×640×64. Then, a residual block (Res) is used to enhance the feature map (residual connections ensure smooth transfer of feature information, avoiding the gradient vanishing problem during training and further improving the feature learning effect), outputting a feature map with a dimension of 640×640×64. Finally, a downsampling operation is performed on the 640×640×64 feature map (usually by using a convolutional layer with a stride of 2 to reduce the spatial size of the feature map and increase the number of channels), ultimately outputting a feature map with a dimension of 320×320×128.

[0129] The second layer is a multimodal feature fusion and further feature extraction layer. Its input is the 320×320×128 feature map obtained from the first layer. This multimodal feature fusion and further feature extraction layer first inputs the 320×320×128 feature map output from the first layer into a convolutional layer (3×3 kernel size, 128 output channels, used to extract higher-level feature information from the image), outputting a 320×320×128 feature map. Then, the GhostConv module processes this feature map (using a 3×3 convolution kernel, 128 output channels, generating redundant features through sparse convolution to optimize computational efficiency while retaining important feature information), outputting a 320×320×128 feature map. Finally, residual blocks are used to process the 320×320×128 feature map. The 0×128 feature map is processed (further enhancing features; residual connections ensure that important features are not lost during training) to output a feature map with a dimension of 320×320×128. Then, a Multimodal Attention-based Fusion Module (MMFM) is used to process the 320×320×128 feature map (weighted fusion of features from RGB, IR, and LR images) to output a feature map with a dimension of 320×320×128. Finally, this feature map is downsampled (the spatial size is reduced from 320×320 to 160×160 through convolution, and the number of channels is increased to 256), ultimately outputting a feature map with a dimension of 160×160×256.

[0130] Furthermore, the specific structural hierarchy of the MMFM module in this invention is as follows:

[0131] The first layer of the MMFM module is the Modality Adaptive Channel Attention (MCA) layer. Its input is a 320×320×128 feature map output from the residual block of the multimodal feature fusion and further feature extraction layer. This MCA layer adopts a squeeze-and-excitation (SE) attention mechanism that combines global average pooling, two fully connected layers, and a sigmoid activation function. It assigns weights to the important channels of each modality in the feature map to output three weighted feature maps with a dimension of 320×320×128.

[0132] The second layer of the MMFM module is the Modality Interaction Spatial Attention (MISA) layer. Its input is the three weighted feature maps with a dimension of 320×320×128 output from the MCA layer. The MISA layer concatenates these feature maps and generates a spatial attention map (used to highlight the location region with the strongest response to the target in different modalities) through 3×3 convolution. Finally, a spatial attention map with a dimension of 320×320×1 is obtained (which acts on each modal feature map to enhance the localization of the target region).

[0133] The third layer of the MMFM module is the Attention-guided Fusion (AGF) module layer. Its input is the spatial attention map with a dimension of 320×320×1 output from the MISA layer. The AGF layer performs weighted fusion on the spatial attention map with a dimension of 320×320×1 and outputs a unified fused feature map with a dimension of 320×320×128.

[0134] The fourth layer of the MMFM module is the Residual+LN (RLN) layer. Its input is the unified fusion feature map with a dimension of 320×320×128 output from the AGF layer. The RLN performs LayerNorm processing on the unified fusion feature map with a dimension of 320×320×128 (to unify the distribution scale). Then, the LayerNorm processing result is added to the original modality fusion feature through residual connection (to improve stability and information retention). Finally, the output feature map has a dimension of 320×320×128.

[0135] The third layer is a deep feature extraction layer. Its input is the 160×160×256 feature map obtained from the second layer. This deep feature extraction layer feeds this 160×160×256 feature map into a convolutional layer (3×3 kernel size, 256 output channels to extract deeper features from the image), outputting a 160×160×256 feature map. Next, the GhostConv module further extracts features from this feature map (still with a 3×3 kernel size and 256 output channels, generating redundant features and improving computational efficiency), outputting another 160×160×256 feature map. Subsequently, through residual... The difference block processes the feature map (enhancing its expressive power and ensuring effective feature transfer) to output a feature map with a dimension of 160×160×256. Then, the MGFP module processes the feature map (performing pyramid-level multi-scale context modeling between different modal information, paying particular attention to the structural information representation of small targets and targets of different sizes, thereby improving the model's robustness to target scale) to output a feature map with a dimension of 80×80×512. Finally, a downsampling operation is performed (reducing the spatial size from 160×160 to 80×80 and increasing the number of channels to 512) to output a feature map with a dimension of 80×80×512.

[0136] Furthermore, the specific structural hierarchy of the MGFP module of this invention is as follows:

[0137] The first layer of the MGFP module is the Modality Group Separation (MGS) layer. Its input is a feature map with a dimension of 160×160×256 output from the residual block in the deep feature extraction layer. The MGS layer divides the multimodal features of the feature map according to the channel dimension, and each group is divided into multiple sub-branches. The output is a feature map with a dimension of 160×160×256.

[0138] The second layer of the MGFP module is the Pyramid Dilation Extraction (PDE) layer. Its input is the feature map with a dimension of 160×160×256 output from the MGS layer. The PDE layer groups the modal sub-features of the feature map and uses 3×3 convolutions with different dilation rates (1, 2, 3) to model the grouping results in parallel (where a dilation rate of 1 is used to extract local fine textures; a dilation rate of 2 is used to extract medium-scale objects; and a dilation rate of 3 is used to extract large-scale contextual relationships) to obtain a pyramid feature map with a dimension of 160×160×256 for each modality.

[0139] The third layer of the MGFP module is the Cross-modality Context Fusion (CCF) layer. Its input is the pyramid feature map with a dimension of 160×160×256 output from the PDE layer for each modality. The CCF layer first concatenates the pyramid feature maps with a dimension of 160×160×256 for each modality, and then inputs them into a module consisting of a 3×3 convolutional layer + GELU activation to obtain a feature map with a more abstract and higher level. Subsequently, an attention-guided cross-modality channel weighting mechanism is introduced to process the obtained feature map with a more abstract and higher level (adaptively adjusting the importance of different modalities and scale branches), and finally outputs a fused feature map with a dimension of 160×160×256.

[0140] The fourth layer of the MGFP module is the Global Residual Merge (GRM) layer. Its input is a fused feature map with a dimension of 160×160×256 output from the CCF layer. The GRM layer performs element-wise weighted fusion of this fused feature map with the 160×160×256 feature map output from the residual block in the deep feature extraction layer. Then, it uses a 1×1 convolution to balance the channel dimensions of the weighted fusion result, and then performs LayerNorm normalization (to ensure feature stability). The final output is a feature map with a dimension of 160×160×256.

[0141] The fourth layer is a high-level feature extraction layer. Its input is the feature map with an output dimension of 80×80×512 from the third layer. This high-level feature extraction layer first inputs this feature map into a convolutional layer (with a kernel size of 3×3 and 512 output channels to further extract high-level semantic information from the image), outputting a feature map with a dimension of 80×80×512. Then, the feature map is processed by the GhostConv module (with a kernel size of 3×3 and 512 output channels to generate redundant features and improve computational efficiency). The process begins by generating a feature map with a dimension of 80×80×512. Then, residual blocks are applied to this 80×80×512 feature map (further enhancing its expressive power and maintaining effective transmission of feature information), resulting in another 80×80×512 feature map. Finally, a downsampling operation is performed on this 80×80×512 feature map (reducing the spatial size from 80×80 to 40×40 and increasing the number of channels to 1024), ultimately outputting a feature map with a dimension of 40×40×1024.

[0142] The fifth layer is the MADSE module, which takes as input the feature maps of each dimension output from the first to fourth layers of the backbone feature extraction module. It processes the input feature maps of each dimension using multi-path dilated convolution and modality saliency map generator, and finally outputs a feature map that integrates multiple information.

[0143] The specific hierarchical structure of the MADSE module is as follows:

[0144] The first layer of the MADSE module is a multi-path dilated convolutional layer (DCP). Its input is the feature maps of various dimensions output from the first to fourth layers of the backbone feature extraction module. This DCP layer processes the input feature maps through three different dilated convolutional paths (each path has a 3×3 kernel size, but uses a different dilation rate to extract features from different receptive fields. Path 1 uses dilation rate=1, i.e., standard convolution, mainly extracting local low-level features, with the output size being the original size, enhancing the perception of local details (such as edges and textures); Path 2 uses dilation rate=2, capturing broader contextual information through a larger receptive field, with the output size being the original size, helping to capture semantic relationships between regions; Path 3 uses dilation rate=3, further expanding the receptive field, extracting global contextual information, and helping to enhance the perception of large areas), to output a feature map processed by dilated convolution.

[0145] The second layer of the MADSE module is the ModalitySaliencyMapGenerator (MSG) layer. Its input is the feature map after dilated convolution. The MSG layer performs salient feature activation on the feature map to output the saliency map of each input modality (RGB, IR, LR) (the output image dimension is the original dimension, used to highlight important regions in the image, such as cracks, hot spots and other key target areas).

[0146] The third layer of the MADSE module is the Spatial Weight Map Fusion (SWF) layer. Its input is the feature map processed by dilated convolution from the DCP layer and the saliency map of the input modality (RGB, IR, LR) from the MSG layer. The SWF layer processes the feature map and the saliency map through Softmax normalization to generate a spatial weight map. (This layer mainly weights different regions of the weight map so that salient regions can receive higher weights, thereby guiding the network to better understand the target features. This mechanism effectively avoids the interference of background noise and enhances the performance of the target region. The final spatial weight map generated by this layer is 1 times the size of the original size (for example, 80×80×512 is 80×80×1), which is used to weight the fused feature map, and the final output is the feature map of the original dimension.)

[0147] The fourth layer of the MADSE module is the Feature Normalization and Residual Connection (FSR) layer. Its input is the spatial weight map output by the SWF layer. The FSR layer first performs layer normalization on the spatial weight map (to ensure that information from different scales and modalities is fused at the same scale). Then, it adds the layer normalized spatial weight map to the feature map input from the first layer of the MADSE module through residual connection (to ensure that information is not lost and the model training process is more stable) to output a feature map.

[0148] The feature fusion module of this invention is also called the neck network module. This module is located after the backbone network. The input of the feature fusion module is the feature map with a dimension of 320×320×128 output from the first layer of the backbone feature extraction module, the feature map with a dimension of 160×160×256 output from the second layer of the backbone feature extraction module, the feature map with a dimension of 80×80×512 output from the third layer of the backbone feature extraction module, and the feature map with a dimension of 40×40×1024 output from the fourth layer of the backbone feature extraction module. The feature fusion module processes these... The four feature maps undergo multi-scale feature fusion and enhancement processing (the core objective of this module is to improve the richness of feature representation and the robustness of object detection by utilizing multi-scale fusion and contextual relationship enhancement; it is mainly used for multi-scale feature fusion and enhancement, aiming to improve the detection capability of objects at different scales and enhance local and global semantic expression capabilities). This results in four fused feature maps: P2 (320×320×128), P3 (160×160×256), P4 (80×80×512), and P5 (40×40×1024). This module integrates the Feature Pyramid Network (FPN), the Path Aggregation Network (PAN), and the Lightweight Transformer Module (LTM), and introduces the Channel Attention Adaptation Module (CAAM) to form a complete uplink and downlink fusion structure, effectively enhancing the discriminativeness and robustness of the features.

[0149] The specific structural hierarchy of the feature fusion module is as follows:

[0150] The first layer is an upsampling and element-wise fusion layer. Its input is the feature maps output from the third and fourth layers of the backbone feature extraction module. This layer first performs 1×1 convolutional dimensionality reduction processing on the feature map output from the fourth layer of the backbone feature extraction module (the input is compressed into a 1×1 convolutional layer, reducing the number of output channels to 512) to obtain a feature map with a dimension of 40×40×512. Then, it performs upsampling processing on this feature map (using bilinear interpolation to upsample it to 80×80) to output a feature map with a dimension of 80×80×512. Finally, it performs element-wise addition and fusion with the feature map output from the third layer of the backbone feature extraction module (80×80×512) (after passing through a 3×3 standard convolutional layer to fuse details and eliminate noise introduced by interpolation) to output a feature map with a dimension of 80×80×512.

[0151] The second layer is a resampling and element-wise fusion layer. Its input is the feature map with a dimension of 80×80×512 output from the first layer and the feature map output from the second layer of the backbone feature extraction module. This layer first performs 1×1 convolutional dimensionality reduction processing on the 80×80×512 feature map output from the first layer (the input is compressed into a 1×1 convolutional layer, and the number of output channels is reduced to 256) to obtain a feature map with a dimension of 80×80×256. Then, it performs upsampling processing on this feature map (using bilinear interpolation to upsample it to 160×160) to output a feature map with a dimension of 160×160×256. Then, it performs element-wise addition and fusion (after a 3×3 standard convolutional layer) on the upsampled feature map and the feature map output from the second layer of the backbone feature extraction module (160×160×256) to output a feature map with a dimension of 160×160×256.

[0152] The third layer is the final upsampling and element-wise fusion layer. Its input is the 160×160×256 feature map output from the second layer and the feature map output from the first layer of the backbone feature extraction module. This layer first performs 1×1 convolutional dimensionality reduction processing on the 160×160×256 feature map output from the second layer (channel compression is performed on the input to the 1×1 convolutional layer, and the number of output channels is reduced to 128) to obtain a feature map with a dimension of 160×160×128. Then, the feature map is upsampled (upsampled to 320×320 using bilinear interpolation) to output a feature map with a dimension of 320×320×128. Then, the upsampled feature map is fused with the feature map (320×320×128) output from the first layer of the backbone feature extraction module element-wise (after a 3×3 standard convolutional layer) to output a feature map with a dimension of 320×320×128.

[0153] The fourth layer is a downsampling and element-wise fusion layer. Its input is the 320×320×128 feature map output from the third layer and the 320×320 feature map output from the first layer of the backbone feature extraction module. This layer first performs a 3×3 convolution with a stride of 2 on the feature map output from the first layer of the backbone feature extraction module (the input 3×3 convolution layer with a stride of 2 is downsampled to 160×160, and the number of channels remains unchanged) to obtain a feature map with a dimension of 160×160×128. Next, the obtained feature map is fused with the 320×320×128 feature map output from the third layer (the channels are adjusted to 256 through a 1×1 convolution, and then element-wise addition is performed) to output a feature map with a dimension of 160×160×256. Finally, a 3×3 convolution is performed on the 160×160×256 feature map to output a feature map with a dimension of 160×160×256.

[0154] The fifth layer is a downsampling and element-wise fusion layer. Its input is the feature map with dimensions of 160×160×256 output from the fourth layer and the feature map output from the fifth layer of the backbone feature extraction module. This layer first performs a 3×3 convolution on the feature map with dimensions of 160×160×256 output from the fourth layer (downsampling to 80×80 from the input 3×3 convolution layer, keeping the number of channels unchanged) to obtain a feature map with dimensions of 80×80×256. Next, the obtained feature map is fused with the feature map output from the fifth layer of the backbone feature extraction module (adjusting the channels to 512 through a 1×1 convolution, and then performing element-wise addition) to obtain a feature map with dimensions of 80×80×512. Finally, a 3×3 convolution is performed on the feature map with dimensions of 80×80×512 to output a feature map with dimensions of 80×80×512.

[0155] The sixth layer is the final downsampling and element-wise fusion layer. Its input is the feature map with dimensions of 80×80×512 output from the fifth layer and the feature map output from the fifth layer of the backbone feature extraction module. This layer first performs a 3×3 convolution on the feature map with dimensions of 80×80×512 output from the fifth layer (downsampling to 40×40 from the input 3×3 convolution layer, keeping the number of channels unchanged) to obtain a feature map with dimensions of 40×40×512. Next, the obtained feature map is fused with the feature map output from the fifth layer of the backbone feature extraction module (adjusting the channels to 1024 through a 1×1 convolution, and then performing element-wise addition) to output a feature map with dimensions of 40×40×1024. Finally, a 3×3 convolution is performed on the obtained feature map with dimensions of 40×40×1024 to output a feature map with dimensions of 40×40×1024.

[0156] The seventh layer is the channel-weighted and context-modeling layer. Its input is the four-dimensional feature maps output from layers one through six. This layer first uses a 3×3 convolution to extract local spatial features from these four-dimensional feature maps, resulting in four-dimensional feature maps (with the same spatial resolution and number of channels as the original). Then, these four-dimensional feature maps are flattened into a sequence (converting the spatial structure into a one-dimensional sequence for input into the downstream self-attention mechanism), resulting in four-dimensional feature maps that retain local spatial feature information. Finally, a lightweight multi-head self-attention mechanism (LTM) is used to process the flattened four-dimensional feature maps. The feature maps of each dimension are processed to obtain four-dimensional feature-enhanced feature maps. Finally, the obtained four-dimensional feature-enhanced feature maps are input into a fully connected layer for normalization (the channel description vector is obtained through global average pooling, ReLU activation is used, and finally Sigmoid activation is used to obtain the channel weights; the channel weights are multiplied element-wise with the input feature map channels to dynamically adjust the response of each channel, enhance key modalities and important feature channels, and suppress redundant information). The final output is a fused feature map of four dimensions: P2 (320×320×128), P3 (160×160×256), P4 (80×80×512), and P5 (40×40×1024).

[0157] The detection head module of this invention takes as input four-dimensional feature maps P2 (320×320×128), P3 (160×160×256), P4 (80×80×512), and P5 (40×40×1024) output by the feature fusion module. The detection head module processes the feature map of each dimension using the corresponding detection head structure (the original P3~P5 detection head structure is extended, and a new P2 detection branch (stride=4) is added to improve the detection performance of small targets, which is specifically used to capture smaller hot spots, cracks and other micro-fault targets). Finally, it outputs a multimodal power plant image with defect category, bounding box coordinates, confidence score and fault category.

[0158] The advantage of step (2) above is that, based on YOLOv12, the GhostConv lightweight design is adopted to reduce the computational burden, a multi-scale dynamic feature extraction module, a multi-modal attention fusion module are added, and a P2 detection head is added to improve the recognition effect of small defects, thereby improving the ability of the improved YOLOv12 model to extract feature information.

[0159] The improved YOLOv12 model of this invention is trained according to the following steps:

[0160] (3-1) Collect multimodal power station image data (including high-resolution visible light images, infrared thermal imaging images and lidar point cloud images) obtained by UAV during power station inspection. Preprocess the multimodal power station image data to obtain preprocessed multimodal power station image data. Use manual annotation and automatic auxiliary tools to perform multi-label annotation (including target location, category and severity level) on the equipment fault targets in the multimodal power station image data to obtain dataset S (which is a multimodal, multi-label dataset). Divide the dataset into training set and test set in an 8:2 ratio.

[0161] Specifically, the preprocessing process in this step is exactly the same as that in step (1) above, and will not be repeated here.

[0162] (3-2) Initialize the parameters of the improved YOLOv12 model. The initial values ​​of the weight parameters are random values ​​output by a truncated normal distribution with a standard deviation of 0.1. The initial value of the bias parameter is set to 0. The initial learning rate is lr=0.0003. A step-wise learning strategy is adopted with a step size of stepsize=200 and a weight gamma=0.1, that is, the learning rate is multiplied by 0.1 every 200 epochs.

[0163] (3-3) For each sample in the training set obtained in step (3-1), input the sample into the first layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map corresponding to the sample with a dimension of 320×320×128.

[0164] (3-4) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 320×320×128 corresponding to the sample obtained in step (3-3) is input into the second layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 160×160×256 corresponding to the sample.

[0165] (3-5) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 160×160×256 corresponding to the sample obtained in step (3-4) is input into the third layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 80×80×512 corresponding to the sample.

[0166] (3-6) For each sample in the training set obtained in step (3-1), the feature map with a dimension of 80×80×512 corresponding to the sample obtained in step (3-5) is input into the fourth layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with a dimension of 40×40×1024 corresponding to the sample.

[0167] (3-7) For each sample in the training set obtained in step (3-1), the feature maps corresponding to the sample obtained in steps (3-3), (3-4), (3-5), and (3-6) are respectively input into the fifth layer of the backbone feature extraction module in the YOLOv12 improved model after initialization in step (2) to obtain the feature maps corresponding to the sample with dimensions of 320×320×128, 160×160×256, 80×80×512, and 40×40×1024.

[0168] (3-8) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 80×80×512 and 40×40×1024 obtained in steps (3-5) and (3-6) respectively are input into the first layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 80×80×512 corresponding to the sample.

[0169] (3-9) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 160×160×256 and 80×80×512 obtained in steps (3-4) and (3-8) respectively are input into the second layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 160×160×256 corresponding to the sample.

[0170] (3-10) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 320×320×128 and 160×160×256 obtained in steps (3-3) and (3-9) is input into the third layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 320×320×128 corresponding to the sample.

[0171] (3-11) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 320×320×128 and 320×320×128 obtained in steps (3-3) and (3-10) is input into the fourth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 160×160×256 corresponding to the sample;

[0172] (3-12) For each sample in the training set obtained in step (3-1), the feature maps with dimensions of 160×160×256 and 160×160×256 obtained in steps (3-7) and (3-11) are input into the fifth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimension of 80×80×512 corresponding to the sample.

[0173] (3-13) For each sample in the training set obtained in step (3-1), the feature map with dimensions of 80×80×512 and 80×80×512 obtained in steps (3-7) and (3-12) is input into the sixth layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the feature map with dimensions of 40×40×1024 corresponding to the sample.

[0174] (3-14) For each sample in the training set obtained in step (3-1), input the feature map corresponding to the sample obtained in steps (3-8) to (3-13) into the seventh layer of the feature fusion module in the YOLOv12 improved model after initialization in step (2) to obtain the fused feature map corresponding to the sample with dimensions of 320×320×128, 160×160×256, 80×80×512, and 40×40×1024.

[0175] (3-15) For each sample in the training set obtained in step (3-1), the feature map corresponding to the sample obtained in step (3-14) is input into the detection head module of the YOLOv12 improved model after initialization in step (2) to obtain the defect category, local bounding box coordinates and confidence score corresponding to the sample.

[0176] (3-16) For each sample in the training set obtained in step (3-1), based on the defect category corresponding to the sample obtained in step (3-15), the defect loss value corresponding to the sample is obtained by using the multi-label FocalLoss function. Based on the bounding box coordinates corresponding to the sample, the bounding box loss value corresponding to the sample is obtained by using the WIoUv2 function. Based on the confidence score corresponding to the sample, the confidence loss value corresponding to the sample is obtained by using the Softmax loss function. Based on the obtained defect loss value, bounding box loss value, and confidence loss value, the total loss corresponding to the sample is obtained.

[0177] To address the class imbalance problem in multi-label classification, this invention employs FocalLoss as the classification loss function for defect category prediction. The defect loss value for each sample is calculated using the following formula:

[0178]

[0179] Where C represents the total number of fault categories, This represents the true label of the i-th type of fault. This represents the predicted probability of the improved YOLOv12 model for the i-th type of fault. The importance coefficients represent the positive and negative samples of the i-th type of fault. This represents the modulation factor (usually set to 2), used to suppress the contribution of easily classified samples to the loss.

[0180] To more accurately assess the overlap between the predicted and ground truth bounding boxes, this invention employs the WIoUv2 loss function for regression error calculation. The loss value for the localization box corresponding to each sample is calculated using the following formula:

[0181]

[0182] in, Indicates the prediction box. Represents a real bounding box. This represents the overlap index output by the WIoUv2 loss function, with a value range of [0,1]. The larger the value, the higher the overlap.

[0183] For determining the existence of a target, this invention uses Softmax cross-entropy as the confidence loss function. The confidence loss value for each sample is calculated using the following formula:

[0184]

[0185] in, This represents the true label of the j-th type of fault. This represents the initial score of the YOLOv12 improved model for the j-th fault type.

[0186] The total loss of the final model is expressed as:

[0187]

[0188] in , , The weighting coefficients for each sub-loss are used to balance the contributions of each task during training, and the sum of the three is 1.

[0189] (3-17) For each sample in the training set obtained in step (3-1), the total loss corresponding to the sample obtained in step (3-16) is used to iteratively train the YOLOv12 improved model using gradient descent until the YOLOv12 improved model reaches the preset number of iterations (300 times in this invention), and the optimal parameters of the YOLOv12 improved model at this time are obtained, thereby obtaining the initially trained YOLOv12 improved model.

[0190] (3-18) Use the test set obtained in step (3-1) to test the YOLOv12 improved model initially trained in step (3-17) until the detection accuracy reaches the optimal level, so as to obtain the final trained YOLOv12 improved model.

[0191] The advantages of steps (3-1) to (3-18) above are that training uses multimodal power plant image data, which significantly improves the defect detection performance of power plant equipment. The combination of FocalLoss, WIoUv2, and Softmax loss functions effectively solves the problems of class imbalance and localization accuracy. Through hierarchical feature extraction and end-to-end training strategies, the improved YOLOv12 model can accurately identify the location, category, and severity of equipment faults.

[0192] Performance Comparison

[0193] To verify the performance advantages of the method of this invention in power plant inspection tasks, a comparative test was conducted with nine mainstream target detection methods on the dataset obtained in step (3-1) of this invention. Test metrics included frame rate (Frames Per Second, FPS), gigafloating-point operations per second (GFLOPs), number of model parameters, average precision (AP), and mean average precision (mAP), comprehensively evaluating aspects such as real-time performance, computational complexity, model size, and detection accuracy. Detailed experimental results are shown in Table 1.

[0194] Table 1 Performance comparison between the present invention and existing methods

[0195]

[0196] According to the experimental results in Table 1, the method of the present invention has significant advantages in target detection performance: the mAP reaches 90.1%, and the AP values ​​for the three categories of missing, crack and spot are 97.1%, 85.7% and 87.3% respectively, all of which are better than the 10 comparative methods, indicating that it still has relatively good detection accuracy and robustness in complex power plant environment.

[0197] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting faults in power station inspection by unmanned aerial vehicles based on improved YOLOv12 and multi-modal deep feature fusion, characterized in that, Comprise: (1) Obtain multi-modal power station image data from the power station field area by the unmanned aerial vehicle, and pretreat the multi-modal power station image data to obtain pretreated multi-modal power station image data; Specifically, first, the infrared thermal imaging image in the multi-modal power station image data fused with light, heat and space information is converted into a pseudo-visible light image by a pseudo-color mapping method to obtain an infrared thermal imaging image highlighting the visual features of high temperature areas; Then, the visible light image in the multi-modal power station image data fused with light, heat and space information is applied to CLAHE to enhance the local contrast, while introducing random Gamma correction, brightness disturbance and image sharpening enhancement methods to obtain a visible light image with clearer image features; After that, the depth map of the laser radar point cloud image in the multi-modal power station image data fused with light, heat and space information is denoised by median filtering, and the pixel value is normalized to obtain a normalized laser radar point cloud image; Finally, the infrared thermal imaging image highlighting the visual features of high temperature areas, the visible light image with clearer image features, and the normalized laser radar point cloud image are integrated to obtain multi-modal power station image data; (2) The pretreated multi-modal power station image data obtained in step (1) is input into a pre-trained YOLOv12 improved model to obtain the final detection result; The YOLOv12 improved model comprises an input module, a backbone feature extraction module, a feature fusion module and a detection head module; The input of the input module is a multi-modal power station image with a size of 640x640x3, and the input module performs normalization processing on the multi-modal power station image, and the output is a normalized multi-modal power station image; The specific structure of the backbone feature extraction module is: The first layer is a preliminary feature extraction layer, and the input of the preliminary feature extraction layer is the normalized multi-modal power station image obtained by the input module; the preliminary feature extraction layer first inputs the normalized multi-modal power station image into a standard convolution layer to obtain low-level features in the multi-modal power station image, and outputs a feature map with a dimension of 640x640x64; then, the feature map is further processed by a GhostConv module to output a feature map with a dimension of 640x640x64; subsequently, the 640x640x64 feature map is processed by a residual block Res for feature enhancement, and a feature map with a dimension of 640x640x64 is output; finally, the 640x640x64 feature map is down-sampled to output a feature map with a dimension of 320x320x128. The second layer is a multi-modal feature fusion and further feature extraction layer, and an input of the second layer is a feature map with a dimension of 320x320x128 obtained by the first layer. The multi-modal feature fusion and further feature extraction layer first inputs the feature map with a dimension of 320x320x128 output by the first layer into a convolution layer to output a feature map with a dimension of 320x320x128; then, the feature map with a dimension of 320x320x128 is processed by using a GhostConv module to output a feature map with a dimension of 320x320x128; then, the feature map with a dimension of 320x320x128 is processed by using a residual block to output a feature map with a dimension of 320x320x128; then, the feature map with a dimension of 320x320x128 is processed by using a multi-modal attention fusion module MMFM to output a feature map with a dimension of 320x320x128; finally, the feature map is down-sampled to finally output a feature map with a dimension of 160x160x256; The third layer is a deep feature extraction layer, and an input of the third layer is a feature map with a dimension of 160x160x256 obtained by the second layer. The deep feature extraction layer inputs the feature map with a dimension of 160x160x256 into a convolution layer to output a feature map with a dimension of 160x160x256; then, the feature map with a dimension of 160x160x256 is further processed by using a GhostConv module to output a feature map with a dimension of 160x160x256; then, the feature map with a dimension of 160x160x256 is processed by using a residual block to output a feature map with a dimension of 160x160x256; then, the feature map with a dimension of 160x160x256 is processed by using an MGFP module to output a feature map with a dimension of 80x80x512; finally, the feature map with a dimension of 80x80x512 is down-sampled to output a feature map with a dimension of 80x80x512; The fourth layer is a high-level feature extraction layer, and an input of the fourth layer is a feature map with a dimension of 80x80x512 output by the third layer. The high-level feature extraction layer first inputs the feature map into a convolution layer to output a feature map with a dimension of 80x80x512; then, the feature map with a dimension of 80x80x512 is processed by using a GhostConv module to output a feature map with a dimension of 80x80x512; then, the feature map with a dimension of 80x80x512 is processed by using a residual block to output a feature map with a dimension of 80x80x512; finally, the feature map with a dimension of 80x80x512 is down-sampled to finally output a feature map with a dimension of 40x40x1024; The fifth layer is an MADSE module, and an input of the fifth layer is each dimension feature map output by the first layer to the fourth layer in the backbone feature extraction module. The MADSE module processes each dimension feature map to finally output a feature map fused with multiple information. The structure hierarchy of the MMFM module is specifically as follows: The first layer of the MMFM module is a modal adaptive channel attention layer MCA, an input of which is a 320x320x128 dimensional feature map output by the multi-modal feature fusion and further feature extraction layer. The MCA layer adopts a squeeze-and-excitation SE attention mechanism composed of a global average pooling layer, two fully connected layers and a sigmoid activation function, weights important channels of each modality in the feature map to output three weighted feature maps with a dimension of 320x320x128; The second layer of the MMFM module is a modal interaction space attention layer MISA, an input of which is the three weighted feature maps with a dimension of 320x320x128 output by the MCA layer. The MISA layer concatenates the feature maps and generates a spatial attention map through a 3x3 convolution, and finally obtains a spatial attention map with a dimension of 320x320x1 The third layer of the MMFM module is an attention fusion module layer AGF, an input of which is the spatial attention map with a dimension of 320x320x1 output by the MISA layer. The AGF layer performs weighted fusion on the spatial attention map with a dimension of 320x320x1, and outputs a unified fused feature map with a dimension of 320x320x128; The fourth layer of the MMFM module is a residual connection and normalization layer RLN, an input of which is the unified fused feature map with a dimension of 320x320x128 output by the AGF layer. The RLN performs LayerNorm processing on the unified fused feature map with a dimension of 320x320x128. Then, the LayerNorm processing result is added to the original modal fusion feature through residual connection, and finally a feature map with a dimension of 320x320x128 is output; The structure of the MGFP module is as follows: The first layer of the MGFP module is a modal group feature splitting layer MGS, an input of which is a 160x160x256 dimensional feature map output by the residual block in the deep feature extraction layer. The MGS layer divides the multi-modal features of the feature map by channel dimension, and each group divides a plurality of sub-branches, and outputs a feature map with a dimension of 160x160x256; The second layer of the MGFP module is a pyramid dilated convolution extraction layer PDE, an input of which is the feature map with a dimension of 160x160x256 output by the MGS layer. The PDE layer groups the modal sub-features of the feature map and uses 3x3 convolution with different dilation rates to model the grouped results in parallel to obtain a pyramid feature map with a dimension of 160x160x256 under each modality; The third layer of the MGFP module is a cross-modal context fusion layer CCF, an input of which is the pyramid feature map with a dimension of 160x160x256 under each modality output by the PDE layer. The CCF layer first concatenates the pyramid feature map with a dimension of 160x160x256 under each modality, and then inputs it into a module composed of a 3x3 convolution layer and a GELU activation to obtain a feature map with more abstract and high-level features. Then the attention-guided cross-modal channel weighting mechanism is introduced to process the obtained feature maps with more abstract and high-level, and finally output the fusion feature map with the dimension of 160x160x256; The fourth layer of the MGFP module is a global residual fusion layer GRM, and the input of the GRM layer is the fusion feature map with the dimension of 160x160x256 output by the CCF layer. The GRM layer performs element-wise weighted fusion on the fusion feature map and the feature map with the dimension of 160x160x256 output by the residual block in the deep feature extraction layer, then balances the channel dimension of the weighted fusion result using 1x1 convolution, and then performs LayerNorm normalization processing, and finally outputs the feature map with the dimension of 160x160x256; The structure hierarchy of the MADSE module is specifically as follows: The first layer of the MADSE module is a multi-path dilated convolution layer DCP, and the input of the DCP layer is the feature maps with different dimensions output by the first layer to the fourth layer in the backbone feature extraction module. The DCP layer processes the input feature maps through three different dilated convolution paths to output the feature maps processed by dilated convolution; The second layer of the MADSE module is a modal saliency map generator layer MSG, and the input of the MSG layer is the feature maps processed by dilated convolution. The MSG layer performs salient feature activation processing on the feature maps to output the saliency map of each input modality; The third layer of the MADSE module is a spatial weight map fusion mechanism layer SWF, and the input of the SWF layer is the feature maps processed by dilated convolution output by the DCP layer and the saliency map of the input modality output by the MSG layer. The SWF layer processes the feature maps and the saliency map through Softmax normalization to generate a spatial weight map; The fourth layer of the MADSE module is a feature standardization and residual connection layer FSR, and the input of the FSR layer is the spatial weight map output by the SWF layer. The FSR layer first performs layer normalization processing on the spatial weight map, and then adds the layer normalized spatial weight map to the feature maps input by the first layer of the MADSE module through residual connection to output the feature maps; The input of the feature fusion module is the feature map with the dimension of 320x320x128 output by the first layer in the backbone feature extraction module, the feature map with the dimension of 160x160x256 output by the second layer in the backbone feature extraction module, the feature map with the dimension of 80x80x512 output by the third layer in the backbone feature extraction module, and the feature map with the dimension of 40x40x1024 output by the fourth layer in the backbone feature extraction module. The feature fusion module performs multi-scale feature fusion and enhancement processing operations on the four feature maps respectively to obtain four dimensionally fused feature maps P2, P3, P4, and P5; The structure hierarchy of the feature fusion module is specifically as follows: The first layer is an up-sampling and element-wise fusion layer, which inputs the feature maps output by the third layer and the fourth layer of the backbone feature extraction module. The layer first performs 1x1 convolution dimension reduction processing on the feature map output by the fourth layer of the backbone feature extraction module to obtain a feature map with a dimension of 40x40x512; then, performs up-sampling processing on the feature map to output a feature map with a dimension of 80x80x512; and then, performs element-wise addition fusion on the up-sampled feature map and the feature map output by the third layer of the backbone feature extraction module to output a feature map with a dimension of 80x80x512; The second layer is a re-up-sampling and element-wise fusion layer, which inputs the feature map with a dimension of 80x80x512 output by the first layer and the feature map output by the second layer of the backbone feature extraction module. The layer first performs 1x1 convolution dimension reduction processing on the feature map with a dimension of 80x80x512 output by the first layer to obtain a feature map with a dimension of 80x80x256; then, performs up-sampling processing on the feature map to output a feature map with a dimension of 160x160x256; and then, performs element-wise addition fusion on the up-sampled feature map and the feature map output by the second layer of the backbone feature extraction module to output a feature map with a dimension of 160x160x256; The third layer is a final up-sampling and element-wise fusion layer, which inputs the feature map with a dimension of 160x160x256 output by the second layer and the feature map output by the first layer of the backbone feature extraction module. The layer first performs 1x1 convolution dimension reduction processing on the feature map with a dimension of 160x160x256 output by the second layer to obtain a feature map with a dimension of 160x160x128; then, performs up-sampling processing on the feature map to output a feature map with a dimension of 320x320x128; and then, performs element-wise addition fusion on the up-sampled feature map and the feature map output by the first layer of the backbone feature extraction module to output a feature map with a dimension of 320x320x128; The fourth layer is a down-sampling and element-wise fusion layer, which inputs the feature map with a dimension of 320x320x128 output by the third layer and the feature map with a dimension of 320x320 output by the first layer of the backbone feature extraction module. The layer first performs 3x3 convolution processing with a step of 2 on the feature map output by the first layer of the backbone feature extraction module to obtain a feature map with a dimension of 160x160x128; then, performs fusion on the obtained feature map and the feature map with a dimension of 320x320x128 output by the third layer to output a feature map with a dimension of 160x160x256; and finally, performs 3x3 convolution processing on the feature map with a dimension of 160x160x256 to output a feature map with a dimension of 160x160x256. The fifth layer is a re-downsampling and element-by-element fusion layer, which inputs the feature map with the dimension of 160x160x256 output by the fourth layer and the feature map output by the fifth layer of the backbone feature extraction module. The layer first performs 3x3 convolution processing on the feature map with the dimension of 160x160x256 output by the fourth layer to obtain a feature map with the dimension of 80x80x256. Then, the obtained feature map is fused with the feature map output by the fifth layer of the backbone feature extraction module to obtain a feature map with the dimension of 80x80x512. Finally, 3x3 convolution processing is performed on the feature map with the dimension of 80x80x512 to output a feature map with the dimension of 80x80x512. The sixth layer is a final down-sampling and element-by-element fusion layer, which inputs the feature map with the dimension of 80x80x512 output by the fifth layer and the feature map output by the fifth layer of the backbone feature extraction module. The layer first performs 3x3 convolution processing on the feature map with the dimension of 80x80x512 output by the fifth layer to obtain a feature map with the dimension of 40x40x512. Then, the obtained feature map is fused with the feature map output by the fifth layer of the backbone feature extraction module to output a feature map with the dimension of 40x40x1024. Finally, 3x3 convolution processing is performed on the obtained feature map with the dimension of 40x40x1024 to output a feature map with the dimension of 40x40x1024. The seventh layer is a channel weighting and context modeling layer, which inputs four kinds of feature maps output by the first to sixth layers. The layer first performs local spatial feature extraction on the four kinds of feature maps by a 3x3 convolution to obtain four kinds of feature maps. Then, the four kinds of feature maps are flattened into a sequence to obtain four kinds of feature maps that retain local spatial feature information. Subsequently, a lightweight multi-head self-attention mechanism LTM is used to process the flattened four kinds of feature maps to obtain four kinds of specific feature enhancement feature maps. Finally, the obtained four kinds of specific feature enhancement feature maps are input into a fully connected layer for normalization processing, and finally four kinds of fusion feature maps P2, P3, P4, and P5 are output. A detection head module inputs the four kinds of feature maps P2, P3, P4, and P5 output by the feature fusion module. The detection head module uses corresponding detection head structures to process each kind of feature map, and finally outputs a multi-modal power station image with defect categories, positioning box coordinates, confidence scores, and fault categories.

2. The unmanned aerial vehicle power station inspection fault detection method based on improved YOLOv12 and multi-modal deep feature fusion according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1-1) The unmanned aerial vehicle obtains multi-modal power station data from the power station area, and adds a timestamp to the multi-modal power station data to obtain multi-modal power station data with a timestamp added; (1-2) The multi-modal power station data with a timestamp added obtained in step (1-1) is subjected to time sequence alignment processing to obtain once-processed multi-modal power station image data; (1-3) Obtain image modal data from the once-processed multi-modal power station data obtained in step (1-2), and perform geometric correction processing on the obtained image modal data using a feature matching combined with a projection transformation method to obtain twice-processed multi-modal power station image data; (1-4) The twice-processed multi-modal power station image data obtained in step (1-3) is sequentially subjected to spatial projection and alignment processing to obtain thrice-processed multi-modal power station image data; (1-5) The twice-processed multi-modal power station image obtained in step (1-3) and the thrice-processed multi-modal power station image obtained in step (1-4) are stacked in channels to obtain multi-modal power station image data fused with illumination, heat and spatial information; (1-6) The multi-modal power station image data fused with illumination, heat and spatial information obtained in step (1-5) is processed using a modal adaptive enhancement and optimization method to obtain quadruple-processed multi-modal power station image data; (1-7) The quadruple-processed multi-modal power station image data obtained in step (1-6) is processed by a data enhancement technique to obtain pre-processed multi-modal power station image data.

3. The unmanned aerial vehicle power station inspection fault detection method based on improved YOLOv12 and multi-modal deep feature fusion according to claim 1 or 2, characterized in that Step (1-3) is specifically that, for visible light images and infrared thermal imaging images, this step is to use a scale-invariant feature transform (SIFT) or a directed scale-invariant feature transform combined with a rotation feature descriptor (ORB) method to perform geometric correction processing; for laser radar point cloud images, a fast point feature histogram (FPFH) combined with a random sample consensus (RANSAC) method is used to perform geometric correction processing; Step (1-4) is specifically that, first, an iterative closest point (ICP) algorithm is used to perform three-dimensional spatial alignment processing on the laser radar point cloud images and visible light images in the twice-processed multi-modal power station image data obtained in step (1-3) to obtain processed laser radar point cloud images and visible light images; Then, based on the results of the alignment processing, a perspective projection is performed to generate a depth map; at the same time, the infrared thermal imaging images and visible light images in the twice-processed multi-modal power station image data obtained in step (1-3) are subjected to projection mapping and interpolation alignment processing to obtain processed infrared thermal imaging images; finally, the processed visible light images, infrared thermal imaging images, laser radar point cloud images, and generated depth map are integrated to obtain thrice-processed spatially aligned multi-modal power station image data sets; The data enhancement technique in step (1-7) includes but is not limited to one or any combination of any number of Mosaic puzzle enhancement methods, MixUp image fusion enhancement techniques, random scale transformation, and color disturbance.

4. The unmanned aerial vehicle power station inspection fault detection method based on improved YOLOv12 and multi-modal deep feature fusion according to claim 3, characterized in that, The improved YOLOv12 model is trained according to the following steps: (3-1) Collecting multi-modal power station image data obtained by the unmanned aerial vehicle in the process of power station inspection, preprocessing the multi-modal power station image data to obtain preprocessed multi-modal power station image data, performing multi-label annotation on the equipment fault targets in the multi-modal power station image data through manual labeling and automatic auxiliary tools to obtain a data set S, and dividing the data set into a training set and a test set according to a ratio of 8:2; (3-2) Initializing parameters of the improved YOLOv12 model to obtain an initialized improved YOLOv12 model; (3-3) For each sample in the training set obtained in step (3-1), inputting the sample into a first layer of the main feature extraction module of the initialized improved YOLOv12 model in step (2) to obtain a feature map corresponding to the sample and having a dimension of 320×320×128; (3-4) For each sample in the training set obtained in step (3-1), inputting the feature map corresponding to the sample and having a dimension of 320×320×128 obtained in step (3-3) into a second layer of the main feature extraction module of the initialized improved YOLOv12 model in step (2) to obtain a feature map corresponding to the sample and having a dimension of 160×160×256; (3-5) For each sample in the training set obtained in step (3-1), inputting the feature map corresponding to the sample and having a dimension of 160×160×256 obtained in step (3-4) into a third layer of the main feature extraction module of the initialized improved YOLOv12 model in step (2) to obtain a feature map corresponding to the sample and having a dimension of 80×80×512; (3-6) For each sample in the training set obtained in step (3-1), inputting the feature map corresponding to the sample and having a dimension of 80×80×512 obtained in step (3-5) into a fourth layer of the main feature extraction module of the initialized improved YOLOv12 model in step (2) to obtain a feature map corresponding to the sample and having a dimension of 40×40×1024; (3-7) For each sample in the training set obtained in step (3-1), inputting the feature maps corresponding to the sample and having dimensions of 320×320×128, 160×160×256, 80×80×512 and 40×40×1024 obtained in steps (3-3), (3-4), (3-5) and (3-6) respectively into a fifth layer of the main feature extraction module of the initialized improved YOLOv12 model in step (2) to obtain the feature maps corresponding to the sample and having dimensions of 320×320×128, 160×160×256, 80×80×512 and 40×40×1024 respectively; (3-8) For each sample in the training set obtained in step (3-1), inputting the feature maps corresponding to the sample and having dimensions of 80×80×512 and 40×40×1024 obtained in steps (3-5) and (3-6) respectively into a first layer of the feature fusion module of the initialized improved YOLOv12 model in step (2) to obtain a feature map corresponding to the sample and having a dimension of 80×80×512; (3-9) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-4) and (3-8) into the second layer of the feature fusion module in the YOLOv12 improved model initialized in step (2) respectively, to obtain the feature map corresponding to the sample with the dimension of 160x160x256; (3-10) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-3) and (3-9) into the third layer of the feature fusion module in the YOLOv12 improved model initialized in step (2), to obtain the feature map corresponding to the sample with the dimension of 320x320x128; (3-11) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-3) and (3-10) into the fourth layer of the feature fusion module in the YOLOv12 improved model initialized in step (2), to obtain the feature map corresponding to the sample with the dimension of 160x160x256; (3-12) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-7) and (3-11) into the fifth layer of the feature fusion module in the YOLOv12 improved model initialized in step (2), to obtain the feature map corresponding to the sample with the dimension of 80x80x512; (3-13) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-7) and (3-12) into the sixth layer of the feature fusion module in the YOLOv12 improved model initialized in step (2), to obtain the feature map corresponding to the sample with the dimension of 40x40x1024; (3-14) For each sample in the training set obtained in step (3-1), input the feature maps corresponding to the sample obtained in steps (3-8) to (3-13) into the seventh layer of the feature fusion module in the YOLOv12 improved model initialized in step (2), to obtain the fusion feature maps corresponding to the sample with the dimensions of 320x320x128, 160x160x256, 80x80x512, 40x40x1024; (3-15) For each sample in the training set obtained in step (3-1), input the feature map corresponding to the sample obtained in step (3-14) into the detection head module of the YOLOv12 improved model initialized in step (2) to obtain the defect class, the bounding box coordinates, and the confidence score corresponding to the sample; (3-16) For each sample in the training set obtained in step (3-1), according to the defect class corresponding to the sample obtained in step (3-15), and using a multi-label FocalLoss function to obtain a defect loss value corresponding to the sample, according to the bounding box coordinates corresponding to the sample, and using a WIoU v2 function to obtain a bounding box loss value corresponding to the sample, according to the confidence score corresponding to the sample, and using a Softmax loss function to obtain a confidence loss value corresponding to the sample, and according to the obtained defect loss value, bounding box loss value, and confidence loss value to obtain a total loss corresponding to the sample; (3-17) For each sample in the training set obtained in step (3-1), the total loss corresponding to the sample obtained in step (3-16) is used to iteratively train the YOLOv12 improved model using gradient descent until the YOLOv12 improved model reaches a predetermined number of iterations, and the optimal parameters of the YOLOv12 improved model at this time are obtained, thereby obtaining a preliminarily trained YOLOv12 improved model; (3-18) Test the YOLOv12 improved model preliminarily trained in step (3-17) using the test set obtained in step (3-1) until the detection accuracy obtained reaches the optimal value, thereby obtaining a finally trained YOLOv12 improved model.

5. The unmanned aerial vehicle power station inspection fault detection method based on improved YOLOv12 and multi-modal deep feature fusion according to claim 4, characterized in that: The initialization process of step (3-2) is as follows: the initial value of the weight parameter is a random value output by a truncated normal distribution with a standard deviation of 0.1, the initial value of the bias parameter is set to 0, the initial learning rate lr=0.0003, a step learning strategy is adopted, the step size stepsize=200, and the weight gamma=0.1, i.e. the learning rate is multiplied by 0.1 every 200 rounds; The defect loss value is calculated using the following formula: ； wherein C represents the total number of fault categories, represents the true label of the i-th fault category, represents the prediction probability of the i-th fault category by the YOLOv12 improved model, represents the importance coefficient of the positive and negative samples of the i-th fault category, represents a modulation factor for suppressing the contribution of easily classified samples to the loss, and i∈[1, C]; The bounding box loss value is calculated using the following formula: ； wherein, represents a prediction box, represents a real box; represents the overlap index output by the WIoU v2 loss function, with a value range of [0, 1], and the greater the value, the higher the overlap. The confidence loss value corresponding to the sample is calculated using the following formula: ； wherein, denotes the true label of the jth fault type, denotes the initial score of the YOLOv12 improved model for the jth fault type, and j∈[1, C]. The total loss of the final model is represented as: ； wherein , and are weighting coefficients for the defect loss, the bounding box loss, and the confidence loss, respectively, and the sum of the three is 1.

6. An unmanned aerial vehicle power station inspection fault detection system based on improved YOLOv12 and multi-modal deep feature fusion, which is realized based on the unmanned aerial vehicle power station inspection fault detection method based on improved YOLOv12 and multi-modal deep feature fusion in claim 1, characterized in that, The unmanned aerial vehicle power station inspection fault detection system comprises: A first module configured to obtain multi-modal power station image data from a power station area by an unmanned aerial vehicle, and to pre-process the multi-modal power station image data to obtain pre-processed multi-modal power station image data; A second module configured to input the pre-processed multi-modal power station image data obtained by the first module into a pre-trained YOLOv12 improved model to obtain a final detection result.

Images