Electric tower wire defect detection method and system based on multi-mode cross-attention mechanism

Abstract

The invention discloses an electric tower wire defect detection method and system based on a multi-mode cross-attention mechanism, the method comprises the steps of inputting preprocessed three-mode data into a constructed and trained electric tower wire defect detection model to obtain an electric tower wire defect detection result, the electric tower wire defect detection model is based on a YOLO detection framework, and the electric tower wire defect detection result is obtained. A multi-modal feature extraction network is adopted to replace an original single-modal backbone network of the YOLO and is used for extracting features from an RGB image, a thermal infrared image and a depth image, and a multi-scale cross-modal fusion module is introduced into a Neck part of the YOLO to replace original splicing Concat operation and is used for fusing multi-modal features; a modal specificity auxiliary supervision strategy is introduced, an auxiliary loss function is added before YOLO T / D branch fusion, and the quality of feature extraction is improved. According to the method, multi-modal feature self-adaption and selective deep fusion can be realized, so that the detection accuracy, robustness and multi-modal cooperation efficiency are improved.

Description

Technical Field

[0001] This invention relates to the detection of defects in power towers and power lines, and more specifically to a method and system for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism. Background Technology

[0002] Existing image recognition-based defect detection methods mostly use a single RGB image as input to perform target detection and defect identification on power tower components (such as bolts, clamps, insulators, crossarms, etc.). However, compared with target detection tasks in general scenarios, transmission line scenarios present the following unique challenges: First, defect targets are usually small in size, such as loose bolts or deformations, which are difficult to distinguish from the background; second, the inspection environment is complex and variable, with the background potentially containing interference information such as mountains, forests, and buildings, and lighting conditions are easily affected by weather and time; third, the visual features of some defects are not obvious, such as heat caused by poor contact of conductor clamps, temperature rise caused by internal equipment faults, early corrosion or slight deformation of metal components, which are not clearly represented in RGB images. Therefore, detection methods based on a single RGB modality suffer from limited information dimensions, making it difficult to meet the high requirements of power line inspection reliability.

[0003] To obtain more comprehensive equipment status information, multimodal sensing technology is increasingly being applied in the field of power transmission line inspection. Common multimodal data include thermal infrared and depth images: thermal infrared images can reflect the temperature distribution of equipment, effectively identifying latent defects such as poor contact and conductive loss that are accompanied by heat generation, and are unaffected by changes in visible light; depth images can provide spatial structure and deformation information of the scene, which is of great significance for identifying corrosion, defects, and structural anomalies. The fusion of RGB, thermal infrared, and depth information can achieve more comprehensive defect characterization, and has therefore become a research focus in recent years. Existing multimodal fusion detection methods mainly include three categories:

[0004] 1. Early fusion methods: These methods directly concatenate data from different modalities during the model input stage and then input them into a single network for detection. While structurally simple, these methods do not consider the physical differences between modalities, easily leading to feature interference and making it difficult to achieve effective cross-modal representation.

[0005] 2. Late-stage fusion method: Independent detection models are built for different modalities, and the results are fused at the output decision layer. This method ignores the deep semantic correlation of multimodal features, which can easily lead to the loss of complementary information.

[0006] 3. Deep fusion method: Fusion is performed in the intermediate layer of network feature extraction (such as feature concatenation or weighted addition). Although the fusion effect is better than the previous two, it generally suffers from fixed fusion strategies and weak adaptability to the scene. It cannot dynamically adjust the contribution weight of each modality feature according to different defect types and input content, and it is easy to introduce redundant features or even noise, which affects the detection performance.

[0007] In summary, existing multimodal defect detection methods still have significant shortcomings in their fusion strategies: they are either too simplistic and crude, leading to feature contamination, or they lack deep interaction, resulting in insufficient exploration of complementary relationships between modalities, making it difficult to achieve efficient and reliable multimodal information fusion. Therefore, how to achieve adaptive and selective effective fusion between different modalities and fully leverage the advantages of each modality remains a key problem that urgently needs to be solved. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism. This method can achieve multimodal feature adaptation and selective deep fusion to improve the accuracy, robustness and multimodal collaborative efficiency of detection, thereby overcoming the technical shortcomings of existing methods in terms of feature contamination, insufficient information utilization and rigid fusion strategies.

[0009] The technical solution to achieve the purpose of this invention is as follows:

[0010] A method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism, comprising:

[0011] Step S1: Acquire RGB image, thermal infrared image and depth image three-modal data of the power tower and power line scene to be detected, and perform preprocessing;

[0012] Step S2: Input the preprocessed trimodal data into the constructed and trained power tower and power line defect detection model to obtain the power tower and power line defect detection results. The power tower and power line defect detection model is based on the YOLO detection framework and adopts a multimodal feature extraction network to replace the original single-modal backbone network of YOLO. Features are extracted from RGB images, thermal infrared images, and depth images respectively. In the Neck part of YOLO, a multi-scale cross-modal fusion module is introduced to replace the original concat operation for fusing multimodal features. A modality-specific auxiliary supervision strategy is introduced to add an auxiliary loss function before the YOLO T / D branch fusion to improve the quality of feature extraction.

[0013] Furthermore, the preprocessing includes size alignment, geometric calibration, and pixel normalization operations.

[0014] Furthermore, the image processing process of the power tower and power line defect detection model is as follows:

[0015] The preprocessed trimodal data is input into a multimodal feature extraction network to extract semantic features, temperature anomaly features, and structural deformation features, respectively.

[0016] The extracted three-modal features are combined according to scale to construct multimodal feature groups with scales P3, P4, and P5 respectively;

[0017] The multimodal features at three scales are input into the multi-scale cross-modal fusion module. The RGB features are used as the guiding modality and cross-modal attention interactions are performed with the thermal infrared and depth modalities respectively. The fused and enhanced multimodal features are then output.

[0018] The enhanced multimodal characteristics are input into the detection head, and the detection results of power tower and power line defects are output after classification and regression prediction.

[0019] Furthermore, the multimodal feature extraction network adopts an asymmetric decoupled multimodal backbone network, SwinTransforme, ConvNeXt, or MobileNet structure.

[0020] Furthermore, the multimodal feature extraction network employs an asymmetric decoupled multimodal backbone network, which includes an RGB backbone branch, a thermal infrared T branch, and a depth D branch. The RGB backbone branch uses the YOLO backbone network structure to construct the RGB feature extractor. Multi-scale semantic features are obtained; the thermal infrared T branch uses a lightweight network as a thermal infrared feature extractor. It is used to extract information reflecting temperature differences and hotspot anomalies; the depth D branch employs a lightweight depth feature extraction network. Extract structural morphology-related features.

[0021] Furthermore, the multi-scale cross-modal fusion module employs attention mechanisms, gated fusion, weighted splicing, or a Transformer-based cross-modal feature interaction method.

[0022] Furthermore, the multi-scale cross-modal fusion module employs an attention mechanism, and the process of fusing input features is as follows:

[0023] Feature flattening and dimensionality alignment: trimodal feature maps at the same scale First through Convolution performs channel mapping to ensure they have the same embedding dimension. Then, flatten each feature map;

[0024] Perform linear transformations on the flattened feature matrices to generate query, key, and value vectors, as follows:

[0025] ;

[0026] in, For learnable parameters, These are the flattened RGB, thermal infrared, and depth feature matrices, respectively. This represents the query vector generated by the RGB modality; Feature index representing thermal infrared modes; Feature indexes representing deep modalities; Characteristic content representing thermal infrared modes; Feature content representing deep modalities;

[0027] Cross-modal attention calculation: RGB is used to guide two cross-modal attention calculations with thermal infrared and depth modes respectively;

[0028] When fusing three-modal features, the fusion is performed by convolution after splicing.

[0029] Furthermore, the two cross-modal attention calculations, guided by RGB and respectively by thermal infrared and depth modes, specifically include:

[0030] (1) RGB to Thermal cross-modal attention:

[0031] The Attention weights are calculated as follows:

[0032] ;

[0033] in For activation function, The dimension of the thermal infrared bond feature is represented by T, which represents the transpose of the matrix.

[0034] By weighting the thermal infrared Value vector, supplementary information extracted from the thermal infrared modes is obtained:

[0035] ;

[0036] (2) RGB to Depth cross-modal attention:

[0037] The attention mechanism for RGB-guided deep features is as follows:

[0038] ;

[0039] The supplementary information extracted from the depth modes is as follows:

[0040] .

[0041] Furthermore, the total loss function of the power tower and power line defect detection model By a main loss and two auxiliary losses and Weighted composition, main loss Before feature fusion in the T and D branches, auxiliary detection heads are introduced and auxiliary losses are applied to the main detection head.

[0042] A power tower and power line defect detection system based on a multimodal cross-attention mechanism includes:

[0043] The image acquisition and processing unit acquires RGB image, thermal infrared image and depth image three-modal data of the power tower and power line scene to be detected, and performs preprocessing.

[0044] The detection unit inputs preprocessed trimodal data into the constructed and trained power tower and power line defect detection model to obtain the power tower and power line defect detection results. The power tower and power line defect detection model is based on the YOLO detection framework and adopts a multimodal feature extraction network to replace the original single-modal backbone network of YOLO. It is used to extract features from RGB images, thermal infrared images, and depth images respectively. In the Neck part of YOLO, a multi-scale cross-modal fusion module is introduced to replace the original concat operation for fusing multimodal features. A modality-specific auxiliary supervision strategy is introduced to add an auxiliary loss function before the YOLO T / D branch fusion to improve the quality of feature extraction.

[0045] Compared with existing single-modal or fixed fusion methods, this invention has significant technical advantages in terms of detection accuracy, feature representation ability, and model robustness, specifically manifested as follows:

[0046] 1. Improved effectiveness of feature extraction

[0047] By designing an asymmetric decoupled backbone, this invention extracts features independently from three modalities: RGB, thermal infrared, and depth. This structure avoids early modal feature contamination while preserving their respective semantic and physical properties, allowing each modality to fully leverage its advantages. Compared to traditional unified backbone or spliced ​​input structures, this invention achieves higher feature resolution and cleaner modal representations in scenarios involving thermal anomalies and structural defects.

[0048] 2. Adaptive Enhancement for Cross-Modal Fusion

[0049] The multi-scale cross-modal attention fusion module (CMFM) proposed in this invention employs a cross-modal query mechanism to achieve "on-demand retrieval" of thermal infrared and depth modalities from RGB modalities. During the fusion process, adaptive information selection and dynamic enhancement at the feature level are achieved through cross-modal attention computation. This mechanism can automatically adjust the fusion weights of different modalities according to the input scene, effectively avoiding modal interference and information redundancy caused by traditional fixed fusion methods, thereby significantly improving the robustness of the model in tasks involving complex backgrounds, illumination changes, and the detection of minute defects.

[0050] 3. Efficient fusion of multi-scale information

[0051] By embedding a CMFM module within the feature pyramid structure, multi-scale features can be bidirectionally transferred between upper and lower layers. Through top-down and bottom-up feature interactions, this invention achieves the joint expression of local details and global structural information, taking into account both the detection needs of minor defects (such as loose bolts) and large-scale structural anomalies (such as crossarm deformation).

[0052] 4. Through the above-described structural improvements and fusion strategies, this invention achieves efficient complementarity and adaptive fusion of multimodal features. Under multi-source input conditions, it can maintain detection stability in different environments. Compared with traditional RGB single-modal detection algorithms, the average accuracy is significantly improved, while reducing false detection and false negative rates. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the overall process of the power tower and power line defect detection method based on the multimodal cross-attention mechanism of the present invention.

[0054] Figure 2 This is a diagram of the overall architecture of MCA-YOLO proposed in this invention.

[0055] Figure 3 This is a diagram of the asymmetric decoupling backbone network structure proposed in this invention.

[0056] Figure 4 This is a diagram of the multi-scale cross-modal attention fusion module proposed in this invention. Detailed Implementation

[0057] Example 1

[0058] This invention provides a method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism. It aims to utilize the complementary characteristics of three modal data types—RGB (visible light), thermal infrared, and depth—to achieve high-precision detection of defects in power towers and power lines. The overall process of this invention is as follows: Figure 1 As shown, it includes:

[0059] Step S1: Multimodal data acquisition and preprocessing

[0060] Acquire RGB images, thermal infrared images, and depth images of the power tower and power line scene to be inspected, denoted as follows: ∈ , and .

[0061] Since the imaging mechanisms and coordinate systems of different modalities are different, this step performs preprocessing operations such as size alignment, geometric calibration, and pixel normalization on the three modal inputs to ensure that the three modalities can achieve spatial consistency and comparability.

[0062] After this step, the aligned trimodal input data is output. .

[0063] Step S2: Trimodal Feature Extraction

[0064] The preprocessed RGB, thermal infrared, and depth images are input into the three branches of the asymmetric decoupled backbone network, respectively, to extract semantic features, temperature anomaly features, and structural deformation features.

[0065] The network employs a three-branch structure, each corresponding to an RGB feature extractor. Thermal infrared feature extractor and deep feature extractor The three operate independently and do not share parameters with each other.

[0066] The output trimodal multiscale feature set is as follows:

[0067]

[0068] in, These are the flattened RGB, thermal infrared, and depth feature matrices, respectively. ; These represent feature maps at three scales (downsampling ratios of 8x, 16x, and 32x, respectively).

[0069] Step S3: Multi-scale feature construction

[0070] The three-modal features extracted from S2 are combined according to scale to construct multimodal feature groups with scales P3, P4, and P5 respectively:

[0071]

[0072] This step provides input for subsequent cross-modal attention fusion, ensuring that different modalities are aligned and fused on the same scale.

[0073] Step S4: Cross-modal attention fusion

[0074] The multimodal features at the three scales in step S3 are input into the Cross-Modality Fusion Module (CMFM), and the RGB features are used as the guiding modality to perform cross-modal attention interactions with the thermal infrared and depth modalities, respectively.

[0075] The CMFM module enables RGB features to "query" supplementary information related to them in thermal infrared and depth modes "on demand" through a cross-modal query mechanism, thereby improving the feature representation capability of the target region.

[0076] After this step, the fused and enhanced multimodal features are output:

[0077]

[0078] Step S5: Multi-scale feature enhancement and fusion

[0079] The fused feature set obtained in step S4 is input into the feature pyramid structure (Neck module in YOLO), and top-down and bottom-up feature transfer and fusion operations are performed. Through cross-layer feature interaction, the multi-scale feature representation capability is further enhanced, enabling the model to detect both micro-defects and macro-structural defects.

[0080] Output the enhanced multi-scale feature map:

[0081]

[0082] Step S6: Defect Detection Output

[0083] The enhanced features output from step S5 are input into the detection head (YOLO Head). After classification and regression prediction, the detection results, including the category, location bounding box, and confidence level of the power tower and power line defects, are output.

[0084]

[0085] in, It includes the category label, location coordinates, and confidence information for each detection box.

[0086] Figure 2This paper illustrates the overall architecture of the proposed multimodal cross-attention mechanism-based power tower and power line defect detection model (MMCA-YOLO). This architecture extends the YOLO detection framework with multimodal capabilities by introducing an asymmetric decoupled multimodal backbone network and a cross-modal attention fusion module, achieving deep fusion and feature enhancement of multimodal information. This architecture is specifically extended to the general YOLO detection framework (YOLO11m). To address the limitations of traditional single-modal detectors in power tower defect detection, this invention primarily makes the following three improvements:

[0087] (1) An asymmetric decoupled multimodal backbone network was adopted to replace the original single-modal backbone of YOLO, which was used to extract features from the three modes of RGB, thermal infrared (T) and depth (D).

[0088] (2) In the Neck (feature pyramid) part of the YOLO framework, a multi-scale cross-modal attention fusion module (CMFM) is introduced to replace the original simple concat operation, which is used to intelligently fuse multimodal features.

[0089] (3) A modality-specific auxiliary supervision strategy was introduced, which added a dedicated auxiliary loss function before T / D branch fusion to ensure high-quality feature extraction.

[0090] Compared with existing YOLO detection methods based on a single RGB modality, the architecture of this invention has the following significant differences and advantages:

[0091] (1) Multimodal information input is adopted to make up for the blind spots of RGB mode recognition in scenarios such as lighting, weak texture and complex background;

[0092] (2) Improve the targeting and efficiency of feature extraction for different modes by using asymmetric decoupling of the backbone structure, and avoid interference between modes;

[0093] (3) Introduce a cross-modal attention mechanism in the feature extraction stage to achieve deep fusion and enhance the model’s ability to express latent defect features.

[0094] Figure 3 A schematic diagram of the asymmetric-decoupled backbone network described in this invention is shown. This network includes three parallel feature extraction branches, which are used to process the data streams of RGB mode, thermal infrared mode, and depth mode, respectively. The design goal of this structure is to ensure the independence and specificity of feature extraction for each mode, avoiding modal interference before fusion.

[0095] The specific structure is explained below:

[0096] 1. RGB Main Branch: Semantic Feature Extraction

[0097] RGB images contain the richest scene semantics, texture, and color features; therefore, this invention considers the RGB branch as the "main branch." This branch uses the standard YOLO backbone network structure to construct the RGB feature extractor. To obtain multi-scale semantic features.

[0098] Given an input RGB image Its output characteristics are as follows:

[0099]

[0100] in, The multi-scale semantic feature maps are respectively downsampled by 8 times, 16 times and 32 times to the input image.

[0101] The RGB features extracted from this branch will serve as the "Query" features in subsequent cross-modal fusion.

[0102] 2. Thermal Infrared T-branch: Physical Thermal Feature Extraction

[0103] Thermal infrared images Although it lacks rich semantic information, it contains important physical quantities reflecting abnormal temperatures and hotspot distribution in power towers and wires. This invention employs a lightweight network as a thermal infrared feature extractor. This is to avoid texture bias interfering with feature learning.

[0104] The output features are:

[0105]

[0106] The thermal infrared branch focuses on extracting information that reflects temperature differences and hotspot anomalies, in order to compensate for latent defect features that cannot be observed in the RGB mode.

[0107] 3. Deep D-branch: Structural geometric feature extraction

[0108] Depth image It can reflect three-dimensional information such as spatial deformation and structural damage of power tower components. This invention employs a lightweight deep feature extraction network. Extract structural morphology-related features.

[0109] The output features are:

[0110]

[0111] The deep branch focuses on extracting 3D structural and deformation features to enhance the model's ability to identify structural defects such as deformation, fracture, and tilt.

[0112] This asymmetric decoupled three-branch structure has the following advantages:

[0113] Features do not contaminate each other: The RGB / T / D branches learn their own modal features independently, avoiding feature interference between modalities caused by premature fusion;

[0114] Preserving the advantages of modal characteristics: RGB provides semantic background information, T is used to detect temperature anomalies, and D is used to identify structural deformation;

[0115] Better computational efficiency: Only the RGB branch uses a deep network, while T and D use a lightweight network, which reduces the overall computational complexity.

[0116] The trimodal multiscale features output by this backbone network will serve as the cross-modal attention fusion module (see...). Figure 4 Input of ).

[0117] Figure 4 The diagram illustrates the structure of the Cross-Modality Fusion Module (CMFM) of this invention. The core idea of ​​this module is to extract the most relevant supplementary information to the RGB features from thermal infrared and depth features respectively at each scale (P3, P4, P5) using RGB modal features as a guide, thereby achieving intermodal correlation modeling and feature enhancement.

[0118] For ease of understanding, the following text will use the term scale. Taking this as an example, we will provide a detailed explanation of this module.

[0119] 1. Feature flattening and dimension alignment

[0120] Three-modal feature maps at the same scale First through Convolution performs channel mapping to ensure they have the same embedding dimension. Then, each feature map is flattened to obtain the following dimensions:

[0121]

[0122] in, The number of spatial locations in the feature map. For channel dimensions.

[0123] This step ensures that attention calculations are performed on all three modes in the same vector space.

[0124] 2. Q / K / V Vector Generation

[0125] The flattened feature matrices are subjected to linear transformations to generate query, key, and value vectors. These vectors are then expressed using right multiplication by the weight matrix, as follows:

[0126]

[0127] in, For learnable parameters, This represents the query vector generated by the RGB modality; Feature index representing thermal infrared modes; Feature indexes representing deep modalities; Characteristic content representing thermal infrared modes; The feature content representing the deep modality.

[0128] At this point, the RGB modal features become the query for the guiding modality, while the thermal infrared and depth features provide the key and value for retrieval, respectively.

[0129] 3. Cross-modal attention computation

[0130] This invention employs RGB to guide two cross-modal attention calculations with thermal infrared and depth modes, respectively.

[0131] (1) RGB to Thermal cross-modal attention

[0132] The attention weights are calculated as follows:

[0133]

[0134] in For activation function, Dimensions representing the thermal infrared bond features; This indicates the numerical scale used to balance the dot product between RGB and Thermal modes, preventing attention bias.

[0135] By weighting the thermal infrared Value vector with this weight, supplementary information extracted from the thermal infrared modes is obtained:

[0136]

[0137] (2) RGB to Depth cross-modal attention

[0138] Similarly, the Attention mechanism for RGB-guided deep features is as follows:

[0139]

[0140] Obtain supplementary information extracted from the depth modes:

[0141]

[0142] Through the above mechanism, RGB features can be "matched on demand" and obtain the most relevant information about the current target region from thermal infrared and depth modes.

[0143] 4. Multimodal fusion output

[0144] When fusing trimodal features, this invention does not simply add them together with weights, but instead uses concatenation to fuse them, in order to retain more complementary modal feature information.

[0145] The fusion process is as follows:

[0146]

[0147] in" "Indicates a splicing operation at the channel level, for Convolutional fusion layers are used to fuse information and suppress redundancy between channels.

[0148] After completing the cross-modal feature fusion described in Example 4, the present invention inputs the fused multi-scale features into the detection output module to achieve the final identification and location of defects in power towers and power lines.

[0149] This invention also represents a key improvement in the design of the loss function, employing a "master-slave joint supervision" strategy to ensure that all branch networks (defined in Example 3) are fully optimized. The total loss function of this invention... By a main loss and two auxiliary losses and Weighted composition.

[0150] Main loss Applied to the main head, which receives fully fused features from Example 4 (CMFM). It is responsible for supervising the model's final detection performance after all modal information is fused.

[0151] Modality-specific auxiliary loss ( and This is a key improvement proposed in this invention to solve the problem of multimodal training. Before feature fusion in the T and D branches, this invention introduces auxiliary detection heads and applies auxiliary losses. Its core innovation lies in "modality specificity": auxiliary loss. Only the defect category related to thermal anomalies is monitored; while auxiliary losses... Only supervise defect categories related to the three-dimensional structure. This design (in...) Figure 2 Zhongyou and This ensures that the T / D branches become "experts" in their respective fields before fusion, providing high-quality feature input to the main fusion module (Example 4).

[0152] Example 2

[0153] This embodiment provides a power tower and power line defect detection system based on a multimodal cross-attention mechanism, including:

[0154] The image acquisition and processing unit acquires RGB image, thermal infrared image and depth image three-modal data of the power tower and power line scene to be detected, and performs preprocessing.

[0155] The detection unit inputs preprocessed trimodal data into the constructed and trained power tower and power line defect detection model to obtain the power tower and power line defect detection results. The power tower and power line defect detection model is based on the YOLO detection framework and adopts a multimodal feature extraction network to replace the original single-modal backbone network of YOLO. It is used to extract features from RGB images, thermal infrared images, and depth images respectively. In the Neck part of YOLO, a multi-scale cross-modal fusion module is introduced to replace the original concat operation for fusing multimodal features. A modality-specific auxiliary supervision strategy is introduced to add an auxiliary loss function before the YOLO T / D branch fusion to improve the quality of feature extraction.

[0156] The core idea of ​​this invention lies in achieving high-precision detection of defects in power towers and power lines through cross-attention fusion of multimodal features. Various alternative implementation methods exist without departing from the principles of this invention.

[0157] In terms of modality selection, in addition to RGB, thermal infrared and depth images, other modality combinations can also be used, such as RGB with lidar point clouds, near-infrared images or multi-temporal visible light images, to adapt to different detection environments.

[0158] Regarding the backbone network structure, the asymmetric decoupled backbone network can be replaced by other networks with feature branching capabilities, such as the Swing Transformer, ConvNeXt, or MobileNet structures, as long as they can achieve independent extraction and encoding of multimodal features.

[0159] In the cross-modal fusion module, in addition to using the attention mechanism, gated fusion, weighted concatenation, or cross-modal feature interaction based on Transformer can also be used. As long as semantic association and feature compensation between different modalities can be achieved, similar technical effects can be achieved.

[0160] In the detection output section, the YOLO series detection head can be replaced with an Anchor-Free or Transformer structure detection head, such as the CenterNet or DETR model, to achieve the same defect identification and localization functions.

[0161] In summary, the core innovation of this invention lies in the "multimodal fusion detection framework based on cross-modal attention mechanism". Its specific implementation can be flexibly adjusted according to the actual application environment, and can still achieve the same or similar technical effects as this invention.

[0162] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism, characterized in that, include: Step S1: Acquire RGB image, thermal infrared image and depth image three-modal data of the power tower and power line scene to be detected, and perform preprocessing; Step S2: Input the preprocessed trimodal data into the constructed and trained power tower and wire defect detection model to obtain the power tower and wire defect detection results. The power tower and wire defect detection model is based on the YOLO detection framework and uses a multimodal feature extraction network to replace the original single-modal backbone network of YOLO. It extracts features from RGB images, thermal infrared images and depth images respectively. In the Neck part of YOLO, a multi-scale cross-modal fusion module is introduced to replace the original concat operation for fusing multimodal features. A modality-specific auxiliary supervision strategy is introduced, adding an auxiliary loss function before YOLO T / D branch fusion to improve the quality of feature extraction.

2. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 1, characterized in that, The preprocessing includes size alignment, geometric calibration, and pixel normalization operations.

3. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 1, characterized in that, The image processing procedure of the power tower and power line defect detection model is as follows: The preprocessed trimodal data is input into a multimodal feature extraction network to extract semantic features, temperature anomaly features, and structural deformation features, respectively. The extracted three-modal features are combined according to scale to construct multimodal feature groups with scales P3, P4, and P5 respectively; The multimodal features at three scales are input into the multi-scale cross-modal fusion module. The RGB features are used as the guiding modality and cross-modal attention interactions are performed with the thermal infrared and depth modalities respectively. The fused and enhanced multimodal features are then output. The enhanced multimodal characteristics are input into the detection head, and the detection results of power tower and power line defects are output after classification and regression prediction.

4. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 1, characterized in that, The multimodal feature extraction network adopts an asymmetric decoupled multimodal backbone network, SwinTransforme, ConvNeXt, or MobileNet structure.

5. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 4, characterized in that, The multimodal feature extraction network employs an asymmetric decoupled multimodal backbone network, which includes an RGB backbone branch, a thermal infrared T branch, and a depth D branch. The RGB backbone branch uses the YOLO backbone network structure to construct the RGB feature extractor. Multi-scale semantic features are obtained; the thermal infrared T branch uses a lightweight network as a thermal infrared feature extractor. It is used to extract information reflecting temperature differences and hotspot anomalies; the depth D branch employs a lightweight depth feature extraction network. Extract structural morphology-related features.

6. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 1, characterized in that, The multi-scale cross-modal fusion module employs attention mechanisms, gated fusion, weighted splicing, or a Transformer-based cross-modal feature interaction method.

7. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 6, characterized in that, The multi-scale cross-modal fusion module employs an attention mechanism, and the process of fusing input features is as follows: Feature flattening and dimensionality alignment: trimodal feature maps at the same scale First through Convolution performs channel mapping to ensure they have the same embedding dimension. Then, flatten each feature map; Perform linear transformations on the flattened feature matrices to generate query, key, and value vectors, as follows: ； in, For learnable parameters, These are the flattened RGB, thermal infrared, and depth feature matrices, respectively. This represents the query vector generated by the RGB modality; Feature index representing thermal infrared modes; Feature indexes representing deep modalities; Characteristic content representing thermal infrared modes; Feature content representing deep modalities; Cross-modal attention calculation: RGB is used to guide two cross-modal attention calculations with thermal infrared and depth modes respectively; When fusing three-modal features, the fusion is performed by convolution after splicing.

8. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 7, characterized in that, The specific steps of using RGB to guide two cross-modal attention calculations with thermal infrared and depth modes include: (1) RGB to Thermal cross-modal attention: The Attention weights are calculated as follows: ； in For activation function, The dimension of the thermal infrared bond feature is represented by T, which represents the transpose of the matrix. By weighting the thermal infrared Value vector, supplementary information extracted from the thermal infrared modes is obtained: ； (2) RGB to Depth cross-modal attention: The attention mechanism for RGB-guided deep features is as follows: ； The supplementary information extracted from the depth modes is as follows: 。 9. The method for detecting defects in power towers and power lines based on a multimodal cross-attention mechanism according to claim 1, characterized in that, Total loss function of power tower and power line defect detection model By a main loss and two auxiliary losses and Weighted composition, main loss Before feature fusion in the T and D branches, auxiliary detection heads are introduced and auxiliary losses are applied to the main detection head.

10. A power tower and power line defect detection system based on a multimodal cross-attention mechanism, implementing the method of any one of claims 1-9, characterized in that, include: The image acquisition and processing unit acquires RGB image, thermal infrared image and depth image three-modal data of the power tower and power line scene to be detected, and performs preprocessing. The detection unit inputs the preprocessed trimodal data into the constructed and trained power tower and wire defect detection model to obtain the power tower and wire defect detection results. The power tower and wire defect detection model is based on the YOLO detection framework and uses a multimodal feature extraction network to replace the original single-modal backbone network of YOLO. It is used to extract features from RGB images, thermal infrared images and depth images respectively. In the Neck part of YOLO, a multi-scale cross-modal fusion module is introduced to replace the original concat operation for fusing multimodal features. A modality-specific auxiliary supervision strategy is introduced, adding an auxiliary loss function before YOLO T / D branch fusion to improve the quality of feature extraction.

Images