Tower pole crack detection method based on multi-model fusion

Abstract

The invention relates to the technical field of image processing, in particular to a tower pole crack detection method based on multi-model fusion, and the method comprises the steps: obtaining an original image containing a power tower pole; processing the original image by using a target detection model so as to position a tower pole and cut the tower pole to obtain a tower pole sub-image; inputting the tower pole sub-images into a plurality of pre-trained classification models for reasoning; and obtaining an output result of each classification model, wherein the output result comprises a judgment label for judging whether a crack exists or not and a corresponding confidence score. According to the method, three different architecture models of YOLOv8-cls, PP-YOLOE-s and RtDetr are integrated, and the problem of one-sided feature extraction of a single model is solved by utilizing the complementary advantages of CNN and Transform in local texture and global semantics. By combining a dual decision strategy of majority voting preliminary screening and F1-Score weighted confidence reexamination, complex background interference is effectively overcome, and the robustness and accuracy of crack detection are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a method for detecting tower cracks based on multi-model fusion. Background Technology

[0002] As a critical support structure for power grid transmission lines, the integrity of power towers directly affects the safe operation of the power system. Currently, the detection of surface cracks on towers mainly relies on high-resolution images acquired by drones or ground inspection equipment. Due to the variable inspection environment, the original images typically contain complex background noise such as sky, vegetation, and buildings, and the tower surface often has interference factors such as corrosion, stains, and shadows.

[0003] In practical engineering applications, image acquisition environments are often more demanding, highly susceptible to drastic changes in light intensity, adverse weather conditions (such as rain, fog, and strong light), and interference from dynamic background clutter. In these unstructured scenarios, traditional single-model architectures often struggle to effectively integrate deep semantic information with shallow geometric details during feature extraction. Especially after multiple downsampling operations, the edge texture features of small or low-contrast targets are easily submerged or lost by background noise, significantly degrading the feature map's ability to represent foreground targets. Furthermore, due to the lack of an adaptive weighting mechanism for multi-scale features, the model struggles to achieve fine-grained feature decoupling when facing targets with large scale spans or local occlusion. This results in insufficient robustness of the system under complex conditions, frequent false positives and false negatives, limiting its practical application performance in all-weather, high-precision detection tasks. Summary of the Invention

[0004] This invention provides a tower crack detection method based on multi-model fusion, aiming to solve the problem in related technologies that, due to the lack of an adaptive weighting mechanism for multi-scale features, the model is unable to achieve fine decoupling of features when facing situations with large target scale spans or local occlusion, resulting in insufficient robustness of the system under complex working conditions and frequent false detections and missed detections.

[0005] This invention provides a tower crack detection method based on multi-model fusion, comprising: acquiring an original image containing a power tower; processing the original image using a target detection model to locate the tower and crop it to obtain a tower sub-image; inputting the tower sub-image into multiple pre-trained classification models for inference; acquiring the output results of each classification model, including a judgment label indicating whether a crack exists and the corresponding confidence score; performing a fusion decision based on the judgment labels and confidence scores of the multiple classification models, wherein the fusion decision includes: performing a final confirmation operation in response to the number of labels indicating the existence of a crack being greater than or equal to a preset number; calculating the weighted average of the confidence scores of all classification models, wherein a weight is assigned to each classification model to obtain a fusion confidence score; the weight is positively correlated with the F1-Score of the corresponding classification model on the validation set; and determining that the tower sub-image has a crack in response to the fusion confidence score being greater than a preset threshold, wherein the preset threshold ranges from 0.55 to 0.65. By combining the inference results of multiple classification models with quantitative judgment (voting) and confidence review based on performance weights for dual verification, the complementarity between models is utilized to correct the judgment bias of a single model. Compared with detection methods that rely solely on a single threshold, this invention significantly improves the robustness and accuracy of tower crack detection in unstructured environments (such as changes in illumination and background clutter), ensuring the reliability of the detection results.

[0006] Furthermore, locating and cropping the tower to obtain a sub-image of the tower includes: obtaining the bounding box coordinates of the tower using the target detection model; applying an edge fill coefficient to the bounding box coordinates for expansion processing to obtain an expanded cropped region; and cropping the tower sub-image from the original image based on the expanded cropped region. To address the problem that direct cropping by the detection box can easily lead to the truncation or loss of cracks located at the edges, an edge fill coefficient is introduced to expand the cropped region. Compared to traditional direct cropping methods, this invention can completely preserve the texture information of the tower edges, effectively preventing the loss of key features due to minor deviations in the target detection box, ensuring the data integrity of subsequent input classification models, and thus improving the recognition accuracy of edge cracks.

[0007] Furthermore, the formula for calculating the fusion confidence level is as follows: In the formula, This represents the weighted average confidence level after fusion; In this embodiment, the number of models is represented. ; Indicates the first The original confidence scores output by each model for the cracked category; Indicates the first The weighting factors for each model are defined. To address the issue that simple averaging methods fail to reflect model performance differences, a weighted average confidence calculation formula based on the validation set F1-Score is constructed. Compared to fusion methods that treat all model results equally, this invention assigns greater decision weight to models that perform better on specific tasks (higher F1-Scores), dynamically optimizing the confidence of the fusion results. This ensures that the final decision is primarily driven by the highly reliable model, while also considering the reference value of other models.

[0008] Furthermore, the multiple pre-trained classification models include: the YOLOv8-cls model based on a convolutional neural network (CNN) architecture; the PP-YOLOE-s model using a RepVGG structure; and the RtDetr classification model based on a Transformer architecture. By integrating these three heterogeneous models—YOLOv8-cls based on a CNN architecture, PP-YOLOE-s based on a RepVGG structure, and RtDetr based on a Transformer architecture—complementary advantages are achieved at the feature extraction level. Compared to single-architecture networks, this method can utilize both CNNs to capture local texture and edge details, and Transformers to capture global geometric structures and long-range dependencies. This allows it to effectively address both small cracks and long cracks with continuous features, overcoming the limitations of single models in feature representation.

[0009] Furthermore, the training process of the YOLOv8-cls model includes: using random cropping and color dithering as data augmentation methods; using the Mixup technique to mix training samples; using the C2f module to extract local texture features; and using the SGD optimizer for parameter updates. For the training of the YOLOv8-cls model, the application of Mixup augmentation and the C2f module enhances the model's sensitivity to local texture features. Compared to the conventional training process, this setup allows the model to better simulate and adapt to visual features under different lighting and corrosion levels, strengthens the model's ability to capture pixel-level edge transitions, and improves the efficiency of recognizing local fine crack textures.

[0010] Furthermore, the PP-YOLOE-s model employs a RepVGG structure and is trained specifically for low-contrast cracks. The training process includes: label smoothing of hard labels; and adjusting the learning rate using a cosine annealing scheduling strategy. For the PP-YOLOE-s model, label smoothing and cosine annealing effectively address the overfitting problem when identifying low-contrast cracks. Compared to training with hard labels, this method prevents the model from overconfidently performing on blurred or weakly textured images, enhances its generalization ability on low-quality images, and allows it to refine its search for local optima even in the later stages of training, improving its ability to detect hidden cracks in damp or shaded areas.

[0011] Furthermore, the RtDetr classification model is based on the Transformer architecture, and its training process includes: dividing the tower sub-image into multiple image patches and adding positional encoding; calculating the global correlation between image patches using a multi-head self-attention mechanism; and updating parameters using the AdamW optimizer. Utilizing the Transformer architecture and multi-head self-attention mechanism of the RtDetr model, the global correlation between image patches is calculated. Compared to traditional convolutional networks limited to local receptive fields, this method can understand the physical extension trajectory and global logical relationship of cracks on the tower surface. Even when cracks are partially obscured by bird droppings or rust, the existence of cracks can be inferred through contextual semantics, significantly enhancing the ability to resist occlusion interference.

[0012] Furthermore, it also includes: in response to determining that a crack exists in the tower sub-image, generating a detection report containing image identifiers, cropped area coordinates, crack presence status, and confidence scores for each classification model. By generating a structured detection report containing image identifiers, precise cropped coordinates, and detailed confidence scores for each model, the detection results are digitized and visualized. Compared to simply outputting alarm signals, this invention provides maintenance personnel with traceable and detailed data support, facilitating rapid fault location and subsequent analysis in the power grid inspection management system, significantly improving the operational efficiency of power grid inspections.

[0013] Furthermore, the target detection model is the RT-DETR model. The RT-DETR model is selected as the pre-detector, utilizing its efficient hybrid encoder and uncertainty minimization mechanism. Compared to traditional detection models, it can more accurately locate and separate the tower body from complex backgrounds, providing high-quality, low-noise input images for subsequent crack classification, thus ensuring the accuracy of the entire detection process from the source.

[0014] Beneficial Effects: By integrating three different architecture models—YOLOv8-cls, PP-YOLOE-s, and RtDetr—and leveraging the complementary advantages of CNN and Transformer in local texture and global semantics, the problem of one-sided feature extraction by a single model is solved. The dual decision-making strategy of majority voting initial screening and F1-score-weighted confidence review effectively overcomes interference from complex backgrounds, significantly improving the robustness and accuracy of crack detection. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the tower crack detection process according to an embodiment of the present invention. Detailed Implementation

[0016] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] like Figure 1 As shown, S101: Obtain the tower sub-image.

[0018] In this embodiment, the original image to be detected is first acquired. This original image is typically captured by a drone or ground inspection equipment against a complex background, containing various elements such as the sky, trees, buildings, and power towers. To fundamentally eliminate the interference of background texture on crack recognition, this step uses a pre-trained RT-DETR (Real-Time Detachment Transformer) model to process the input inspection image.

[0019] Specifically, RT-DETR is a real-time object detection model based on the Transformer architecture. Compared to traditional CNN architectures (such as the YOLO series), it introduces an efficient hybrid encoder and an uncertainty-minimizing query selection mechanism. During model training, a large-scale power grid inspection image dataset is used for supervised learning, and the AdamW optimizer is used for parameter iteration. The optimal learning rate is set to [value missing]. The training epochs were set to 300 to ensure that the model could accurately extract the global structural features of the tower.

[0020] During model inference, the original image is input, and the bounding box coordinates of the tower target are output. Based on the bounding box coordinates Image processing algorithms are used to crop a sub-image containing only the main body of the tower from the original image. This step transforms the complex problem of searching for cracks in the whole image into a more focused problem of classifying specific regions, effectively eliminating background noise interference from tree branches, building edges, and other textures similar to cracks.

[0021] When the RT-DETR model outputs bounding box coordinates, for example, if the resolution of the input original image is... Pixels, the model may be in coordinates A tower target was identified at this location. To ensure the integrity of the crack information, the trimming algorithm does not directly cut according to this coordinate, but instead adds a dynamic edge fill coefficient. For example, take Then the cropping area will expand outward by 5%, and the final cropping range will become... This effectively prevents cracks located at the edge of the tower from being cut off due to boundary box fine-tuning, ensuring the integrity of the input data for subsequent classification models.

[0022] S102: Parallel crack identification based on a multi-heterogeneous classification model.

[0023] In one embodiment, to overcome the limitations of a single model in feature extraction, each obtained tower sub-image is simultaneously input into three pre-trained, heterogeneous deep learning models for parallel inference. The three heterogeneous models are the YOLOv8-cls model, the PP-YOLOE-s model, and the RtDetr classification model.

[0024] The model training process is as follows: Using tens of thousands of high-definition images collected by drones, tens of thousands of tower sub-image samples are extracted using the detection model described in step S101 of this solution. Power experts manually annotate the cracks in the sub-images, classifying them as having cracks. With no cracks There are two categories, which are divided into training set, validation set and test set in a ratio of 8:1:1.

[0025] To ensure consistency in parallel inference, all input images must undergo normalization before entering the network. Image pixel values ​​are from... Linear mapping to The interval is calculated by subtracting the mean and dividing by the standard deviation. Input tensors are uniformly adjusted to a four-dimensional structure. ,in For batch size, (RGB channels) .

[0026] The YOLOv8-cls model focuses on learning the local texture features of cracks and pixel-level edge transitions. Random cropping and color dithering are used as core enhancement techniques to simulate the visual characteristics of the tower surface under different lighting and corrosion levels. Mixup data augmentation technology is used to proportionally blend tower images against different backgrounds to enhance the model's adaptability to complex interference environments. The model input tensor is... During training, the backbone network constructs rich gradient flows through the C2f module. The model utilizes small-sized convolutional kernels ( The sub-images are scanned layer by layer to capture subtle derivative changes at the crack edges. Cross-entropy loss is used as the optimization objective. The SGD (Stochastic Gradient Descent) optimizer is used, with a momentum factor of 0.937 and an initial learning rate of 0.01. A learning rate warmup strategy is employed to ensure rapid and stable convergence of the model in the early stages of training. At the end of the model, a global average pooling layer is used to compress the spatial dimension and output a probability vector of length 2. .in, This represents the probability that the image does not contain cracks. This represents the probability that the image contains cracks. In this embodiment, the probability value corresponding to the crack category in the vector is taken. As the original confidence score of the model At the same time, hard labels are determined based on the probability values. :like (or If the label is determined, then the label is determined. ,otherwise This results in a well-trained YOLOv8-cls model.

[0027] It's worth noting that the YOLOv8-cls model, as a typical example of CNN-based architecture, is based on a deep convolutional neural network (CNN) architecture, with its core unit being the C2f module. Through rich gradient flow branches, this model is extremely sensitive to high-frequency components of images. With an input sub-image size of 224×244, its multi-layer convolutional kernels can extract subtle local textures such as crack edge slopes and color abrupt changes layer by layer. For physical fracture lines on the tower surface caused by stress, YOLOv8-cls can quickly respond to pixel jumps in their spatial location.

[0028] The PP-YOLOE-s model is specifically trained for low-contrast, fine cracks, balancing detection accuracy with industrial-grade inference speed. During training, PP-YOLOE-s employs a RepVGG-style multi-branch structure, including... convolution, A composite module of convolutions and skip connections (Identity) is used to capture extremely weak signal features. This structure enables the model to possess strong multi-scale feature modeling capabilities during training. Label smoothing techniques are introduced. For example, hard labels are smoothed from... Adjusted to This effectively prevents the model from making overly arbitrary judgments about low-quality images during training, enhancing its robustness in identifying weakly textured cracks. A cosine annealing scheduling strategy is employed. During training, the learning rate decreases with increasing epochs using a cosine curve, ensuring that the model can finely search for local optima in the later stages of training, accurately capturing minute cracks in the shaded areas of the tower or on damp surfaces. The input is normalized... The tensor, after being fused by structural reparameterization, outputs a confidence score representing the probability of crack existence. This results in a well-trained PP-YOLOE-s model.

[0029] It should be noted that the PP-YOLOE-s model incorporates the CSPNet structure and Dynamic Label Assignment (TAL) strategy. In this embodiment, the model is optimized for identifying low-contrast cracks. When power towers are damp or shaded, the color difference between the cracks and the background is minimal. PP-YOLOE-s utilizes its unique RepVGG structure reparameterization technique to fuse branch structures during the inference phase, ensuring both extremely fast response speed and enhanced ability to extract subtle signals, effectively compensating for the missed detection problem of standard CNNs under weak features.

[0030] The RT-DETR classification model utilizes a Transformer architecture, focusing on identifying continuous cracks with global geometric logic. Unlike the previous two, RT-DETR divides the image into a series of... The model learns the relative spatial relationships between these image patches through positional encoding, enabling it to understand the physical trajectory of the crack extending across the tower surface. During training, the model utilizes multi-head self-attention to calculate the correlation between any two pixel patches in the image. Even if the crack is partially obscured by foreign objects, the model can infer its existence through global logic. Because the Transformer structure is sensitive to parameters, the AdamW optimizer is used during training, with a learning rate set to [value missing]. A significant weight decay (WeightDecay=0.05) is applied to prevent overfitting. The training epochs are set to 300 to ensure the model fully learns long-range pixel dependencies. The input tensor is serialized and then fed into the Transformer encoder. The model outputs confidence scores through a fully connected classification head. This leads to a well-trained RT-DETR classification model.

[0031] It's worth noting that the RtDetr classification model utilizes the Transformer's self-attention mechanism to calculate the global correlation between pixel blocks by serializing the image. Cracks often have continuous geometric shapes, and the RtDetr classification model can identify transverse, diagonal, or irregular mesh-like cracks with long-distance logical relationships. Even if a crack is partially obscured by foreign objects (such as bird droppings or rust), the model can infer the crack's extension trajectory based on global contextual semantics, exhibiting stronger robustness against occlusion.

[0032] For each input tower image, the system launches a multi-threaded parallel inference engine. Three models operate independently, each outputting a category probability distribution. The predicted probability (i.e., confidence score) for the cracked category is recorded. and the corresponding hard-label results In this context, 1 represents a crack and 0 represents no crack.

[0033] S103: Multi-model decision fusion and cross-validation.

[0034] In one embodiment, in order to reduce the false detection rate and false negative rate, this embodiment does not rely on the output of a single model, but adopts a dual decision-making mechanism that combines majority voting and confidence weighting.

[0035] For the initial screening based on majority voting, the number of models that were identified as having cracks is counted from the outputs of the three models. .like If at least two models identify a crack, the tower is preliminarily determined to have a crack, and the process proceeds to final confirmation. This multi-model decision fusion mechanism leverages the complementarity between models to improve the accuracy of the detection results.

[0036] For review based on weighted average confidence level. If Alternatively, to further verify the reliability of the results, a weighted average confidence level can be calculated. Considering the differences in performance of different models on the validation set, a weighted calculation formula needs to be constructed.

[0037] The formula is based on the fact that models with different architectures have varying precision and recall rates when handling specific tasks. To maximize the overall performance of the system, models that perform better on the validation set should be given higher weights.

[0038] The calculation formula is as follows: ;in: Indicates the fusion confidence level; In this embodiment, the number of models is represented. ; Indicates the first The original confidence scores output by each model for the cracked category; Indicates the first The weight factors of each model.

[0039] It should be noted that the weighting factor The value is determined by the model's F1-Score on the offline validation set. For example, if the F1-Scores of YOLOv8-cls, PP-YOLOE-s, and RtDetr on the validation set are 0.85, 0.90, and 0.88, respectively, then the normalized weights of the YOLOv8-cls model, PP-YOLOE-s model, and RtDetr classification model can be set to 0.32, 0.34, and 0.34, respectively.

[0040] As the formula shows, the final weighted average confidence score is a linear combination of the confidence scores of each model. Models with high weights (such as PP-YOLOE-s) output higher confidence scores. At that time, it will significantly increase The value reflects the system's greater reliance on models with stable performance.

[0041] Finally, based on the fusion confidence score, it is compared with a preset fusion threshold. Comparison: If If so, it is ultimately determined to be a crack; if If the result is positive, it is ultimately determined to be crack-free. In this embodiment, the fusion threshold... Preferred .when At times, the system is overly sensitive to crack detection, easily misidentifying stains or watermarks on the tower surface as cracks, leading to an increased false detection rate; when At times, the system is overly conservative, only issuing an alarm when multiple models give extremely high confidence levels, which can easily miss minute cracks, leading to an increased false negative rate. Therefore, [the system should be modified / improved]. Settings Between, especially This approach can effectively control the false alarm rate while ensuring recall, achieving the optimal balance of detection performance.

[0042] To more clearly illustrate the fusion mechanism, suppose an image of a tower column suspected of having minor rust cracks is acquired during an actual inspection. The performance of the three models is as follows: Model 1 (YOLOv8-cls): Limited by local lighting interference, it determines that there are no cracks and outputs a confidence score. ,Label Model 2 (PP-YOLOE-s): Captured subtle textures, identified cracks, and output confidence scores. ,Label Model 3 (RtDetr-cls): Based on the global geometry, it determines that there is a crack and outputs the confidence score. ,Label Step 1: Initial screening by majority vote. Statistical results are obtained. .because If the criterion of "at least two models are determined to be present" is met, the system will initially trigger an early warning and enter the weight calculation stage.

[0043] Then, a weighted average is calculated, with the normalized weights known to be... Substitute the values ​​into the formula to calculate the fusion confidence level: .

[0044] Finally, a threshold determination is performed, and the calculated threshold is used to determine the threshold. With preset threshold Compare. Because The system eliminated the interference from missed detections in Model 1 and ultimately determined that the tower was "cracked". This case demonstrates that multi-model fusion can effectively utilize better-performing models (such as Model 2 and Model 3) to correct the judgment errors of poor-performing models.

[0045] S104: Generation and output of detection results.

[0046] In this embodiment, the final crack determination results for all towers are summarized. The system generates a structured inspection report containing a unique image identifier (ID), the cropping coordinates of the tower in the original image, the crack presence status (yes / no), and the original confidence details for each sub-model. This report can be directly imported into the power grid inspection management system, allowing maintenance personnel to quickly locate towers requiring special attention.

[0047] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for detecting tower cracks based on multi-model fusion, characterized in that, include: Obtain the original image containing the power tower poles; The original image is processed using an object detection model to locate the tower pole and crop it to obtain a tower pole sub-image; The tower sub-images are input into multiple pre-trained classification models for inference; Obtain the output results of each classification model, including the judgment label for whether a crack exists and the corresponding confidence score; A fusion decision is made based on the judgment labels and confidence scores of the multiple classification models. The fusion decision includes: in response to the number of labels that are judged to have cracks being greater than or equal to a preset number, a final confirmation operation is performed. The final confirmation operation includes: calculating the weighted average of the confidence scores of all classification models to obtain the fusion confidence score, wherein each classification model corresponds to a weight, and the weight is positively correlated with the F1-Score of the corresponding classification model on the validation set; If the fusion confidence level is greater than a preset threshold, it is determined that there is a crack in the tower sub-image.

2. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, Locate the tower and crop it to obtain a sub-image of the tower, including: The target detection model is used to obtain the bounding box coordinates of the tower. The bounding box coordinates are expanded by applying an edge fill factor to obtain the expanded clipping region; The tower sub-image is obtained by cropping the original image based on the expanded cropping region.

3. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, The formula for calculating the fusion confidence score is: ； In the formula, This represents the weighted average confidence level after fusion; In this embodiment, the number of models is represented. ; Indicates the first The original confidence scores output by each model for the cracked category; Indicates the first The weight factors of each model.

4. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, The multiple pre-trained classification models include: YOLOv8-cls model based on convolutional neural network (CNN) architecture; The PP-YOLOE-s model adopts the RepVGG structure; The RtDetr classification model based on the Transformer architecture.

5. The tower crack detection method based on multi-model fusion according to claim 4, characterized in that, The training process of the YOLOv8-cls model includes: Random cropping and color dithering were used as data augmentation methods; Mixup technology was used to mix training samples; local texture features were extracted using the C2f module; and the SGD optimizer was used for parameter updates.

6. The tower crack detection method based on multi-model fusion according to claim 4, characterized in that, The PP-YOLOE-s model adopts the RepVGG structure and is trained for low-contrast cracks. The training process also includes: label smoothing of hard labels; and adjusting the learning rate using a cosine annealing scheduling strategy.

7. The tower crack detection method based on multi-model fusion according to claim 4, characterized in that, The RtDetr classification model is based on the Transformer architecture, and its training process includes: The tower sub-image is divided into multiple image blocks and positional encoding is added; the global correlation between image blocks is calculated using a multi-head self-attention mechanism; and the parameters are updated using the AdamW optimizer.

8. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, Also includes: In response to the determination that a crack exists in the tower sub-image, a detection report is generated that includes image identifiers, cropped region coordinates, crack presence status, and confidence scores for each classification model.

9. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, The preset threshold value ranges from 0.55 to 0.

65.

10. The tower crack detection method based on multi-model fusion according to claim 1, characterized in that, The target detection model is the RT-DETR model.

Images