Method and system for detecting defects of insulators of power transmission lines based on machine vision

Abstract

The application discloses a power transmission line insulator defect detection method and system based on machine vision, relates to the technical field of machine vision, and comprises the following steps: combining a pre-trained DenseCLIP model with a YOLOv11 network, extracting insulator image features through a visual encoder of the DenseCLIP model and converting the insulator image features into visual embedding vectors, constructing a text prompt according to a defect category and converting the text prompt into a language embedding vector, calculating the similarity of the visual and language embedding vectors, generating a pixel-text matching score map to guide the learning of the YOLOv11 network, and constructing an insulator defect detection model; training a mixed training set constructed by combining artificial samples and real samples by using a transfer learning stage-by-stage training strategy, inputting an insulator image to be detected into the insulator defect detection model, obtaining an enhanced feature map fused by visual and language embedding vectors and a pixel-text score map, and generating a final prediction result. The application improves the detection accuracy and real-time performance of insulator defects and significantly improves the intelligent operation and maintenance efficiency of a power grid.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and more specifically, to a method and system for detecting defects in power transmission line insulators based on machine vision. Background Technology

[0002] In recent years, with the rapid development of power systems, the operation and maintenance of transmission lines have faced numerous challenges. Insulators, as key components of transmission lines, are crucial for ensuring the safe and stable operation of the power grid through defect detection. However, traditional manual inspection methods suffer from low efficiency, high cost, and poor safety. While drone inspection technology has certain advantages, it still faces many technical bottlenecks in complex environments. Several deep learning-based detection algorithms have been proposed for insulator defect detection in transmission lines, but they still have limitations in practical applications.

[0003] In existing technologies, insulator defect areas exhibit significant scale differences and morphological concealment compared to the main target. Most models employ high downsampling rates, making it difficult to learn the feature information of small target defects, thus affecting detection accuracy. This is especially true for low-contrast defects such as minor damage and blurred corrosion, which are prone to missed or false detections. Existing detection algorithms generally suffer from large model parameters and high computational complexity, limiting their operation to demanding PC platforms and hindering deployment on embedded platforms. This fails to meet the real-time performance and hardware resource constraints required for actual transmission line inspections. Furthermore, the scarcity and difficulty in labeling real defect samples, coupled with a lack of high-quality labeled samples, restricts model training effectiveness and generalization ability. Uneven data distribution further exacerbates the training difficulties. The complex background interference and variable lighting conditions of overhead line corridors significantly increase the difficulty of target localization. Existing algorithms lack robustness in complex environments, easily impacting detection performance. Therefore, how to solve the problems of insufficient accuracy of existing transmission line insulator defect detection technology in complex environments, difficulty in deployment on embedded platforms, and dependence on large amounts of data are urgent technical challenges. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a machine vision-based method and system for detecting insulator defects in transmission lines. This system achieves high-precision, real-time insulator defect detection under limited sample conditions, significantly improving the efficiency of intelligent operation and maintenance of the power grid.

[0005] The first aspect of this invention provides a machine vision-based method for detecting defects in transmission line insulators, comprising the following steps:

[0006] Digital twins of insulators are constructed, artificial samples with different defect patterns are generated, and real images from multiple scenarios are collected. Based on three types of labeling systems, including normal insulators, broken defects, and flashover defects, real samples with defect labels are constructed.

[0007] The pre-trained DenseCLIP model is combined with the YOLOv11 network. The visual encoder of the DenseCLIP model extracts the features of the insulator image and converts them into visual embedding vectors. Text prompts are constructed according to the defect category and converted into language embedding vectors. The similarity between the visual and language embedding vectors is calculated, and a pixel-text matching score map is generated to guide the YOLOv11 network to learn and build an insulator defect detection model.

[0008] The artificial samples and the real samples are constructed into a mixed training set according to a preset ratio. Based on the mixed training set, the insulator defect detection model is trained using a phased training strategy of transfer learning. The trained insulator defect detection model is then deployed.

[0009] The image of the insulator to be detected is imported into the deployed insulator defect detection model to obtain the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and the final prediction result is generated.

[0010] In this scheme, digital twins of insulators are constructed, artificial samples of different defect modes are generated, and real images from multiple scenarios are collected simultaneously. Based on a three-category annotation system including normal insulators, damaged defects, and flashover defects, real samples with defect labels are constructed, specifically as follows:

[0011] The geometric specifications and material information of the insulator are obtained as the basis for modeling. A parametric model is established in 3ds Max software to generate three-dimensional models of insulators with different voltage levels. Based on the three-dimensional models of the insulators, the optical properties of the insulating material in the insulators are configured, and the surface states of different aging degrees are simulated.

[0012] A digital twin of the insulator is generated using the simulated 3D model of the insulator, and artificial sample datasets of different defect modes are constructed through Boolean operations.

[0013] Real images of insulators under multiple scenarios are acquired using a multi-source acquisition method. A three-category annotation system is constructed, including normal insulators, damaged insulators, and flashover defects. The real images of insulators are then labeled with defect categories using the annotation system to construct a real sample dataset.

[0014] In this scheme, the visual encoder of the CLIP model extracts the features of the insulator image and converts them into visual embedding vectors. Based on the defect category, text prompts are constructed and converted into language embedding vectors. The similarity between the visual and language embedding vectors is calculated, and a pixel-text matching score map is generated to guide the YOLOv11 network learning to build an insulator defect detection model.

[0015] An insulator defect detection model is constructed based on the pre-trained DenseCLIP model and YOLOv11 network. The DarkNet backbone network of the YOLOv11 network is used to extract multi-scale insulator image features, and the original feature pyramid structure is retained for target detection to obtain visual feature flow.

[0016] Load the pre-trained DenseCLIP model, use the CLIP-based visual encoder to extract pixel-level visual embeddings, and use the text encoder to generate semantic embeddings of insulator defect categories to obtain the language feature stream;

[0017] The similarity between the visual embedding vectors in the visual feature stream and the language embedding vectors in the language feature stream is calculated to obtain a similarity matrix. A pixel-text matching score map is generated to represent the semantic association strength between the identified image region and the defect category, guiding the YOLOv11 network learning.

[0018] A cross-attention module is inserted before the detection head of the YOLOv11 network to dynamically weight and fuse the visual embedding vector of the YOLOv11 object detection network with the language embedding vector of DenseCLIP, thereby enhancing the sensitivity to low-contrast defects.

[0019] A small target detection layer is added to the top of the YOLOv11 feature pyramid. It directly receives the high-density feature input from DenseCLIP and downsamples the pixel-text score map of DenseCLIP to different scales. It is then concatenated with the multi-level feature map of YOLOv11 channel by channel. The defect detection result and defect localization are obtained through the detection head.

[0020] In this solution, a context-aware prompting strategy is adopted to optimize the text embedding of the language model, specifically as follows:

[0021] The system automatically generates a suitable text prompt template based on the content of the input insulator image. The image context information is injected into the text encoding process through the Transformer module of DenseCLIP, visual context features are extracted, and the visual context features are combined with a learnable query vector to generate dynamic text prompts.

[0022] After the dynamic text prompts are generated, the CoOp method is used to perform learnable context optimization on the defect category description using post-language model prompts, generating semantic embeddings. Then, the Transformer decoder uses the text features as query vectors to interact with the insulator image features, and updates the language feature stream through residual connections.

[0023] In this scheme, the artificial samples and the real samples are constructed into a hybrid training set according to a preset ratio. Based on the hybrid training set, the insulator defect detection model is trained using a phased training strategy of transfer learning. Specifically:

[0024] A hybrid training set is constructed by combining artificial and real samples according to a preset ratio, and a data augmentation strategy is executed in the hybrid training set. The hybrid training set is then used for phased transfer learning training.

[0025] In the first-order training, artificial samples were used for pre-training, the first three layers of the YOLOv11 backbone network were frozen, the parameters of the high-level network were optimized, and the classification loss and regression loss were calculated using artificial samples, so that the model could initially grasp the basic semantic representation of the general structure and defect detection of insulators.

[0026] In the second-order training, mixed samples from the mixed training set are used for joint training, all network parameters are unfrozen, a domain adaptation layer is introduced, a gradient reversal layer is added after the output of the feature pyramid layer of the YOLOv11 network, the feature distribution difference between artificial samples and real samples is minimized through adversarial training, and the feature distance between artificial samples and real samples is quantified using the maximum mean difference loss, forcing the model to learn domain-invariant features.

[0027] A validation set is constructed using real samples, and a cross-validation strategy is used to validate the model. When the model performance meets the preset requirements, the model parameters with the minimum validation loss are output to obtain the trained insulator defect detection model.

[0028] In this scheme, an improved focus loss function and GIoU regression loss are designed in the training of the insulator defect detection model to form a balanced training of classification-regression dual branches. The improved focus loss function dynamically adjusts the classification loss weight through the pixel-text matching score map, so that the model pays more attention to difficult-to-classify samples and defects that are easy to miss during the training process.

[0029] The GIoU regression loss uses the pixel-text score map provided by DenseCLIP to correct the bounding box offset, thereby improving the model's ability to detect easily missed defects and its localization accuracy.

[0030] In this scheme, the preprocessed image of the insulator to be detected is imported into the deployed insulator defect detection model to obtain the enhanced feature map after fusion of visual and language embedding vectors and the pixel-text score map, generating the final prediction result, specifically:

[0031] The images of the insulators to be detected are obtained and standardized as input to the insulator defect detection model. Multi-scale feature maps are generated through the DarkNet backbone network of the YOLOv11 network via multi-layer convolution and feature pyramid.

[0032] Load predefined defect category text hints, generate language embedding vectors for defect categories using DenseCLIP's text encoder, dynamically optimize text features using post-language model hints, extract pixel-level visual embedding vectors from the input image using DenseCLIP's visual encoder, calculate cosine similarity with language feature vectors, and generate pixel-text score maps.

[0033] The pixel-text score map is downsampled to the same size as the feature maps of each layer of the YOLOv11 network. The visual embedding vector is semantically weighted by multiplying channel by channel. A cross-attention module is inserted in front of the detection head of the YOLOv11 network to generate an enhanced feature map that integrates semantic information.

[0034] The enhanced feature map is input into the detection head of the YOLOv11 network to predict bounding boxes, class confidence, and defect localization accuracy at multiple scales. The average pixel-text score within the detection box is used as the semantic confidence, which is multiplied by the class confidence of YOLOv11 to obtain the final defect score.

[0035] Based on the final defect score, a prediction box that meets the preset requirements is obtained as the final defect detection result.

[0036] The second aspect of the present invention provides a machine vision-based transmission line insulator defect detection system, which includes a data acquisition and preprocessing module, a sample generation and annotation module, an insulator defect detection module, a hybrid sample training and transfer learning module, and an inference and result generation module.

[0037] The data acquisition and preprocessing module acquires insulator images from multiple sources and preprocesses them.

[0038] The sample generation and annotation module is responsible for constructing digital twins of insulators, generating artificial samples with different defect patterns, and constructing real samples with defect labels based on three types of annotation systems including normal insulators, broken defects, and flashover defects from real images collected from multiple scenarios.

[0039] The insulator defect detection module combines the pre-trained DenseCLIP model with the YOLOv11 network. It extracts insulator image features through the visual encoder of the DenseCLIP model and converts them into visual embedding vectors. It constructs text prompts based on defect categories and converts them into language embedding vectors. It calculates the similarity between visual and language embedding vectors and generates a pixel-text matching score map to guide the YOLOv11 network learning and build an insulator defect detection model.

[0040] The hybrid sample training and transfer learning module constructs a hybrid training set by combining artificial samples and real samples in a preset ratio. Based on the hybrid training set, a phased training strategy of transfer learning is used to train the insulator defect detection model, and the trained insulator defect detection model is output for deployment.

[0041] The reasoning and result generation module imports the preprocessed image of the insulator to be detected into the deployed insulator defect detection model, obtains the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and generates the final prediction result.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention improves the accuracy and real-time performance of insulator defect detection in transmission lines while reducing the computational complexity and resource requirements of the model. Compared to existing technologies, it borrows the DenseCLIP framework to fuse pre-trained visual-linguistic knowledge (such as the CLIP model) with the YOLOv11 object detection network. By transforming the image-text matching problem into a pixel-text matching problem and using pixel-text score maps to guide model learning, it significantly improves the model's semantic understanding of insulator defects. A context-aware cueing strategy is employed to optimize the text embedding of the language model, enabling the model to better utilize pre-trained visual-linguistic knowledge.

[0044] This invention constructs a complete technical chain of artificial sample generation, hybrid transfer learning, and loss function optimization. It utilizes artificial samples to expand the dataset, alleviating the problem of scarce real samples and improving the model's ability to identify defects in small samples. Simultaneously, by freezing some layers of the YOLOv11 backbone network, it retains the ability to extract general structural features of insulators, and introduces a domain adaptation layer to minimize the difference in feature distribution between artificial and real samples. This effectively alleviates the "domain shift" problem in cross-domain transfer and enhances the model's sensitivity to low-contrast defects in real-world scenes.

[0045] This invention, based on the YOLOv11 model, enhances the detection capability for small target defects such as insulators by adding a small target detection layer, accurately detecting small target defects in insulators, especially low-contrast defects. The model undergoes lightweight design and quantization processing, enabling it to run efficiently on low-computing-power, low-memory edge computing platforms, meeting the real-time requirements of inspections. Furthermore, the loss function is optimized to address sample distribution imbalance and positioning bias, comprehensively enhancing the model's detection performance and generalization ability, providing an efficient and reliable insulator defect detection solution for intelligent operation and maintenance of transmission lines. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0047] Figure 1 A flowchart of a machine vision-based method for detecting defects in transmission line insulators is shown.

[0048] Figure 2 The flowchart for constructing an insulator defect detection model is shown;

[0049] Figure 3 A flowchart for obtaining the predicted results of defects in the insulator to be detected is shown;

[0050] Figure 4 A framework diagram of the training process for the insulator defect detection model is shown.

[0051] Figure 5 A block diagram of a machine vision-based insulator defect detection system for power transmission lines is shown. Detailed Implementation

[0052] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0054] like Figure 1 As shown, this embodiment provides a machine vision-based method for detecting defects in transmission line insulators, including:

[0055] S102, construct digital twins of insulators, generate artificial samples of different defect modes, and collect real images of multiple scenarios. Based on three types of labeling systems including normal insulators, broken defects, and flashover defects, construct real samples with defect labels.

[0056] S104 combines the pre-trained DenseCLIP model with the YOLOv11 network. The visual encoder of the DenseCLIP model extracts the features of the insulator image and converts them into visual embedding vectors. Text prompts are constructed according to the defect category and converted into language embedding vectors. The similarity between the visual and language embedding vectors is calculated, and a pixel-text matching score map is generated to guide the YOLOv11 network to learn and build an insulator defect detection model.

[0057] S106, construct a mixed training set by combining the artificial samples and the real samples according to a preset ratio, train the insulator defect detection model based on the mixed training set using a phased training strategy of transfer learning, and output the trained insulator defect detection model for deployment.

[0058] S108: Import the image of the insulator to be detected into the deployed insulator defect detection model, obtain the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and generate the final prediction result.

[0059] It should be noted that the geometric specifications and material standards for disc insulators in the national standard GB / T 7253-2005 were strictly followed. The geometric specifications and material information of the insulators were obtained as the basis for modeling. A parametric model was established in 3ds Max software. By adjusting core parameters such as the diameter of the umbrella disc, the spacing between umbrellas, and the inclination angle of the umbrella skirts, three-dimensional models of insulators for different voltage levels were generated. Based on the three-dimensional models of the insulators, the optical properties of typical insulating materials such as ceramics, glass, and composite silicone rubber in the insulators were configured, and the surface states of different aging degrees were simulated. The simulated three-dimensional models of the insulators were used to generate digital twins of the insulators. Artificial sample datasets of different defect modes were constructed through Boolean operations, including: structural defects such as cracks on the edge of the umbrella disc and corrosion of the steel cap; electrical defects such as flashover channels and local carbonization; and dirt defects such as uneven accumulation of dirt and bird droppings. These served as a prior feature library for transfer learning. Using multi-source acquisition methods such as drone inspections and manual tower climbing for photography, real images of insulators under various scenarios were obtained. A three-category labeling system was constructed, including normal insulators, damaged insulators, and flashover defects. The real images of insulators were labeled with defect categories using the labeling system to construct a real sample dataset.

[0060] Figure 2 A flowchart for constructing an insulator defect detection model is shown.

[0061] According to an embodiment of the present invention, insulator image features are extracted by the visual encoder of the CLIP model and converted into visual embedding vectors. Text prompts are constructed according to the defect category and converted into language embedding vectors. The similarity between the visual and language embedding vectors is calculated, and a pixel-text matching score map is generated to guide the YOLOv11 network learning, thereby constructing an insulator defect detection model.

[0062] S202, based on the pre-trained DenseCLIP model and YOLOv11 network, an insulator defect detection model is constructed. The DarkNet backbone network of the YOLOv11 network is used to extract multi-scale insulator image features, and the original feature pyramid structure is retained for target detection to obtain visual feature flow.

[0063] S204, Load the pre-trained DenseCLIP model, use the CLIP-based visual encoder to extract pixel-level visual embeddings, and use the text encoder to generate semantic embeddings of insulator defect categories to obtain the language feature stream;

[0064] S206. The similarity between the visual embedding vector in the visual feature stream and the language embedding vector in the language feature stream is calculated to obtain a similarity matrix. A pixel-text matching score map is generated to represent the semantic association strength between the identified image region and the defect category, and to guide the YOLOv11 network learning.

[0065] S208, a cross-attention module is inserted in front of the detection head of the YOLOv11 network to dynamically weight and fuse the visual embedding vector of the YOLOv11 object detection network with the language embedding vector of DenseCLIP, thereby enhancing the sensitivity to low-contrast defects.

[0066] S210 adds a small target detection layer to the top of the YOLOv11 feature pyramid, directly receives the high-density feature input of DenseCLIP, downsamples the pixel-text score map of DenseCLIP to different scales, and concatenates it with the multi-level feature map of YOLOv11 channel by channel. The defect detection result and defect localization are obtained through the detection head.

[0067] It's important to note that DenseCLIP's visual encoder receives the input image and converts it into visual embedding vectors containing visual features, providing image-level feature representations for subsequent pixel-to-text matching. Simultaneously, the text encoder receives specific textual cues (such as descriptions of insulator defect categories) and converts them into text embedding vectors, providing the model with semantic information. The Transformer module combines the visual and text embedding vectors, employing complex self-attention mechanisms and other operations to deeply explore the potential connections between the image and text, generating feature representations infused with rich semantic understanding. Based on this, the similarity between the image and text embedding vectors is calculated to generate a high-precision pixel-to-text score map, thus transforming the traditional image-to-text matching problem into a more refined pixel-to-text matching problem. Calculating the similarity between the visual and text embedding vectors yields the pixel-to-text matching score map, providing nuanced semantic guidance for dense prediction tasks, enabling the model to accurately locate pixel regions in the image that match the text description. The pixel-to-text score map guides the YOLOv11 network's learning, enabling the network to more accurately locate and identify insulator defect locations, significantly improving the model's semantic understanding of insulator defects. Based on the fused feature representation and pixel-text score map, the image decoder generates the final prediction result. Whether for tasks such as semantic segmentation or object detection boxes, the image decoder can accurately classify and locate each pixel or region of the image based on the fused information and score map, thereby outputting a detailed interpretation of the image content.

[0068] A small target detection layer is added to the YOLOv11 network to enhance the detection capability for small target defects such as insulators. By using high-resolution feature maps for detection and introducing multi-scale feature fusion technology, the model can more accurately identify easily missed defects such as minor damage and blurred corrosion.

[0069] It should be noted that an adapted text prompt template is automatically generated based on the input insulator image content. The DenseCLIP Transformer module injects image context information into the text encoding process, extracts visual context features, and combines these features with a learnable query vector to generate dynamic text prompts, making the text prompts more closely aligned with the image content. After the dynamic text prompts are generated, a post-language model prompt uses the CoOp method to perform learnable context optimization on the defect category description, generating semantic embeddings. Then, the Transformer decoder uses the text features as query vectors to interact with the insulator image features, updating the language feature stream through residual connections.

[0070] It should be noted that the insulator defect detection model is trained using a phased transfer learning training strategy based on the hybrid training set. A hybrid training set is constructed by combining artificial and real samples in a 4:6 ratio, and data augmentation strategies are implemented within this set. Phased transfer learning training is then performed using this hybrid training set. Since the insulator defect dataset is very small, direct training would lead to overfitting and low accuracy. For the few-shot problem, transfer learning learns prior knowledge from a large-scale dataset and transfers the model parameters to the small-shot dataset, thus improving the accuracy of target detection in small samples. Figure 4 As shown, the training is divided into two stages. The first stage uses a large number of artificial insulator defect samples to obtain a pre-trained model. The second stage loads the pre-trained model obtained in the first stage and trains it using a mixture of a certain number of artificial samples and real samples. The testing stage uses real insulator defect samples for testing.

[0071] In the first-order training, artificial samples were used for pre-training. The first three layers of the YOLOv11 backbone network were frozen, and the parameters of the higher-level networks were optimized. Classification and regression losses were calculated using artificial samples, enabling the model to initially grasp the basic semantic representation of the general structure of insulators and defect detection. In the second-order training, mixed samples from the mixed training set were used for joint training. All network parameters were unfrozen, a domain adaptation layer was introduced, and a gradient inversion layer was added after the output of the feature pyramid layer of the YOLOv11 network. Adversarial training was used to minimize the difference in feature distribution between artificial and real samples, effectively alleviating the domain shift problem in cross-domain transfer. The maximum mean difference loss was used to quantify the feature distance between artificial and real samples, forcing the model to learn domain-invariant features. Artificial and real samples were mixed proportionally in each batch, and the weight of artificial samples was gradually reduced to shift the training focus of the model towards real data. A validation set was built using real samples, and a cross-validation strategy was used to validate the model. When the model performance met the preset requirements, the model parameters with the minimum validation loss were output to obtain the trained insulator defect detection model. Lightweight designs such as pruning, quantization, and distillation are implemented on the YOLOv11 network to ensure its efficient operation on low-computing-power, low-memory edge computing platforms (such as RK3588), enabling real-time detection of insulator defects and meeting the needs of intelligent operation and maintenance of the power grid.

[0072] It should be noted that in the training of the insulator defect detection model, an improved focus loss function and GIoU regression loss are designed to form a balanced training model with two branches: classification and regression. This addresses the problems of sample distribution imbalance and localization bias, improving the model's ability to detect easily missed defects and its localization accuracy. The improved focus loss function is introduced into the classification loss. This function dynamically adjusts the classification loss weights through the pixel-text matching score map, making the model pay more attention to difficult-to-classify samples and easily missed defects during training. The dynamic weight adjustment mechanism alleviates the class imbalance problem, making the model focus more on difficult-to-classify samples and enhancing classification robustness in complex scenarios. The bounding box regression loss adopts a distribution-aware cross-entropy loss, modeling the bounding box coordinate prediction as a probability distribution problem. By measuring the difference between the predicted distribution and the true offset, the bounding box position is precisely adjusted. The GIoU regression loss uses the pixel-text score map provided by DenseCLIP to correct the bounding box offset, improving the model's ability to detect easily missed defects and its localization accuracy. Meanwhile, the text embedding of the language model is optimized through a context-aware prompting strategy, including pre-language model prompts and post-language model prompts, to further improve the model's detection accuracy for small target defects.

[0073] Figure 3 A flowchart is shown to obtain the predicted results of defects in the insulator to be detected.

[0074] According to an embodiment of the present invention, the preprocessed image of the insulator to be detected is imported into the deployed insulator defect detection model to obtain the enhanced feature map after fusion of visual and language embedding vectors and the pixel-text score map, thereby generating the final prediction result, specifically as follows:

[0075] S302, the image of the insulator to be detected is obtained and standardized as the input of the insulator defect detection model. Through the DarkNet backbone network of the YOLOv11 network, multi-scale feature maps are generated through multi-layer convolution and feature pyramid.

[0076] S304, load predefined defect category text hints, generate language embedding vectors for defect categories through DenseCLIP's text encoder, dynamically optimize text features using post-language model hints, extract pixel-level visual embedding vectors from the input image using DenseCLIP's visual encoder, calculate cosine similarity with language feature vectors, and generate pixel-text score maps.

[0077] S306, the pixel-text score map is downsampled to the same size as the feature maps of each layer of the YOLOv11 network, the visual embedding vector is semantically weighted by multiplying channel by channel, and a cross-attention module is inserted in front of the detection head of the YOLOv11 network to generate an enhanced feature map that integrates semantic information.

[0078] S308, The enhanced feature map is input into the detection head of the YOLOv11 network to predict the bounding box, class confidence and defect localization accuracy at multiple scales. The average pixel-text score within the detection box is used as the semantic confidence, which is multiplied by the class confidence of YOLOv11 to obtain the final defect score.

[0079] S310, Based on the final defect score, obtain a prediction box that meets the preset requirements as the final defect detection result.

[0080] It should be noted that the insulator image to be detected is adjusted to the model input size, maintaining the aspect ratio and avoiding deformation through edge padding. The adjusted insulator image is then normalized and converted to a color space consistent with the training. Visual feature extraction is performed in the YOLOv11 network, retaining high-resolution feature layers specifically for handling small target defects. Pixel-text scores are used to identify the semantic association strength between each pixel region in the image and each category of text (value range [-1,1]), for example, high-score regions (>0.7) may correspond to rust descriptions. In semantically guided feature weighting, visual features are semantically weighted by multiplying channel by channel, highlighting regions related to defects. For example, high-score regions of rust text will enhance the convolutional feature response at the corresponding location. Through context-aware attention fusion, using text embeddings as query vectors and visual features as key-value pairs, enhanced features incorporating semantic information are generated, enabling the model to dynamically adjust the detection strategy based on language descriptions.

[0081] The YOLOv11 network's detection head predicts bounding boxes using a 160×160 large-scale feature map for detecting minute defects, such as crack width; an 80×80 medium-scale feature map for identifying medium-sized defects, such as localized corrosion; and a 40×40 small-scale feature map for locating overall anomalies, such as missing insulators. The mean pixel-to-text score within the detected bounding box is used as the semantic confidence score, multiplied by the YOLOv11 classification confidence score to obtain the final defect score. Predicted boxes with low semantic scores are filtered to reduce false positives, and predicted boxes with high overlap are merged, retaining the highest-scoring detection results. The output is a labeled detection image, including defect category labels and bounding boxes. Preferably, the detection results are output in a structured format, generating a JSON report containing defect location (pixel coordinates), category, confidence score, and semantic matching score for subsequent processing by the maintenance system.

[0082] Figure 5 A block diagram of a machine vision-based insulator defect detection system for power transmission lines is shown.

[0083] The second aspect of the present invention provides a machine vision-based transmission line insulator defect detection system 5, which includes a data acquisition and preprocessing module 501, a sample generation and annotation module 502, an insulator defect detection module 503, a mixed sample training and transfer learning module 504, and a reasoning and result generation module 505.

[0084] The data acquisition and preprocessing module acquires insulator images from multiple sources and preprocesses them.

[0085] The sample generation and annotation module is responsible for constructing digital twins of insulators, generating artificial samples with different defect patterns, and constructing real samples with defect labels based on three types of annotation systems including normal insulators, broken defects, and flashover defects from real images collected from multiple scenarios.

[0086] The insulator defect detection module combines the pre-trained DenseCLIP model with the YOLOv11 network. It extracts insulator image features through the visual encoder of the DenseCLIP model and converts them into visual embedding vectors. It constructs text prompts based on defect categories and converts them into language embedding vectors. It calculates the similarity between visual and language embedding vectors and generates a pixel-text matching score map to guide the YOLOv11 network learning and build an insulator defect detection model.

[0087] The hybrid sample training and transfer learning module constructs a hybrid training set by combining artificial samples and real samples in a preset ratio. Based on the hybrid training set, a phased training strategy of transfer learning is used to train the insulator defect detection model, and the trained insulator defect detection model is output for deployment.

[0088] The reasoning and result generation module imports the preprocessed image of the insulator to be detected into the deployed insulator defect detection model, obtains the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and generates the final prediction result.

[0089] A third aspect of the present invention provides a computer-readable storage medium including a machine vision-based method program for detecting defects in transmission line insulators. When the machine vision-based method program for detecting defects in transmission line insulators is executed by a processor, it implements the steps of the machine vision-based method for detecting defects in transmission line insulators.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms. Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0091] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting defects in transmission line insulators based on machine vision, characterized in that, Includes the following steps: Digital twins of insulators are constructed, artificial samples with different defect patterns are generated, and real images from multiple scenarios are collected. Based on three types of labeling systems, including normal insulators, broken defects, and flashover defects, real samples with defect labels are constructed. The pre-trained DenseCLIP model is combined with the YOLOv11 network. The visual encoder of the DenseCLIP model extracts the features of the insulator image and converts them into visual embedding vectors. Text prompts are constructed according to the defect category and converted into language embedding vectors. The similarity between the visual and language embedding vectors is calculated, and a pixel-text matching score map is generated to guide the YOLOv11 network to learn and build an insulator defect detection model. The artificial samples and the real samples are constructed into a mixed training set according to a preset ratio. Based on the mixed training set, the insulator defect detection model is trained using a phased training strategy of transfer learning. The trained insulator defect detection model is then deployed. The image of the insulator to be detected is imported into the deployed insulator defect detection model to obtain the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and the final prediction result is generated. The insulator image features are extracted using the CLIP model's visual encoder and converted into visual embedding vectors. Text cues are constructed based on defect categories and converted into linguistic embedding vectors. The similarity between the visual and linguistic embedding vectors is calculated, and a pixel-to-text matching score map is generated to guide the YOLOv11 network learning, thus constructing an insulator defect detection model. Specifically: An insulator defect detection model is constructed based on the pre-trained DenseCLIP model and YOLOv11 network. The DarkNet backbone network of the YOLOv11 network is used to extract multi-scale insulator image features, and the original feature pyramid structure is retained for target detection to obtain visual feature flow. Load the pre-trained DenseCLIP model, use the CLIP-based visual encoder to extract pixel-level visual embeddings, and use the text encoder to generate semantic embeddings of insulator defect categories to obtain the language feature stream; The similarity between the visual embedding vectors in the visual feature stream and the language embedding vectors in the language feature stream is calculated to obtain a similarity matrix. A pixel-text matching score map is generated to represent the semantic association strength between the identified image region and the defect category, guiding the YOLOv11 network learning. A cross-attention module is inserted before the detection head of the YOLOv11 network to dynamically weight and fuse the visual embedding vector of the YOLOv11 object detection network with the language embedding vector of DenseCLIP, thereby enhancing the sensitivity to low-contrast defects. A small target detection layer is added to the top of the YOLOv11 feature pyramid. It directly receives the high-density feature input from DenseCLIP and downsamples the pixel-text score map of DenseCLIP to different scales. It is then concatenated with the multi-level feature map of YOLOv11 channel by channel. The defect detection result and defect localization are obtained through the detection head.

2. The method for detecting defects in transmission line insulators based on machine vision according to claim 1, characterized in that, A digital twin of an insulator is constructed, generating artificial samples with different defect patterns. Simultaneously, real-world images from multiple scenarios are acquired. Based on a three-category annotation system encompassing normal insulators, damaged insulators, and flashover defects, real-world samples with defect labels are constructed. Specifically: The geometric specifications and material information of the insulator are obtained as the basis for modeling. A parametric model is established in 3ds Max software to generate three-dimensional models of insulators with different voltage levels. Based on the three-dimensional models of the insulators, the optical properties of the insulating material in the insulators are configured, and the surface states of different aging degrees are simulated. A digital twin of the insulator is generated using the simulated 3D model of the insulator, and artificial sample datasets of different defect modes are constructed through Boolean operations. Real images of insulators under multiple scenarios are acquired using a multi-source acquisition method. A three-category annotation system is constructed, including normal insulators, damaged insulators, and flashover defects. The real images of insulators are then labeled with defect categories using the annotation system to construct a real sample dataset.

3. The method for detecting defects in transmission line insulators based on machine vision according to claim 1, characterized in that, The text embedding of the language model is optimized using a context-aware prompting strategy, specifically: The system automatically generates a suitable text prompt template based on the content of the input insulator image. The image context information is injected into the text encoding process through the Transformer module of DenseCLIP, visual context features are extracted, and the visual context features are combined with a learnable query vector to generate dynamic text prompts. After the dynamic text prompts are generated, the CoOp method is used to perform learnable context optimization on the defect category description using post-language model prompts, generating semantic embeddings. Then, the Transformer decoder uses the text features as query vectors to interact with the insulator image features, and updates the language feature stream through residual connections.

4. The method for detecting defects in transmission line insulators based on machine vision according to claim 1, characterized in that, A hybrid training set is constructed by combining the artificial samples and the real samples according to a preset ratio. Based on this hybrid training set, a phased training strategy using transfer learning is employed to train the insulator defect detection model. Specifically: A hybrid training set is constructed by combining artificial and real samples according to a preset ratio, and a data augmentation strategy is executed in the hybrid training set. The hybrid training set is then used for phased transfer learning training. In the first-order training, artificial samples were used for pre-training, the first three layers of the YOLOv11 backbone network were frozen, the parameters of the high-level network were optimized, and the classification loss and regression loss were calculated using artificial samples, so that the model could initially grasp the basic semantic representation of the general structure and defect detection of insulators. In the second-order training, mixed samples from the mixed training set are used for joint training, all network parameters are unfrozen, a domain adaptation layer is introduced, a gradient reversal layer is added after the output of the feature pyramid layer of the YOLOv11 network, the feature distribution difference between artificial samples and real samples is minimized through adversarial training, and the feature distance between artificial samples and real samples is quantified using the maximum mean difference loss, forcing the model to learn domain-invariant features. A validation set is constructed using real samples, and a cross-validation strategy is used to validate the model. When the model performance meets the preset requirements, the model parameters with the minimum validation loss are output to obtain the trained insulator defect detection model.

5. The method for detecting defects in transmission line insulators based on machine vision according to claim 4, characterized in that, In the training of the insulator defect detection model, an improved focus loss function and GIoU regression loss are designed to form a balanced training of classification and regression branches. The improved focus loss function dynamically adjusts the classification loss weight through the pixel-text matching score map, so that the model pays more attention to difficult-to-classify samples and defects that are easy to miss during the training process. The GIoU regression loss uses the pixel-text score map provided by DenseCLIP to correct the bounding box offset, thereby improving the model's ability to detect easily missed defects and its localization accuracy.

6. The method for detecting defects in transmission line insulators based on machine vision according to claim 1, characterized in that, The preprocessed insulator image to be detected is imported into the deployed insulator defect detection model to obtain the enhanced feature map after fusion of visual and language embedding vectors and the pixel-text score map, generating the final prediction result, specifically: The images of the insulators to be detected are obtained and standardized as input to the insulator defect detection model. Multi-scale feature maps are generated through the DarkNet backbone network of the YOLOv11 network via multi-layer convolution and feature pyramid. Load predefined defect category text hints, generate language embedding vectors for defect categories using DenseCLIP's text encoder, dynamically optimize text features using post-language model hints, extract pixel-level visual embedding vectors from the input image using DenseCLIP's visual encoder, calculate cosine similarity with language feature vectors, and generate pixel-text score maps. The pixel-text score map is downsampled to the same size as the feature maps of each layer of the YOLOv11 network. The visual embedding vector is semantically weighted by multiplying through each channel. A cross-attention module is inserted in front of the detection head of the YOLOv11 network to generate an enhanced feature map that integrates semantic information. The enhanced feature map is input into the detection head of the YOLOv11 network to predict bounding boxes, class confidence, and defect localization accuracy at multiple scales. The average pixel-text score within the detection box is used as the semantic confidence, which is multiplied by the class confidence of YOLOv11 to obtain the final defect score. Based on the final defect score, a prediction box that meets the preset requirements is obtained as the final defect detection result.

7. A machine vision-based defect detection system for transmission line insulators, characterized in that, The system implements the machine vision-based transmission line insulator defect detection method as described in any one of claims 1-6, comprising a data acquisition and preprocessing module, a sample generation and annotation module, an insulator defect detection module, a hybrid sample training and transfer learning module, and an inference and result generation module; The data acquisition and preprocessing module acquires insulator images from multiple sources and preprocesses them. The sample generation and annotation module is responsible for constructing digital twins of insulators, generating artificial samples with different defect patterns, and constructing real samples with defect labels based on three types of annotation systems including normal insulators, broken defects, and flashover defects from real images collected from multiple scenarios. The insulator defect detection module combines the pre-trained DenseCLIP model with the YOLOv11 network. It extracts insulator image features through the visual encoder of the DenseCLIP model and converts them into visual embedding vectors. It constructs text prompts based on defect categories and converts them into language embedding vectors. It calculates the similarity between visual and language embedding vectors and generates a pixel-text matching score map to guide the YOLOv11 network learning and build an insulator defect detection model. The hybrid sample training and transfer learning module constructs a hybrid training set by combining artificial samples and real samples in a preset ratio. Based on the hybrid training set, a phased training strategy of transfer learning is used to train the insulator defect detection model, and the trained insulator defect detection model is output for deployment. The reasoning and result generation module imports the preprocessed image of the insulator to be detected into the deployed insulator defect detection model, obtains the enhanced feature map after the fusion of visual and language embedding vectors and the pixel-text score map, and generates the final prediction result.

Images