Power transmission line foreign matter detection method based on multi-component semantic segmentation and hybrid Transform-CNN

Abstract

The invention relates to the field of intelligent monitoring of electric power facilities, in particular to a power transmission line foreign matter detection method based on multi-component semantic segmentation and hybrid Transform-CNN (Convolutional Neural Network). According to the method, the Transform module is combined with the CNN model, multi-component semantic segmentation is performed on the to-be-analyzed image through the decoder based on the UNet network, and the method can be applied to rapid detection under complex conditions and numerous interference environments. Therefore, the foreign matter detection of the power transmission line for performing high-precision segmentation and real-time detection on the attached foreign matters on multiple parts such as a lead, an insulator and a fitting of the power transmission line in a complex scene is realized, the safety level alarm is performed when an abnormal trigger mechanism is reached, and the test shows that the IoU is improved by 15% and the false detection rate is reduced by 40% in rainy and foggy weather, so that the safety of the power transmission line is improved. The FPS reaches 32 frames per second (real-time edge calculation is met), the accuracy and robustness of foreign matter detection of the power transmission line are greatly improved, and the timeliness of safety early warning of the power transmission line is also greatly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring of power facilities, and in particular to a method for detecting foreign objects on power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN. Background Technology

[0002] Currently, there is a lack of prior knowledge about power transmission scenarios. The difficulty in detection under extreme weather conditions lies in poor image quality, such as interference from rain, snow, and fog, which may affect the performance of the model. At the same time, there is a lack of explicit modeling of the spatial distribution of conductors and the co-occurrence probability of foreign objects and insulators, resulting in a high false detection rate of equipment.

[0003] Traditional foreign object detection methods rely on manual inspection or threshold segmentation, which are greatly affected by lighting, weather, and background interference. They cannot adapt to complex environments, such as rain, snow, and fog, which cause image blurring and significant background noise, making it difficult to solve the interference of foreign objects and dynamic environments.

[0004] Meanwhile, existing deep learning models are mostly designed for single components (such as insulators), without considering the high false detection rate when components overlap or occlude due to the spatial correlation of multiple components. Furthermore, various factors such as the large differences in the size and shape of foreign objects result in a limited receptive field of traditional CNNs, making it difficult to capture global contextual information.

[0005] However, segmentation models based on bi-branch or multi-branch networks (such as variants of U-Net) have high computational costs, insufficient real-time performance, and are difficult to deploy to edge devices.

[0006] Especially when rain and snow cause image blurring, the threshold segmentation method has a false detection rate of over 60%; existing CNN models have a false detection rate of 35% for foreign objects <5cm.

[0007] Therefore, there is a need for a foreign object detection method for power transmission lines that can perform high-precision segmentation and real-time detection of foreign objects attached to multiple components such as conductors, insulators, and fittings in complex scenarios. Summary of the Invention

[0008] The purpose of this invention is to overcome the above-mentioned shortcomings in the prior art and provide a method for detecting foreign objects in power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0010] A method for detecting foreign objects in power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN includes the following steps:

[0011] a: Obtain the image to be analyzed;

[0012] b: Input the image to be analyzed into a pre-trained transmission line anomaly image detection model;

[0013] c: The abnormal image detection model of the transmission line outputs the foreign object detection result of the image to be analyzed;

[0014] d: When the foreign object detection result satisfies the anomaly triggering mechanism built into the transmission line anomaly image detection model, a safety level alarm is issued;

[0015] The abnormal image detection model for power transmission lines includes an encoder, a Transformer attention mechanism module, and a decoder connected in sequence; the encoder uses a ResNet network, and the decoder is based on the UNet network architecture.

[0016] As a preferred embodiment of the present invention, the pre-training of the transmission line anomaly image detection model includes the following steps:

[0017] S1: Acquire multi-angle images of power transmission lines, as well as images of power transmission lines containing foreign objects; and after removing low-quality images, label and classify the remaining images, outputting the initial dataset.

[0018] The labeling classification includes component category and foreign object type; the component category includes wires, insulators, and hardware; the foreign object type includes bird nests and hanging objects.

[0019] S2: The initial dataset is preprocessed using the SAM model to obtain the augmented dataset;

[0020] S3: Divide the augmented dataset evenly into a training set, a validation set, and a test set according to a preset ratio;

[0021] S4: Construct an anomaly image detection model for transmission lines;

[0022] S5: Construct a knowledge graph and an anomaly triggering mechanism, and embed the anomaly image detection model of the transmission line;

[0023] S6: Input the training set into the transmission line anomaly image detection model for training. After the model converges, the trained transmission line anomaly image detection model is obtained.

[0024] As a preferred embodiment of the present invention, step S2 includes the following steps:

[0025] S21: Input the images from the initial dataset into the SAM model to generate masks for foreign object regions and power transmission components;

[0026] S22: Based on the generated mask, perform various enhancement operations on the image; the various enhancement operations include one or more of geometric transformation, photometric transformation, and defect simulation processing;

[0027] S23: Perform mask correction on the enhanced image to make the image annotation information correspond to the image content, and output as an enhanced dataset.

[0028] As a preferred embodiment of the present invention, the encoder further includes a lightweighting process for the last n convolutional layers; the lightweighting process includes:

[0029] Multi-scale convolution processing: 3×3 and 5×5 convolution kernels are used in parallel in each residual block to fuse multi-scale contextual information through feature concatenation, thereby enhancing adaptability to defects of different sizes;

[0030] Noise-resistant convolution processing: A BCN layer is added after the convolutional layer, and the input feature map is preprocessed with Gaussian blur.

[0031] As a preferred embodiment of the present invention, the Transformer attention mechanism module includes two layers of axial Transformer blocks, and the axial Transformer blocks include learnable two-dimensional position codes.

[0032] As a preferred embodiment of the present invention, the decoder further includes a channel attention module added before the input; and replaces the bilinear interpolation upsampling layer of the original UNet network with an upsampling layer with deformable convolution.

[0033] As a preferred embodiment of the present invention, the exception triggering mechanism in S5 includes:

[0034] S51: Based on the knowledge graph, define the mapping relationship between the number of insulators and the voltage level;

[0035] S52: When an abnormal number of insulators is detected, it is determined that the voltage level is mismatched, and the corresponding safety level alarm is triggered according to the set triggering conditions.

[0036] S53: For abnormal areas detected by visible light images, perform secondary verification using infrared thermal imaging data to determine whether the detected abnormality is a false detection caused by light and shadow interference. If it is determined to be a false detection, cancel the alarm and upload the judgment result.

[0037] As a preferred embodiment of the present invention, the security level alarm includes level one, level two, and level three;

[0038] Level 1 response refers to high-risk issues that require immediate handling. Its triggering conditions are one or more of the following: conductor breakage, insulator bursting, and hardware detachment.

[0039] Level 2 refers to medium-level problems that need to be addressed within 24 hours. Its triggering conditions are one or more of the following: vibration damper misalignment, pressure equalization ring loosening, and porcelain insulator cracking.

[0040] Level 3 refers to potential hazards that need to be addressed within 7 days. The triggering conditions are one or more of the following: loose nuts, minor corrosion, and small foreign objects.

[0041] Furthermore, when the alarm location is in a strong wind area or a high humidity area affected by extreme weather, the safety level alarm will be upgraded by one level.

[0042] As a preferred embodiment of the present invention, the pre-training further includes S7:

[0043] S7: Input the validation set into the trained transmission line anomaly image detection model, and evaluate the model by using performance evaluation metrics;

[0044] The performance evaluation metrics include Intersection over Union (IoU), Dice coefficient, average IoU, precision, recall, average accuracy (AP), average precision (mAP), and frames per second (FPS). The calculation formulas for the performance evaluation metrics are as follows:

[0045] Intersection over Union (IoU):

[0046]

[0047] Dice coefficient:

[0048]

[0049] Average IoU:

[0050]

[0051] Average Precision (AP):

[0052] AP = Area under the Precision-Recall curve

[0053]

[0054] Mean accuracy mAP:

[0055]

[0056] Where A is the predicted region, B is the true region, N is the number of categories, and IoU is the distance between the predicted region and the true region. iLet be the intersection-union ratio (IoU) for the i-th class, be precision, be recall, be TP (the number of instances that were actually positive but were classified as positive by the classifier), be FP (the number of instances that were actually negative but were classified as positive by the classifier), be FN (the number of instances that were actually positive but were classified as negative by the classifier), and be AP (the number of instances that were actually positive but were classified as negative by the classifier). i Let be the average precision of the i-th category.

[0057] A foreign object detection device for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the methods described above.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] This invention combines a Transformer module with a CNN model and uses a UNet-based decoder to perform multi-component semantic segmentation on the image to be analyzed, making it applicable for rapid detection under complex conditions and numerous interferences. This enables high-precision segmentation and real-time detection of foreign objects (FOOs) on multiple components of power transmission lines, such as conductors, insulators, and fittings, in complex scenarios. When an anomaly trigger mechanism is reached, a safety level alarm is issued. Testing showed a 15% improvement in IoU (Interval of Value) and a 40% reduction in false detection rate under rainy and foggy weather conditions, achieving an FPS of 32 frames per second (meeting real-time edge computing requirements). This significantly improves the accuracy and robustness of FEO detection on power transmission lines and greatly enhances the timeliness of safety warnings for power transmission lines. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the method flow of a foreign object detection method for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN as described in Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram of the model pre-training process of the abnormal image detection model of the transmission line in the foreign object detection method of the transmission line based on multi-component semantic segmentation and hybrid Transformer-CNN as described in Embodiment 2 of the present invention;

[0062] Figure 3 This is a schematic diagram of the S2 preprocessing process in the foreign object detection method for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN described in Embodiment 2 of the present invention;

[0063] Figure 4This is a schematic diagram of the architecture of the abnormal image detection model for transmission lines in the foreign object detection method for transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN described in Embodiment 2 of the present invention;

[0064] Figure 5 This is a schematic diagram of the knowledge graph structure in the foreign object detection method for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN described in Embodiment 2 of the present invention;

[0065] Figure 6 This is a schematic diagram of the structure of a foreign object detection device for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN, which utilizes the foreign object detection method for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN described in the foregoing embodiments of the present invention, as described in Embodiment 3 of the present invention. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0067] This invention proposes a cross-domain knowledge-enhanced multi-scale semantic segmentation framework. By transferring tumor detection techniques from medical imaging, such as multi-scale context perception and contrastive learning, to power transmission line anomaly detection, and combining scene priors and sequential classification strategies, it achieves high-precision foreign object localization and classification under extreme weather conditions. Then, based on the knowledge graph of power transmission line towers, it identifies hardware, estimates the voltage level of the power transmission line by the number of insulators, and estimates the surrounding environment by location or background to identify whether there are anomalies in the component assembly.

[0068] Example 1

[0069] like Figure 1 As shown, a method for detecting foreign objects in power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN includes the following steps:

[0070] a: Obtain the image to be analyzed.

[0071] b: Input the image to be analyzed into a pre-trained transmission line anomaly image detection model.

[0072] c: The abnormal image detection model of the transmission line outputs the foreign object detection result of the image to be analyzed.

[0073] d: When the foreign object detection result satisfies the abnormal triggering mechanism built into the transmission line abnormal image detection model, a safety level alarm is issued.

[0074] The abnormal image detection model for power transmission lines includes an encoder, a Transformer attention mechanism module, and a decoder connected in sequence; the encoder uses a ResNet network; and the decoder is based on the UNet network architecture.

[0075] Example 2

[0076] This embodiment is a specific implementation of the pre-training of the transmission line anomaly image detection model in the transmission line foreign object detection method based on multi-component semantic segmentation and hybrid Transformer-CNN described in Embodiment 1. Figure 2 As shown, it includes the following steps:

[0077] S1: Obtain multi-angle images of power transmission lines, as well as images of power transmission lines containing foreign objects; and after removing low-quality images, label and classify the remaining images, outputting the initial dataset.

[0078] The labeling classification includes component categories and foreign object types; the component categories include wires, insulators, and hardware; the foreign object types include bird nests and hanging objects.

[0079] In this embodiment, a drone equipped with a high-resolution camera is used to conduct inspections along the power transmission line, enabling clear capture of details along the line. The drone flies along a preset route, capturing images of key components such as towers, conductors, insulators, and fittings from multiple angles to ensure the capture of abnormal image details such as damaged fittings, bird nests, and hanging objects on the conductors. All acquired images are then transmitted to local storage or a server. The images undergo initial screening, using image quality assessment algorithms to remove low-quality images that are blurry, have abnormal exposure (overexposure or underexposure exceeding thresholds), or have poor angles (shooting angle deviation exceeding 30°). The remaining valid images are categorized according to component type (conductors, insulators, fittings) and object type (bird nests, hanging objects), and pixel-level semantic annotations are performed using professional annotation tools (such as LabelMe). The annotations include target outlines and category labels, forming an initial dataset.

[0080] S2: As Figure 3 As shown, the initial dataset is preprocessed using the SAM model to obtain the augmented dataset.

[0081] S21: Input the images from the initial dataset into the SAM (Segment Anything Model). Utilize its efficient image segmentation capabilities to generate masks for foreign object regions (bird nests, hanging objects) and power transmission components (wires, insulators, hardware). Simultaneously, the mask resolution is consistent with the original image to ensure pixel-level alignment.

[0082] S22: Based on the generated mask, perform various enhancement operations on the image; the various enhancement operations include one or more of geometric transformation, photometric transformation, and defect simulation processing.

[0083] Furthermore, in this embodiment, the geometric transformations include: random rotation (15), horizontal / vertical flipping, and scaling (0.8-1.2 times). The photometric transformations include: adding Gaussian noise (standard deviation ≤ 20), adjusting brightness (±20%), contrast (15%), and saturation (10%). The defect simulation includes: simulating early defect morphologies such as minor cracks and stains in defect-free areas using mask fusion technology.

[0084] S23: Perform mask correction on the enhanced image to make the image annotation information correspond to the image content, and output as an enhanced dataset.

[0085] The mask correction is achieved by filling mask holes using morphological closing operations and adjusting mask boundaries using an edge alignment algorithm.

[0086] S3: Divide the augmented dataset evenly into a training set, a validation set, and a test set according to a preset ratio.

[0087] Furthermore, this embodiment uses a 7:2:1 ratio for allocation, and employs a stratified sampling method during the partitioning process to ensure that each subset maintains a balanced distribution in terms of component categories and foreign object types.

[0088] S4: Construct an anomaly image detection model for transmission lines;

[0089] A power transmission line anomaly image detection model is constructed for UAV detection platforms with limited image quality under extreme weather conditions. A single-branch real-time semantic segmentation network architecture is adopted, which integrates the Transformer attention mechanism and an improved UNet to balance detection accuracy and real-time performance.

[0090] like Figure 4 As shown, the model consists of three parts: an encoder, a Transformer feature enhancement module, and a decoder. The encoder includes a multi-scale noise-resistant design, potentially using multiple parallel processing branches, each handling inputs at different scales, and includes noise-resistant annotations. The Transformer module uses an axial attention mechanism, which may require different attention modules, such as an axial segmentation attention head. The decoder includes channel attention and deformable upsampling. Channel attention can be achieved by connecting feature maps using attention modules, while deformable upsampling may be represented by deformable convolution or interpolation methods. The entire architecture should be lightweight and efficient, for example, by using parameter sharing or a reduced number of layers. High-precision segmentation is achieved through feature fusion or skip connections.

[0091] The specific structure is as follows:

[0092] Encoder design (CNN feature extraction):

[0093] The encoder also includes a lightweighting process for the last n convolutional layers;

[0094] Furthermore, taking the ResNet-34 model as an example, this embodiment uses ResNet-34 as the basic backbone network, retaining the first three convolutional layers (Conv1-Conv3) for shallow feature extraction, and introducing lightweight improvements in the Conv4-Conv5 layers. The lightweight processing includes:

[0095] Multi-scale convolution processing: 3×3 and 5×5 convolution kernels are used in parallel in each residual block to fuse multi-scale contextual information through feature concatenation, thereby enhancing adaptability to defects of different sizes (such as micro cracks and large bird nests).

[0096] Noise-resistant convolution processing: A BCN (Batch-Channel Normalization) layer is added after the convolutional layer, and the input feature map is preprocessed with Gaussian blur (kernel size 3×3, standard deviation 0.5) to improve robustness to low-quality images.

[0097] Transformer attention mechanism module design:

[0098] The Transformer attention mechanism module includes two layers of axial Transformer blocks to achieve global context modeling. Specifically, it includes the following design:

[0099] 1. Axial attention mechanism: Expand the two-dimensional feature map by rows and columns respectively, and perform one-dimensional self-attention calculation to reduce computational complexity (from O(H)). 2 W 2 The computational efficiency is reduced to O(HW(H+W)), which is suitable for the lightweight computing needs of drone platforms.

[0100] 2. Location code fusion: Learnable two-dimensional location codes are added to the Transformer input, combined with prior spatial structure of transmission components (such as periodic arrangement of insulators), to enhance the model's understanding of the spatial relationships of components.

[0101] Decoder Design (Improved UNet):

[0102] 1. Skip Connection Optimization: Introduce a Channel Attention Module (CAM) in the skip connection between the encoder and decoder to assign weights to the shallow features output by the encoder and focus on channels with high semantic information (such as edge and texture channels).

[0103] 2. Upsampling improvement: An upsampling layer with deformable convolution is used to replace the traditional bilinear interpolation in the original UNet network. The sampling position is dynamically adjusted according to the input features to improve the segmentation accuracy of edge regions (such as sharp angles of hardware and wire contours).

[0104] Output layer design:

[0105] The feature map is mapped to the number of categories (5 categories in total, including parts and foreign objects) using a 1×1 convolution. A pixel-level classification probability map is generated using the Softmax function, and the output resolution is consistent with the input image.

[0106] S5: Construct a knowledge graph and an anomaly triggering mechanism, and embed the abnormal image detection model of the transmission line.

[0107] Furthermore, such as Figure 5 As shown, the knowledge graph includes an entity layer, a relation layer, and an attribute layer. The entity layer includes nodes such as insulators and voltage levels; the relation layer includes relationships such as "matching" and "triggering anomalies"; and the attribute layer includes attributes such as the number of insulators and the voltage range. The matching logic between the number of insulators and the voltage level needs to be highlighted, possibly indicated by red arrows or special annotations. The anomaly triggering relationship connects entities and attributes; it is triggered when the quantities do not match. Its construction includes:

[0108] Based on expert knowledge in the field of power transmission lines, a knowledge graph containing the following entities and relationships is constructed:

[0109] 1. Entity type: poles, conductors, insulators, fittings, voltage level, safety specifications, defect type.

[0110] 2. Relationship definition:

[0111] "Installed at" (insulator-tower);

[0112] "Matching voltage level" (fittings - voltage level);

[0113] "May lead to" (Defect type - security hazard).

[0114] Attribute settings: Insulators include attributes such as "quantity", "model" and "string length", while fittings include attributes such as "material" and "applicable voltage level".

[0115] Furthermore, in the semantic segmentation results output by the model, insulator instances are counted (based on connected component analysis). When a mismatch is detected between the number of insulators on a certain tower and the standard number of insulators (within a 5% fluctuation range) for that type of tower in the knowledge graph, a "voltage level mismatch" alarm is triggered. Specifically, the abnormal triggering mechanism construction process includes:

[0116] 1. Instance counting: The insulator segmentation mask is segmented into instances using the watershed algorithm, and the number N of insulators on a single tower is counted.

[0117] 2. Knowledge Query: Based on the tower model, retrieve the standard insulator quantity N_std and allowable fluctuation range [L,U] from the knowledge graph.

[0118] 3. Anomaly detection: If An alarm event will be generated, associated with the potential risk of voltage level mismatch (such as insufficient insulation strength).

[0119] The specific implementation process of the exception triggering mechanism includes:

[0120] S51: Based on the knowledge graph, define the mapping relationship between the number of insulators and the voltage level (e.g., a single string of insulators for a 110kV line has approximately 7 pieces);

[0121] S52: When an abnormal number of insulators is detected (e.g., deviation > ±10%), it is determined that the voltage level is mismatched, and the corresponding safety level alarm is triggered according to the set triggering conditions.

[0122] S53: For abnormal areas detected by visible light images, perform secondary verification using infrared thermal imaging data to determine whether the detected abnormality is a false detection caused by light and shadow interference. If it is determined to be a false detection, cancel the alarm and upload the judgment result.

[0123] This step involves secondary verification of abnormal areas detected by visible light images, combined with infrared thermal imaging data (temperature difference > 5℃), to eliminate false detections caused by light and shadow interference, thereby reducing the false detection rate by 40%.

[0124] The security level alarms include Level 1, Level 2, and Level 3;

[0125] Level 1 is a high-risk issue that requires immediate attention and is likely to cause power grid paralysis or disaster risk. Its triggering conditions are one or more of the following: conductor breakage, insulator bursting, and fitting detachment. In this embodiment, the triggering conditions are set as conductor breakage (IoU≥0.8), insulator bursting (area ratio>50%), and fitting detachment (detection confidence level≥0.95).

[0126] Level 2 refers to medium-sized problems that need to be addressed within 24 hours. The triggering conditions are one or more of the following: vibration damper offset, pressure equalization ring loosening, and porcelain insulator crack. In this embodiment, the triggering conditions are set as vibration damper offset (displacement > 5cm), pressure equalization ring loosening (angle deviation > 15°), and porcelain insulator crack (length > 10mm).

[0127] Level 3 refers to potential hazards that need to be addressed within 7 days, which can cause partial discharge and equipment damage. The triggering conditions are one or more of the following: loose nuts, minor corrosion, and small foreign objects. In this embodiment, the triggering conditions are set as follows: loose nuts (≥2 exposed threads), minor corrosion (area percentage 10%-30%), and small foreign objects (area <50cm²). 2 ).

[0128] Furthermore, if no safety level alarm is triggered and the false detection rate exceeds the threshold, the transmission line has no clear risk and is marked as pending review, which can be reviewed during the next inspection.

[0129] Furthermore, when the alarm location is in a strong wind area or a high humidity area affected by extreme weather, the safety level alarm is upgraded by one level. The specific safety level classification is shown in Table 1 below:

[0130] Table 1 Safety Level Classification Table

[0131]

[0132] Furthermore, the following foreign object detection results can be generated based on this table:

[0133] Defect Report: Match the segmentation results output by the model with defect templates in the knowledge base to generate a structured report containing the defect location (latitude and longitude coordinates), type, size, and confidence level, and attach a segmentation visualization image.

[0134] Dynamic level assessment: Combining real-time meteorological data (e.g., when humidity > 80%, the stain defect level is automatically upgraded by one level) and equipment operating parameters (e.g., when the wire temperature is abnormal, the risk of hanging object defects increases), the safety level is dynamically adjusted through a rule engine.

[0135] Multi-channel early warning: Based on the security level, early warning information is pushed through SMS, operation and maintenance platform pop-ups, emails and other means. Level 1 and Level 2 defects are accompanied by drone inspection video clips to assist operation and maintenance personnel in making quick decisions.

[0136] S6: Input the training set into the transmission line anomaly image detection model for training. After the model converges, the trained transmission line anomaly image detection model is obtained.

[0137] Furthermore, the pre-training also includes S7:

[0138] S7: Input the validation set into the trained transmission line anomaly image detection model, and evaluate the model by using performance evaluation metrics;

[0139] The performance evaluation metrics include Intersection over Union (IoU), Dice coefficient, average IoU, precision, recall, average accuracy (AP), average precision (mAP), and frames per second (FPS). The calculation formulas for the performance evaluation metrics are as follows:

[0140] Intersection over Union (IoU) is used to calculate the percentage of overlap between the predicted and ground truth masks.

[0141]

[0142] The Dice coefficient measures the similarity between the segmentation result and the true label:

[0143]

[0144] Average IoU: For multi-class segmentation tasks, calculate the mean IoU for each class.

[0145]

[0146] Average precision (AP) is calculated based on the area under the precision-recall curve (PR curve):

[0147] AP = Area under the Precision-Recall curve

[0148] Accuracy

[0149] Recall rate

[0150] Mean Average Precision (mAP): The mean AP of each class in a multi-class detection task.

[0151]

[0152] Where A is the predicted region, B is the true region, N is the number of categories, and IoU is the distance between the predicted region and the true region. i Let be the intersection-union ratio (IoU) of the i-th class, where TP represents the number of instances that are actually positive and are classified as positive by the classifier, FP represents the number of instances that are actually negative but are classified as positive by the classifier, FN represents the number of instances that are actually positive but are classified as negative by the classifier, and AP represents the number of instances that are actually positive but are classified as negative by the classifier. i Let be the average precision of the i-th category.

[0153] Through the above steps, the constructed single-branch real-time semantic segmentation network integrating Transformer-CNN can effectively address the image quality limitations under extreme weather conditions. Combined with knowledge graphs and security level mechanisms, it achieves accurate semantic segmentation and anomaly detection of multiple components of power transmission lines, providing reliable technical support for intelligent operation and maintenance. The above-disclosed embodiment is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Therefore, equivalent variations made according to the claims of the present invention are still within the scope of the present invention.

[0154] Example 3

[0155] like Figure 6 As shown, a foreign object detection device for power transmission lines based on multi-part semantic segmentation and hybrid Transformer-CNN includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the foreign object detection method for power transmission lines based on multi-part semantic segmentation and hybrid Transformer-CNN described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0156] Furthermore, the foreign object detection device for transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN can be a desktop computer, mobile phone, tablet computer, wearable device, or any other device capable of deep information recognition based on multi-component semantic segmentation and hybrid Transformer-CNN for transmission lines.

[0157] Furthermore, the processor may include one or more processing cores. The processor connects various parts within the transmission line foreign object detection device based on multi-component semantic segmentation and hybrid Transformer-CNN using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0158] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets, such as instructions or code sets used to implement the foreign object detection method for transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN provided in this application. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the foreign object detection device for transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN (such as a mapping table of modulation sequences and depth, image data, spectrogram data, etc.).

[0159] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0160] When the integrated units of the present invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. The computer-readable storage medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memories such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that executes any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting foreign objects in power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN, characterized in that, Includes the following steps: a: Obtain the image to be analyzed; b: Input the image to be analyzed into a pre-trained transmission line anomaly image detection model; c: The abnormal image detection model of the transmission line outputs the foreign object detection result of the image to be analyzed; d: When the foreign object detection result satisfies the anomaly triggering mechanism built into the transmission line anomaly image detection model, a safety level alarm is issued; The abnormal image detection model for power transmission lines includes an encoder, a Transformer attention mechanism module, and a decoder connected in sequence; the encoder uses a ResNet network, and the decoder is based on the UNet network architecture.

2. The method for detecting foreign objects in power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN as described in claim 1, characterized in that, The pre-training of the transmission line anomaly image detection model includes the following steps: S1: Acquire multi-angle images of power transmission lines, as well as images of power transmission lines containing foreign objects; and after removing low-quality images, label and classify the remaining images, outputting the initial dataset. The labeling classification includes component category and foreign object type; the component category includes wires, insulators, and hardware; the foreign object type includes bird nests and hanging objects. S2: The initial dataset is preprocessed using the SAM model to obtain the augmented dataset; S3: Divide the augmented dataset evenly into a training set, a validation set, and a test set according to a preset ratio; S4: Construct an anomaly image detection model for transmission lines; S5: Construct a knowledge graph and an anomaly triggering mechanism, and embed the anomaly image detection model of the transmission line; S6: Input the training set into the transmission line anomaly image detection model for training. After the model converges, the trained transmission line anomaly image detection model is obtained.

3. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, S2 includes the following steps: S21: Input the images from the initial dataset into the SAM model to generate masks for foreign object regions and power transmission components; S22: Based on the generated mask, perform various enhancement operations on the image; the various enhancement operations include one or more of geometric transformation, photometric transformation, and defect simulation processing; S23: Perform mask correction on the enhanced image to make the image annotation information correspond to the image content, and output as an enhanced dataset.

4. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, The encoder further includes a lightweighting process for the last n convolutional layers; the lightweighting process includes: Multi-scale convolution processing: 3×3 and 5×5 convolution kernels are used in parallel in each residual block to fuse multi-scale contextual information through feature concatenation, thereby enhancing adaptability to defects of different sizes; Noise-resistant convolution processing: A BCN layer is added after the convolutional layer, and the input feature map is preprocessed with Gaussian blur.

5. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, The Transformer attention mechanism module includes two layers of axial Transformer blocks, and the axial Transformer blocks include learnable two-dimensional position codes.

6. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, The decoder also includes a channel attention module added before the input; and replaces the original UNet network's bilinear interpolation upsampling layer with an upsampling layer with deformable convolution.

7. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, The exception triggering mechanism in S5 includes: S51: Based on the knowledge graph, define the mapping relationship between the number of insulators and the voltage level; S52: When an abnormal number of insulators is detected, it is determined that the voltage level is mismatched, and the corresponding safety level alarm is triggered according to the set triggering conditions. S53: For abnormal areas detected by visible light images, perform secondary verification using infrared thermal imaging data to determine whether the detected abnormality is a false detection caused by light and shadow interference. If it is determined to be a false detection, cancel the alarm and upload the judgment result.

8. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 7, characterized in that, The security level alarms include Level 1, Level 2, and Level 3; Level 1 response refers to high-risk issues that require immediate handling. Its triggering conditions are one or more of the following: conductor breakage, insulator bursting, and hardware detachment. Level 2 refers to medium-level problems that need to be addressed within 24 hours. Its triggering conditions are one or more of the following: vibration damper misalignment, pressure equalization ring loosening, and porcelain insulator cracking. Level 3 refers to potential hazards that need to be addressed within 7 days. The triggering conditions are one or more of the following: loose nuts, minor corrosion, and small foreign objects. Furthermore, when the alarm location is in a strong wind area or a high humidity area affected by extreme weather, the safety level alarm will be upgraded by one level.

9. The method for detecting foreign objects in transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN according to claim 2, characterized in that, The pre-training also includes S7: S7: Input the validation set into the trained transmission line anomaly image detection model, and evaluate the model by using performance evaluation metrics; The performance evaluation metrics include Intersection over Union (IoU), Dice coefficient, average IoU, precision, recall, average accuracy (AP), average precision (mAP), and frames per second (FPS). The calculation formulas for the performance evaluation metrics are as follows: Intersection over Union (IoU): Dice coefficient: Average IoU: Average Precision (AP): AP = Area under the Precision-Recall curve Mean accuracy mAP: Where A is the predicted region, B is the true region, N is the number of categories, and IoU is the distance between the predicted region and the true region. i Let be the intersection-union ratio (IoU) for the i-th class, be precision, be recall, be TP (the number of instances that were actually positive but were classified as positive by the classifier), be FP (the number of instances that were actually negative but were classified as positive by the classifier), be FN (the number of instances that were actually positive but were classified as negative by the classifier), and be AP (the number of instances that were actually positive but were classified as negative by the classifier). i Let be the average precision of the i-th category.

10. A foreign object detection device for power transmission lines based on multi-component semantic segmentation and hybrid Transformer-CNN, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

Images