Intelligent identification method and system for power distribution line defects based on visual multi-modal fusion

Abstract

The application provides a power distribution line defect intelligent identification method and system based on visual multi-modal fusion, and belongs to the technical field of power distribution line defect identification; solves the problem of low defect discovery rate of existing unmanned aerial vehicle inspection; comprises the following steps: collecting pictures of infrared cameras and zoom cameras at different far and near focal lengths through an unmanned aerial vehicle; solving an affine matrix for each group of collected infrared images and RGB images, and obtaining an affine matrix for affine transformation of the RGB images to the infrared images; training all collected RGB images using a yolov5l target detection model; cutting the infrared and visible light spliced video stream captured by the unmanned aerial vehicle, then transmitting the visible light video stream to the trained yolov5l target detection model for detection, judging the detected target, if it is a component defect, directly marking in the image, if it is a component target, mapping to the infrared image through the affine matrix, and judging whether the component has a thermal defect; the application is applied to line defect identification.

Description

Intelligent Identification Method and System for Power Distribution Line Defects Based on Visual Multimodal Fusion Technical Field

[0001] This invention provides a method and system for intelligent identification of power distribution line defects based on visual multimodal fusion, belonging to the field of power distribution line defect identification technology. Background Technology

[0002] With the deepening of the State Grid's digital construction, building a clean, digital, and intelligent new power system has become the development direction for the power industry's transformation. Deepening the perception and analysis of the power grid and equipment status is crucial for ensuring the reliability and security of the power system. As the main transmission system connecting power supply equipment and customers, the distribution network plays a vital role in power production and operation. Its safety and reliability directly determine the function and efficiency of power supply and consumption equipment. Currently, distribution network maintenance teams still rely heavily on manual inspections for line patrols. Power workers traverse mountains and valleys, using manual observation or drone imagery to inspect existing power lines. This method suffers from drawbacks such as high workload, long work cycles, unintuitive data, low accuracy, low reuse rates, and difficulty in working in complex terrain.

[0003] In recent years, the State Grid Corporation of China has actively promoted collaborative inspections using helicopters, drones, and manual labor, resulting in a continuous improvement in the operation and maintenance level of power distribution lines. Drone inspections reduce the safety risks associated with personnel climbing towers and are characterized by high efficiency, high quality, and minimal impact from terrain conditions, making them an important tool for the safer, more efficient, and more precise management of power distribution lines. While the number of defects discovered using drones has increased significantly, some hardware defects are still not detected by drones during ground-based manual inspections and tower maintenance, leading to changes in the original maintenance plan, increased workload, and delays. The low efficiency of drone-based defect detection is mainly due to low positioning accuracy, low ambient light, and electromagnetic interference. Summary of the Invention

[0004] To address the problem of low defect detection rate in existing UAV inspections, this invention proposes an intelligent identification method and system for power distribution line defects based on visual multimodal fusion.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an intelligent identification method for power distribution line defects based on visual multimodal fusion, comprising the following steps:

[0006] S1: Use a drone to set up images captured by an infrared camera and a zoom camera at different focal lengths, and record the focal length information of the infrared camera and the zoom camera and the corresponding captured images.

[0007] S2: Solve the affine matrix for each set of infrared and RGB images acquired, and find the affine matrix of the RGB image affine transformation to the infrared image;

[0008] S3: Train the YOLOv5L object detection model on all the acquired RGB images. There are two types of training: one is power transmission line components, and the other is component defects. Deploy the trained YOLOv5L object detection model on the edge device carried by the drone.

[0009] S4: The infrared and visible light video streams captured by the drone are cropped and separated. The visible light video stream is then transmitted to the YOLOv5L target detection model trained on the edge device for detection. Finally, the detected targets are judged. If they are component defects, they are directly marked in the image. If they are component targets, they are mapped to the infrared image through an affine matrix. The highest and lowest temperatures are calculated in the infrared image annotation box to determine whether the above components have thermal defects, and then they are marked in the image.

[0010] In step S1, the two images captured by the infrared camera and the zoom camera are images of the same object size and position captured by the UAV at the same location and altitude of the same component.

[0011] In step S3, the YOLOv5L object detection model inserts a CBAM attention mechanism module after each CSP1_X module in the YOLOv5L base network to transmit important location information from multiple feature maps at different scales to the FPN+PAN network structure.

[0012] The network structure of the backbone part of the YOLOv5L target detection model includes the following modules connected in sequence: a first CBS module, a second CBS module, a first CSP1_3 module, a first CBAM module, a third CBS module, a CSP1_6 module, a second CBAM module, a fourth CBS module, a CSP1_9 module, a third CBAM module, a fifth CBS module, a second CSP1_3 module, a fourth CBAM module, and an SPPF module.

[0013] The component defects include body defects and channel defects. Body defects include joint overheating, hot pins, missing nuts, broken insulators, flashover, bird nests, and floating objects. Channel defects include tree obstructions.

[0014] A power distribution line defect intelligent identification system based on visual multimodal fusion includes a drone and a digital inspection and management platform. The drone is equipped with an infrared camera, a zoom camera, an edge device, and an RTK positioning system. The edge device is deployed with a power distribution line defect intelligent identification method based on visual multimodal fusion. An LED light source is fixed on the drone's body.

[0015] The intelligent digital inspection and control platform integrates existing line databases, power distribution network 3D point cloud databases, waypoint databases, sample databases, and model databases.

[0016] The zoom camera is equipped with a 25mm fixed-focus lens with an aperture of F1.8.

[0017] The LED light source is fixed to the drone body via a remotely controllable base, forming a dynamic light source.

[0018] The beneficial effects of this invention compared to the prior art are as follows: The intelligent identification method and system for power distribution line defects based on visual multimodal fusion provided by this invention improves the detection rate of hardware defects by equipping a drone with a high-precision RTK positioning system, a customized 25mm telephoto lens, and an added LED light source. It can discover more hidden dangers during the inspection of the line itself, keep the line in good condition, and ensure the safe and stable operation of the line.

[0019] After modification, drones can discover more hidden dangers at once during line inspections, avoiding the need for secondary inspections due to "blurred images" or "dark images," thereby reducing electricity costs and the number of vehicle inspections, as well as personnel and vehicle dispatch costs.

[0020] The photos taken by drones during inspections are of higher quality, and a "six-photo" version of power distribution lines can be created. By establishing a drone inspection photo library for the 10kV and above line towers under its jurisdiction, drone photos that meet the requirements can be stored to provide a basis for later operation and maintenance, reducing the number of manual inspections. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings:

[0022] Figure 1 is a flowchart of the method of the present invention;

[0023] Figure 2 is a schematic diagram of the general network structure of the yolov5l model;

[0024] Figure 3 is a schematic diagram of the basic network structure of the improved yolov5l target detection model of the present invention;

[0025] Figure 4 is a flowchart of the infrared image detection using the yolov5l target detection model of the present invention;

[0026] Figure 5 is a schematic diagram of the CBAM attention mechanism module of the present invention;

[0027] Figure 6 is a schematic diagram of the channel attention module;

[0028] Figure 7 is a schematic diagram of the spatial attention module. Detailed Implementation

[0029] As shown in Figures 1 to 7, this invention provides a method for intelligent identification of power distribution line defects based on visual multimodal fusion, comprising the following steps:

[0030] S1: Use a drone to set up images captured by an infrared camera and a zoom camera at different focal lengths, and record the focal length information of the infrared camera and the zoom camera and the corresponding captured images.

[0031] S2: Solve the affine matrix for each set of infrared and RGB images acquired, and find the affine matrix of the RGB image affine transformation to the infrared image;

[0032] S3: Train the YOLOv5L object detection model on all the acquired RGB images. There are two types of training: one is power transmission line components, and the other is component defects. Deploy the trained YOLOv5L object detection model on the edge device carried by the drone.

[0033] S4: The infrared and visible light video streams captured by the drone are cropped and separated. The visible light video stream is then transmitted to the YOLOv5L target detection model trained on the edge device for detection. Finally, the detected targets are judged. If they are component defects, they are directly marked in the image. If they are component targets, they are mapped to the infrared image through an affine matrix. The highest and lowest temperatures are calculated in the infrared image annotation box to determine whether the above components have thermal defects, and then they are marked in the image.

[0034] In step S1, the two images captured by the infrared camera and the zoom camera are images of the same object size and position captured by the UAV at the same location and altitude of the same component.

[0035] In step S3, the YOLOv5L object detection model inserts a CBAM attention mechanism module after each CSP1_X module in the YOLOv5L base network to transmit important location information from multiple feature maps at different scales to the FPN+PAN network structure.

[0036] The network structure of the backbone part of the YOLOv5L target detection model includes the following modules connected in sequence: a first CBS module, a second CBS module, a first CSP1_3 module, a first CBAM module, a third CBS module, a CSP1_6 module, a second CBAM module, a fourth CBS module, a CSP1_9 module, a third CBAM module, a fifth CBS module, a second CSP1_3 module, a fourth CBAM module, and an SPPF module.

[0037] The component defects include body defects and channel defects. Body defects include joint overheating, hot pins, missing nuts, broken insulators, flashover, bird nests, and floating objects. Channel defects include tree obstructions.

[0038] A power distribution line defect intelligent identification system based on visual multimodal fusion includes a drone and a digital inspection and management platform. The drone is equipped with an infrared camera, a zoom camera, an edge device, and an RTK positioning system. The edge device is deployed with a power distribution line defect intelligent identification method based on visual multimodal fusion. An LED light source is fixed on the drone's body.

[0039] The intelligent digital inspection and control platform integrates existing line databases, power distribution network 3D point cloud databases, waypoint databases, sample databases, and model databases.

[0040] The zoom camera is equipped with a 25mm fixed-focus lens with an aperture of F1.8.

[0041] The LED light source is fixed to the drone body via a remotely controllable base, forming a dynamic light source.

[0042] The present invention proposes an intelligent identification method for power distribution line defects based on visual multimodal fusion, the main steps of which are as follows:

[0043] 1. First, use the drone to set up the infrared camera and zoom camera to capture images at several different focal lengths (the objects in the two images should be roughly the same size and position), and record the focal length information of the infrared camera and zoom camera and the corresponding images they captured.

[0044] 2. Solve the affine matrix for each set of infrared and RGB images acquired, and obtain the affine matrix of the RGB image affine transformation to the infrared image;

[0045] 3. Train the YOLOv5 object detection model on all the acquired RGB images. There are two main types of training: one is power transmission line components, and the other is component defects. Deploy the trained model to the edge device carried by the drone.

[0046] 4. The infrared and visible light video streams captured by the drone are cropped and separated. The visible light video stream is then transmitted to the target detection model trained in the edge device for detection. Finally, the detected targets are judged. If they are component defects (bird nests, floating objects, broken porcelain bottles, etc.), they are directly marked in the image. If they are component targets, they are mapped to the infrared image through an affine matrix. The highest and lowest temperatures are calculated in the infrared image annotation box to determine whether the component has thermal defects, and then it is marked in the image.

[0047] This invention employs an improved YOLOv5l target detection model to detect defects in visible light images. The general YOLOv5l target detection model mainly consists of three parts: Backbone, Neck, and Prediction. Its network structure is shown in Figure 2. The network structure mainly consists of CBS, CSP1_X, SPPF, Res unit, and CSP2_X modules, where Conv is the convolution operation, BN is the normalization operation, SiLU is the activation function used, add refers to adding the values ​​corresponding to the feature maps of the same size, Concat refers to concatenating the feature maps of the same size, MaxPool refers to max pooling, and Unsample refers to upsampling.

[0048] Some small components and defects on power distribution lines are small and their defect features are difficult to extract, which can easily lead to missed detections. To address this issue, CBAM (Convolutional Block Attention Module) was inserted into the YOLOv5l base network to extract more rich feature information. Its structure is shown in Figure 5.

[0049] CBAM contains two sub-modules: the Channel Attention Module (CAM) and the Spatial Attention Module (SAM). As shown in Figures 6 and 7, the Channel Attention Module allows the network model to selectively focus on important channels during training and inference. The Spatial Attention Module weights the locations containing important features in the feature map, further improving the model's ability to detect target locations.

[0050] As shown in Figure 3, the improved YOLOv5L target detection model of this invention inserts a CBAM attention mechanism module after each CSP1_X module in the YOLOv5L basic network. Because the feature map output by each CSP1_X module is of different sizes, after inserting CBAM, the important positional information of multiple feature maps of different scales will be transmitted to the FPN+PAN network structure, which can further improve the detection accuracy of power distribution line components and their defects.

[0051] Due to the diverse types of defects in power distribution network equipment, a single sensing device cannot meet application requirements. This invention addresses the defect types of key components of 10 kV power distribution network inspections by completing multimodal data acquisition and annotation based on "visible light + infrared". Based on the annotated trainable dataset, a defect identification model for transmission lines in 10 kV power distribution networks was trained. Tests show that the real-time identification accuracy on mobile devices can reach 95% for defects such as joint overheating, missing pins, missing nuts, insulator damage, flashover, bird nests, floating objects, and channel defects such as tree obstructions.

[0052] This invention aims to detect the temperature of targets in infrared images. However, it is difficult to detect targets in infrared images directly using the YOLOv5 target detection model. Therefore, a method for detecting targets in infrared images based on visible light images is proposed. The main process is as follows: (1) By calibrating the RGB camera and the infrared camera, the matrix mapping the RGB image to the infrared image is obtained; (2) The visible light image is input into the YOLOv5 target detection model to obtain the detection result; (3) The detection result is mapped onto the infrared image through the mapping matrix to determine the specific location of the target in the infrared image. The detection schematic diagram is shown in Figure 4.

[0053] Once the target and its location are known in the infrared image, thermal defect diagnosis can be performed. This invention can detect both infrared and visible light defects, and the detection results are shown in Table 1 below:

[0054]

[0055] Table 1. Defect detection results.

[0056] This invention also proposes an intelligent power distribution line defect identification system based on visual multimodal fusion. It primarily addresses the problems of low positioning accuracy, low ambient brightness, and blurred or dark images of the acquired line targets due to electromagnetic interference in existing drone-based power distribution line defect detection. The system improves upon existing drones by adding a high-precision RTK positioning system, achieving accurate positioning within a 10cm range, and providing high efficiency and defect detection rate for single-tower inspections. An LED light source is added to the drone's fuselage for supplemental lighting, directly attached to the landing gear without damaging the original internal electrical structure or auxiliary equipment. This provides aerial illumination along with the drone, significantly improving the brightness of the photographed object, while also offering low power consumption, low cost, and easy installation. The LED light source can be mounted on a remotely controllable base to create a dynamic light source. A custom-designed 25mm telephoto lens significantly improves image quality; even at 6x magnification, the pole components are still clearly visible, with excellent imaging effects and significantly improved image resolution. This allows for long-distance imaging of components without approaching live wires, avoiding electromagnetic interference.

[0057] The intelligent identification system for distribution line defects based on visual multimodal fusion of the present invention also includes a digital intelligent inspection and control platform set up for ground monitoring. This platform integrates existing basic data such as the line database, and combines data such as the distribution network 3D point cloud database, flight point database, sample database, and model database to build the first digital warehouse of distribution network lines. It has core functions such as intelligent inspection equipment management, distribution network equipment management, flight route management, task management, remote control, and channel services.

[0058] The intelligent identification system for power distribution line defects based on visual multimodal fusion proposed in this invention forms an inspection process that includes 3D path planning, autonomous flight, intelligent focusing, "infrared + visible light" intelligent identification, real-time information transmission, and report generation. In specific inspection tasks, front-line inspection personnel only need to issue the inspection task to the drone, which will then autonomously complete the inspection task without any human intervention.

[0059] Regarding the specific structure of this invention, it should be noted that the connection relationships between the various component modules used in this invention are definite and achievable. Except as specifically described in the embodiments, their specific connection relationships can bring about corresponding technical effects and solve the technical problems proposed by this invention without relying on the execution of corresponding software programs. The models of the components, modules, and specific components appearing in this invention, the connection methods between them, and the conventional usage methods and expected technical effects brought about by the above technical features, unless specifically described, are all publicly disclosed content in patents, journal articles, technical manuals, technical dictionaries, and textbooks that can be obtained by those skilled in the art before the application date, or belong to conventional technology, common knowledge, and other existing technologies in this field. There is no need to elaborate, which makes the technical solution provided in this case clear, complete, and achievable, and can reproduce or obtain corresponding physical products based on this technical means.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent identification of power distribution line defects based on visual multimodal fusion, characterized in that: The steps include: S1: Using the drone, set up the infrared camera and zoom camera to capture images at different focal lengths, and record the focal length information of the infrared camera and zoom camera and the corresponding captured images; S2: Solve the affine matrix for each set of infrared and RGB images acquired, and find the affine matrix of the RGB image affine transformation to the infrared image; S3: Train the YOLOv5L object detection model on all acquired RGB images. There are two training types: power line components and component defects. Deploy the trained YOLOv5L object detection model to the edge device mounted on the drone. The YOLOv5L object detection model inserts a CBAM attention mechanism module after each CSP1_X module in the YOLOv5L base network, transmitting important location information from multiple feature maps at different scales to the FPN+PAN network structure. The backbone network structure of the YOLOv5L object detection model includes a first CBS module, a second CBS module, a first CSP1_3 module, and a first CBAM module connected in sequence. The module consists of the following modules: the third CBS module, the CSP1_6 module, the second CBAM module, the fourth CBS module, the CSP1_9 module, the third CBAM module, the fifth CBS module, the second CSP1_3 module, the fourth CBAM module, and the SPPF module. S4: The video stream captured by the UAV, which is a stitched image of infrared and visible light, is cropped and separated. Then, the visible light video stream is transmitted to the YOLOv5L target detection model trained in the edge device for detection. Finally, the detected target is judged. If it is a component defect, it is directly labeled in the image. If it is a component target, it is mapped to the infrared image through an affine matrix. The highest and lowest temperatures are calculated in the infrared image annotation box to determine whether the component has thermal defects, and then it is labeled in the image.

2. The intelligent identification method for power distribution line defects based on visual multimodal fusion according to claim 1, characterized in that: In step S1, the two images captured by the infrared camera and the zoom camera are images of the same object size and position captured by the UAV at the same location and altitude of the same component.

3. The intelligent identification method for power distribution line defects based on visual multimodal fusion according to claim 1, characterized in that: The component defects include body defects and channel defects. Body defects include joint overheating, hot pins, missing nuts, broken insulators, flashover, bird nests, and floating objects. Channel defects include tree obstructions.

4. A power distribution line defect intelligent identification system based on visual multimodal fusion, characterized in that: The system includes a drone and a digital inspection and control platform. The drone is equipped with an infrared camera and a zoom camera, an edge device, and an RTK positioning system. The edge device is equipped with the intelligent identification method for power distribution line defects based on visual multimodal fusion as described in any one of claims 1-3. The drone is equipped with an LED light source.

5. The intelligent identification system for power distribution line defects based on visual multimodal fusion according to claim 4, characterized in that: The intelligent digital inspection and control platform integrates existing line databases, power distribution network 3D point cloud databases, waypoint databases, sample databases, and model databases.

6. The intelligent identification system for power distribution line defects based on visual multimodal fusion according to claim 4, characterized in that: The zoom camera is equipped with a 25mm fixed-focus lens with an aperture of F1.

8.

7. The intelligent identification system for power distribution line defects based on visual multimodal fusion according to claim 4, characterized in that: The LED light source is fixed to the drone body via a remotely controllable base, forming a dynamic light source.

Images