Automatic identification method and system for power transmission and transformation equipment based on visible light and infrared light fusion

Abstract

The invention discloses an automatic identification method and system for power transmission and transformation equipment based on visible light and infrared dual-light fusion. The method comprises the following steps that a visible light image and an infrared image of the same scene are synchronously collected through a dual-light camera carried by an unmanned aerial vehicle; preprocessing the visible light image and the infrared image, including denoising, geometric correction and spatial alignment, in the spatial alignment, feature points are extracted through an SURF algorithm, and image registration is optimized through an RANSAC algorithm; and feature extraction is carried out on the fused image by using a deep convolutional neural network OfficientNet-Lite, and a multi-scale feature map is generated. The method has the beneficial effects that high-precision identification is realized; a dynamic fusion strategy and an attention mechanism enable the insulator crack (gt; 0.1 mm), the detection rate reaches 92%; the anti-interference capability is high, the false alarm rate is reduced from 15% to 4% through a dust compensation algorithm, and the influence of emissivity errors is reduced through material self-adaptive calibration; the whole process from data acquisition to report generation is free of human intervention, and the labor cost is reduced by 70%.

Description

Technical Field

[0001] The present invention relates to the field of automatic identification technology, and in particular to a method and system for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light. Background Art

[0002] The safe and stable operation of power transmission and transformation equipment is an important guarantee for the reliability of the power system. Traditional inspections mainly rely on manual visual inspection or single sensors (such as visible light cameras or infrared thermal imagers) to monitor equipment status, which has the following limitations: Single detection method and poor environmental adaptability: Visible light imaging is easily affected by environmental factors such as lighting changes, rain and fog, and it is difficult to clearly capture equipment details (such as insulator cracks) in low illumination or complex background scenes.

[0003] Although infrared thermal imaging can reflect abnormal equipment temperature, its spatial resolution is low and it cannot accurately locate tiny defects (such as rust on wire clamps). It is also easily affected by environmental temperature, humidity, and dust, resulting in large temperature measurement errors.

[0004] Insufficient image fusion technology: Existing dual-light fusion methods mostly use fixed weights (such as linear superposition) or simple feature splicing, which leads to loss of fused image details and noise amplification (see reference 1: CN112950674A).

[0005] There is a lack of targeted enhancement of key areas of the equipment. For example, when the contrast between the cracks on the surface of the insulator and the background is low, traditional algorithms find it difficult to effectively improve target visibility.

[0006] Defect recognition accuracy and efficiency are limited: Traditional rule-based image processing algorithms (such as threshold segmentation and morphological operations) have poor generalization capabilities for complex device morphologies, with a false detection rate as high as 15%-20% (see Reference 2: CN113256636A).

[0007] Existing deep learning models (such as YOLOv3 and Faster R-CNN) have high computational complexity and are difficult to implement real-time detection (frame rate <10fps) on drone embedded platforms.

[0008] Inaccurate temperature calibration and status assessment: Infrared temperature measurement often ignores the reflectivity differences of the device surface material (such as the different emissivity of metal and ceramic), resulting in temperature measurement errors (Reference 3: CN114166118A).

[0009] Existing methods do not effectively fuse multi-source data (such as ultrasonic and short-wave infrared) and are unable to detect hidden defects such as internal discharge or oxide layer degradation.

[0010] To address these issues, recent research has attempted to combine visible light and infrared data (e.g., Reference 4: CN115294252A proposes a wavelet transform-based fusion method). However, these methods offer limited dynamic adaptability and fail to address the challenges of lightweight deployment and multi-sensor collaborative calibration. Therefore, a method and system for automatic identification of power transmission and distribution equipment based on the fusion of visible light and infrared is urgently needed to address inspection needs in complex environments. Summary of the Invention

[0011] The main technical problem solved by the present invention is to provide a method and system for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared dual light, so as to solve one or more of the above-mentioned existing technical problems.

[0012] In order to solve the above technical problems, the present invention adopts a technical solution: a method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared dual light, characterized by: The following steps are involved: (1) Using a dual-light camera mounted on a UAV to simultaneously capture visible light and infrared images of the same scene; (2) preprocessing the visible light image and the infrared image, including denoising, geometric correction and spatial alignment, wherein the spatial alignment extracts feature points using a SURF algorithm and optimizes image registration using a RANSAC algorithm; (3) Construct a dual-light fusion model and perform weighted fusion of the registered visible light image and infrared image. The fusion formula is: Pixel fused =k Pixel visible +(1−k) Pixel infrared Where k is a dynamic weight factor, which is adaptively adjusted based on the local contrast and edge strength of the image; (4) Use the deep convolutional neural network EfficientNet-Lite to extract features from the fused image and generate multi-scale feature maps; (5) Target detection is performed on the multi-scale feature map through a feature recognition network to identify the type and location of power transmission and transformation equipment (including insulators, wire clamps, and switches), and output the equipment bounding box and category label; (6) Associate the recognition results with the temperature data in the infrared image, calibrate the temperature value based on the equipment material and environmental parameters and short-wave infrared (SWIR) sensor data, and generate an equipment status report.

[0013] T calibrated =T measured +ΔT env ·(1−e −α·Humidity )·f(Material) Among them, f(Material) is the reflectivity compensation function based on the material category.

[0014] In some embodiments, the step (3) subsequently performs semantic segmentation on the visible light image through the UNet network to extract the key areas of the power transmission and transformation equipment, and dynamically adjusts the fusion weight factor k according to the segmentation results; the step (4) subsequently embeds the CBAM attention mechanism module in the feature extraction layer to enhance the detection sensitivity of small defects; the step (6) subsequently analyzes the surface material of the equipment through the visible light image and dynamically adjusts the emissivity parameter ε. The calibration formula is expanded to: T calibrated =T measured +ΔT env ·(1−e −α·Humidity )·f(Material) Among them, f(Material) is the reflectivity compensation function based on the material category.

[0015] In some embodiments, the geometric correction in step (2) includes: Perform distortion correction on visible light images and use Zhang Zhengyou calibration method to obtain the camera intrinsic parameter matrix; Nonlinear temperature mapping is performed on infrared images, and the grayscale value is converted into absolute temperature through the blackbody radiation law.

[0016] In some embodiments, the dynamic weight factor kk in step (3) is calculated as follows: k=Edge visible +λ⋅Texturevisible / ( Edge visible +Edge infrared +λ⋅Texturevisible) Among them, Edge is the edge strength extracted by the Canny-Hough joint algorithm, Texture is the local texture complexity, and λ is the balance coefficient.

[0017] In some embodiments, the deep convolutional neural network MobileNetV3 network in step (4) includes: 16 layers of depth-wise separable convolutional layers, each followed by an h-swish activation function; The three feature recognition layers perform target detection on feature maps with resolutions of 13×13, 26×26, and 52×52, respectively. Three anchor boxes are preset for each feature layer.

[0018] In some embodiments, the temperature calibration method in step (6) further comprises: (6b) Multi-angle infrared data is collected through a multi-UAV collaborative inspection system, and SLAM technology is used to reconstruct the three-dimensional temperature field of the equipment; (6c) Ultrasonic detection data and temperature data are integrated to construct a multi-dimensional fault diagnosis model to determine internal discharge defects.

[0019] An automatic identification system for power transmission and transformation equipment based on the fusion of visible light and infrared light, including: Dual-light image acquisition module, used to synchronously acquire visible light images and infrared images; Image preprocessing module, which performs geometric correction, spatial alignment and noise filtering; Dual-light fusion module, which fuses visible light and infrared images based on a dynamic weight allocation strategy; A lightweight deep learning recognition module that uses the EfficientNet-Lite network to extract features and identify device types and locations; The temperature calibration module combines environmental parameters, material analysis results and SWIR sensor data to correct infrared temperature measurement data; the multi-machine collaborative control module realizes multi-UAV task allocation and data synchronization based on 5G communication; the result output module generates equipment status reports and visual 3D thermal maps and defect location models.

[0020] In some embodiments, a dual-light image acquisition module is integrated into a UAV platform, including: Visible light camera: resolution not less than 20 million pixels, frame rate ≥ 30fps; Infrared thermal imager: temperature measurement range -20℃~+350℃, thermal sensitivity ≤50mK; Shortwave infrared (SWIR) sensor: spectral range 900-1700nm, resolution 640×512; RTK positioning unit, accuracy ±1cm, and integrated SLAM algorithm to enhance positioning stability in complex terrain.

[0021] In some embodiments, the result output module further includes: The human-computer interaction interface supports manual labeling of falsely detected / missed targets and dynamically optimizes the model through online learning; the LSTM time series analysis unit predicts trends in historical temperature data and generates early warning signals.

[0022] In some embodiments, the drone is equipped with an ultrasonic sensor to collect discharge signals inside the equipment and fuse them with temperature data to determine the defect type.

[0023] The beneficial effects of the present invention are: high-precision identification: the dynamic fusion strategy and attention mechanism enable the detection rate of insulator cracks (>0.1mm) to reach 92%.

[0024] Strong anti-interference capability: The dust compensation algorithm reduces the false alarm rate from 15% to 4%, and the material adaptive calibration reduces the impact of emissivity errors.

[0025] Fully automated operation and maintenance: No human intervention from data collection to report generation, reducing labor costs by 70%. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which: Figure 1 This is a system overall architecture block diagram of an automatic identification method and system for power transmission and transformation equipment based on the fusion of visible light and infrared dual light of the present invention.

[0027] Figure 2 This is an image acquisition and preprocessing flow chart of an automatic identification method and system for power transmission and transformation equipment based on the fusion of visible light and infrared dual light of the present invention.

[0028] Figure 3 This is a schematic diagram of a dual-light fusion algorithm of an automatic identification method and system for power transmission and transformation equipment based on the fusion of visible light and infrared light in the present invention.

[0029] Figure 4 This is a structural diagram of a deep learning model of a method and system for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light in the present invention.

[0030] Figure 5 The present invention discloses a method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light, and a temperature calibration and multi-sensor fusion block diagram of the system.

[0031] Figure 6 This is a schematic diagram of a multi-UAV collaborative inspection method and system for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light of the present invention.

[0032] Figure 7 This is a visualization and interactive interface design diagram of an automatic identification method and system for power transmission and transformation equipment based on the fusion of visible light and infrared light in the present invention.

[0033] Figure 8 This is a block diagram of a timing analysis and early warning system for an automatic identification method and system of power transmission and transformation equipment based on the fusion of visible light and infrared dual light of the present invention. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] like Figures 1 to 8 As shown, the embodiment of the present invention includes: 1. Image Acquisition and Preprocessing (1) Bi-optical image acquisition: Drone platform: A DJI Matrice 300 RTK drone was used, equipped with a visible light camera (Sony IMX586, 48MP, 30fps), an infrared thermal imager (FLIR A8582, 640×512 resolution, temperature measurement range -20°C to +350°C, thermal sensitivity 50mK), and a shortwave infrared (SWIR) sensor (Xenics Gobi 640, spectral range 900–1700nm).

[0036] Synchronous acquisition: Hardware trigger signals ensure time synchronization between visible light and infrared images (error <1ms), and the RTK positioning unit (accuracy ±1cm) records the geographic coordinates of each frame.

[0037] (2) Preprocessing process: Visible light image distortion correction: Zhang Zhengyou calibration method is used, using a 10×7 checkerboard calibration plate to calculate the camera intrinsic parameter matrix (focal length f x =3200,f y =3200, main point c x =2400, c y =1350) and distortion coefficients (k1=−0.12, k2=0.03).

[0038] Infrared image temperature mapping: Based on the blackbody radiation formula T=c2 / λ⋅ln(c1 / λ 5 ⋅G+1), convert the grayscale value G into absolute temperature.

[0039] Spatial alignment: The SURF algorithm was used to extract feature points (≥500 feature points were extracted from a single image), and the RANSAC algorithm was used to optimize the registration (1000 iterations, with an inlier threshold of 2 pixels). The final registration error was ≤1.5 pixels.

[0040] 2. Dual-light fusion and target recognition (3) Dual-light fusion model: Dynamic weight factor calculation: Edge strength: Use Canny edge detection (high and low threshold ratio 3:1) combined with Hough transform to extract straight line edges and calculate local edge strength Edge visible and Edge infrared .

[0041] Semantic segmentation: Using the UNet network (4-layer encoder, 64 / 128 / 256 / 512 channels per layer), the visible light image is segmented into key areas such as insulators and wire clamps, and the probability map S is output. mask .

[0042] Weight formula optimization: k=(Edge visible +0.5⋅S mask ) / (Edge visible +Edge infrared +0.5⋅S mask ) (Balance coefficient λ=0.5, giving priority to enhancing details in key areas of the device).

[0043] (4) Lightweight deep learning model: Network structure: EfficientNet-Lite (depth scaling factor 1.2, width scaling factor 1.1) is used, embedded in the CBAM attention module (channel attention + spatial attention), and outputs multi-scale feature maps (52×52 / 26×26 / 13×13).

[0044] Training configuration: Dataset: 100,000 images of power transmission and transformation equipment (visible light + infrared), annotated with five types of objects, including insulators and wire clamps.

[0045] Optimizer: Adam (learning rate 1e-4, weight decay 5e-4), trained for 100 epochs, mAP reached 94.3%.

[0046] (5) Target detection and positioning: Anchor frame design: Based on the insulator aspect ratio (2:1~5:1), three sets of anchor frames (16×8, 32×16, 64×32) are preset.

[0047] Prediction formula: b x =σ(t x )+c x ,by=σ(t y )+c y ,b w =p w ⋅e tw ,b h =p h ⋅e th (grid coordinates (cx,cy), anchor box size (pw,ph), predicted offset (tx,ty,tw,th)).

[0048] 3. Temperature calibration and condition assessment (6) Temperature calibration method: Material reflectivity compensation: Through HSV color space analysis of visible light images, metals (H<30, S>0.5) and ceramics (H>150, S<0.3) are distinguished and the emissivity ε is set. 金属 =0.85, ε 陶瓷 =0.93.

[0049] The calibration formula expands to: T calibrated =T measured +ΔT env ⋅(1−e −0.2⋅Humidity )⋅ε Material Multi-sensor data fusion: SWIR sensors detect the oxide layer on the surface of equipment (wavelength 1200-1400nm) to assist in determining the contamination level of insulators.

[0050] The ultrasonic sensor (frequency 40kHz) detects internal discharge signals. If the sound pressure level is >60dB and the temperature is >80°C, it is determined to be an internal discharge defect.

[0051] (7) Multi-machine collaboration and 3D reconstruction: Task allocation: Three drones are assigned to patrol areas based on the Hungarian algorithm, with 5G communication latency less than 10ms.

[0052] SLAM 3D reconstruction: Using the ORB-SLAM3 algorithm, combined with RTK and IMU data, a 3D point cloud model of the device is generated (accuracy ±3cm), and temperature field data is superimposed to generate a 3D thermal map.

[0053] 4. Result output and interaction optimization (8) Visualization and manual intervention: Heat map generation: Use the Matplotlib library to map temperature data to a rainbow color scale, with abnormal areas (temperature difference > 10°C) highlighted in red.

[0054] Human-computer interaction interface: Developed based on PyQt, it supports manual selection of misdetected targets. The model updates weights within 10 minutes through online learning (learning rate 1e-5).

[0055] (9) Time series analysis and early warning: The LSTM network (hidden layer with 128 units) analyzes the device's historical temperature (sampling interval is 1 hour) and predicts the temperature trend for the next 24 hours. If the slope is greater than 2°C / h, an alert is triggered.

[0056] Working principle of this technical solution Dual-light fusion enhances target visibility: By fusing visible light texture and infrared thermal features through dynamic weighting, combined with semantic segmentation to focus on key areas, it solves the problem of detail loss in traditional fusion algorithms.

[0057] Lightweight models improve real-time performance: EfficientNet-Lite and MobileNetV3 networks maintain 94% accuracy while achieving an inference speed of 45fps (NVIDIA Jetson AGX Xavier platform), meeting the real-time processing requirements of drones.

[0058] Precise multimodal data calibration: By integrating SWIR, ultrasonic, and environmental parameters, the temperature measurement error is reduced from ±5°C to ±1.2°C, and the defect detection rate is increased to 97.6%.

[0059] Intelligent collaboration and 3D visualization: Multiple drones collaborate over 5G to expand the inspection range (covering a 5km radius), while SLAM 3D reconstruction assists with precise positioning, tripling operation and maintenance efficiency.

[0060] The advantages of this technical solution are: High-precision identification: Dynamic fusion strategy and attention mechanism achieve a 92% detection rate for insulator cracks (>0.1mm).

[0061] Strong anti-interference capability: The dust compensation algorithm reduces the false alarm rate from 15% to 4%, and the material adaptive calibration reduces the impact of emissivity errors.

[0062] Fully automated operation and maintenance: No human intervention from data collection to report generation, reducing labor costs by 70%.

[0063] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light, characterized by: The following steps are involved: (1) Using a dual-light camera mounted on a UAV to simultaneously capture visible light and infrared images of the same scene; (2) preprocessing the visible light image and the infrared image, including denoising, geometric correction and spatial alignment, wherein the spatial alignment extracts feature points using a SURF algorithm and optimizes image registration using a RANSAC algorithm; (3) Construct a dual-light fusion model and perform weighted fusion of the registered visible light image and infrared image. The fusion formula is: Pixel fused =k·Pixel visible +(1−k)·Pixel infrared Where k is a dynamic weight factor, which is adaptively adjusted based on the local contrast and edge strength of the image; (4) Use the deep convolutional neural network EfficientNet-Lite to extract features from the fused image and generate multi-scale feature maps; (5) Target detection is performed on the multi-scale feature map through a feature recognition network to identify the type and location of the power transmission and transformation equipment, and output the equipment bounding box and category label; (6) Associate the recognition results with the temperature data in the infrared image, calibrate the temperature value based on the equipment material and environmental parameters and short-wave infrared sensor data, and generate an equipment status report.

2. The method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 1, characterized in that: In step (3), the semantic segmentation of the visible light image is subsequently performed through the UNet network to extract the key areas of the power transmission and transformation equipment, and the fusion weight factor k is dynamically adjusted according to the segmentation results; in step (4), the CBAM attention mechanism module is subsequently embedded in the feature extraction layer to enhance the detection sensitivity of small defects; in step (6), the surface material of the equipment is subsequently analyzed through the visible light image, and the emissivity parameter ε is dynamically adjusted. The calibration formula is expanded to: T calibrated =T measured +ΔT env ·(1−e −α·Humidity)·f(Material) Among them, f(Material) is the reflectivity compensation function based on the material category.

3. The method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 1 is characterized in that: The geometric correction in step (2) includes: Perform distortion correction on visible light images and use Zhang Zhengyou calibration method to obtain the camera intrinsic parameter matrix; Nonlinear temperature mapping is performed on infrared images, and the grayscale value is converted into absolute temperature through the blackbody radiation law.

4. The method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 1 is characterized in that: The calculation method of the dynamic weight factor kk in step (3) is: k=Edge visible +λ⋅Texturevisible / ( Edge visible +Edge infrared +λ⋅Texturevisible) Among them, Edge is the edge strength extracted by the Canny-Hough joint algorithm, Texture is the local texture complexity, and λ is the balance coefficient.

5. The method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 1 is characterized in that: The deep convolutional neural network MobileNetV3 network in step (4) includes: 16 layers of depth-wise separable convolutional layers, each followed by an h-swish activation function; The three feature recognition layers perform target detection on feature maps with resolutions of 13×13, 26×26, and 52×52, respectively. Three anchor frames are preset in each feature layer.

6. The method for automatic identification of power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 1, characterized in that: The temperature calibration method in step (6) further includes: (6b) Multi-angle infrared data is collected through a multi-UAV collaborative inspection system, and SLAM technology is used to reconstruct the three-dimensional temperature field of the equipment; (6c) Ultrasonic detection data and temperature data are integrated to construct a multi-dimensional fault diagnosis model to determine internal discharge defects.

7. An automatic identification system for power transmission and transformation equipment based on the fusion of visible light and infrared light, characterized in that: include: Dual-light image acquisition module, used to synchronously acquire visible light images and infrared images; Image preprocessing module, which performs geometric correction, spatial alignment and noise filtering; Dual-light fusion module, which fuses visible light and infrared images based on a dynamic weight allocation strategy; A lightweight deep learning recognition module that uses the EfficientNet-Lite network to extract features and identify device types and locations; Temperature calibration module, which combines environmental parameters, material analysis results and SWIR sensor data to correct infrared temperature measurement data; The multi-machine collaborative control module realizes multi-UAV task allocation and data synchronization based on 5G communication; the result output module generates equipment status reports and visual 3D heat maps and defect location models.

8. The automatic identification system for power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 7 is characterized in that: The dual-light image acquisition module is integrated into the UAV platform and includes: Visible light camera: resolution not less than 20 million pixels, frame rate ≥ 30fps; Infrared thermal imager: temperature measurement range -20℃~+350℃, thermal sensitivity ≤50mK; Shortwave infrared sensor: spectral range 900-1700nm, resolution 640×512; RTK positioning unit, accuracy ±1cm, and integrated SLAM algorithm to enhance positioning stability in complex terrain.

9. The automatic identification system for power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 7 is characterized in that: The result output module also includes: a human-computer interaction interface that supports manual labeling of false detection / missed detection targets and dynamically optimizes the model through online learning; an LSTM time series analysis unit that predicts trends in historical temperature data and generates early warning signals.

10. The automatic identification system for power transmission and transformation equipment based on the fusion of visible light and infrared light according to claim 7, characterized in that: The drone is equipped with an ultrasonic sensor to collect discharge signals inside the equipment and fuse them with temperature data to determine the type of defect.

Images