A power equipment inspection method based on deep learning dual-light feature fusion

Abstract

This invention discloses a power equipment inspection method based on deep learning dual-light feature fusion, comprising: simultaneously acquiring visible light images and infrared thermal imaging images of the power equipment using a binocular acquisition module mounted on an inspection robot; extracting significant straight line features from the two images using a gradient-based edge detection operator, and solving the homography matrix using line segment descriptors and geometric constraints to achieve online spatiotemporal registration without a calibration board; constructing a deep learning-based dual-stream feature extraction network to map the registered infrared image as a spatial attention mask, and performing weighted fusion of the visible light feature map using Hadamard product; finally, outputting equipment category, location, and temperature information based on the fused feature map, and determining faults using a dual threshold of confidence and temperature. This invention effectively solves the problems of texture loss in traditional infrared detection and the failure of visible light nighttime detection, achieving accurate location and diagnosis of power equipment faults.

Description

Technical Field

[0001] This invention relates to the field of intelligent power inspection technology, and more specifically, to a power equipment inspection method based on deep learning dual-light feature fusion. Background Technology

[0002] With the deepening of smart grid construction, the intelligent and automated inspection of power equipment has become crucial to ensuring the safe and stable operation of the power grid. Traditional manual inspection methods suffer from drawbacks such as low efficiency, high risk, and strong subjectivity. Therefore, automated platforms such as inspection robots and drones equipped with various sensors are gradually becoming the mainstream inspection methods. Among numerous sensing technologies, visual inspection based on visible light imaging and temperature detection based on infrared thermal imaging are two core and complementary technical approaches.

[0003] Visible light cameras provide high-resolution, textured images, suitable for equipment appearance identification, component location, and detection of some surface defects (such as damage, dirt, and displacement). However, their performance is heavily dependent on ambient lighting conditions; their detection capability drops sharply in complex lighting environments such as nighttime, foggy days, or strong backlighting, and they cannot detect potential overheating hazards inside equipment. Infrared thermal imagers, on the other hand, infer the temperature distribution of objects by receiving their infrared radiation. They can directly detect localized overheating caused by poor contact, overload, or internal faults, and have the advantages of penetrating smoke and fog and being unaffected by light. However, infrared images typically have low spatial resolution and lack texture details, making it difficult to accurately distinguish and locate specific faulty components. They are also susceptible to interference from environmental heat sources (such as solar reflection and ground radiation), leading to false alarms.

[0004] To combine the advantages of visible light and infrared images and improve the overall performance of power equipment fault diagnosis, various image fusion and target detection schemes have been proposed in existing technologies. For example, patent document CN116681979A discloses a method for target detection by fusing edge detection, feature point matching, and deep learning networks; patent document CN117557775A proposes a detection network that utilizes a cross-attention structure to achieve deep feature fusion of infrared and visible light images. These schemes have achieved certain results in specific scenarios.

[0005] However, after in-depth analysis and practice, the existing technology still has the following prominent problems and limitations: 1. Shallow fusion layer and insufficient information complementarity: Many methods remain at the pixel-level weighted averaging or simple feature channel stitching level. This shallow fusion layer is prone to introducing noise from infrared images (such as hotspot noise and non-uniformity) into the fusion result, resulting in blurred edges and loss of details, failing to achieve effective alignment and complementary advantages of the two modalities in deep semantics.

[0006] 2. Poor registration robustness and dependence on specific conditions: Accurate image registration is a prerequisite for multimodal fusion. Existing methods mostly rely on extracting point features (such as SIFT and SURF) for matching. However, in power line inspection scenarios, point features such as corner points and spots in infrared images are often unclear and unstable, especially in areas with weak texture. In addition, many solutions rely on pre-deployed physical calibration boards for calibration. Once the inspection platform (such as robots or drones) experiences a slight change in camera posture or vibration during operation, the pre-calibrated parameters will become invalid, making it difficult to meet the requirements of online adaptive registration in field, long-distance, and dynamic operations, resulting in insufficient robustness.

[0007] 3. The fault determination logic is simplistic, leading to a high false alarm rate: Most existing methods rely solely on visual inspection confidence levels or a single temperature threshold for fault determination. This makes the system prone to false alarms: for example, misclassifying a non-equipment high-temperature area under sunlight as a fault (infrared single-mode defect), or misidentifying other objects resembling electrical equipment as faulty equipment (visible light single-mode defect). There is a lack of a joint determination mechanism that can cross-validate both "visual appearance" and "thermal characteristics."

[0008] 4. High algorithm complexity and significant real-time challenges: Some solutions that use complex deep learning models for feature extraction and fusion may improve accuracy, but the large number of model parameters and computational complexity make them difficult to deploy on edge devices with limited computing power and power consumption (such as mobile robots and drone payloads), affecting the real-time performance and versatility of inspection.

[0009] In summary, existing power equipment inspection technologies still face a series of challenges in achieving high-precision, robust, and low-false-alarm-rate visible light and infrared multimodal fusion detection. These challenges include inefficient fusion methods, stringent registration conditions, simplistic decision logic, and difficulties in real-time deployment. Therefore, there is an urgent need for an intelligent power equipment inspection method that can achieve online adaptive high-precision registration, deep semantic feature adaptive fusion, dual visual and temperature verification, and balance detection accuracy with edge computing efficiency. Summary of the Invention

[0010] In view of the above-mentioned technical problems in related technologies, this invention proposes a power equipment inspection method based on deep learning dual-light feature fusion, which can overcome the above-mentioned shortcomings of the prior art.

[0011] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows: A power equipment inspection method based on deep learning dual-light feature fusion includes the following steps: S1 Construct a multimodal synchronous acquisition system and data preprocessing: Establish a rigid binocular acquisition system containing a visible light camera and an infrared thermal imager. Achieve synchronous acquisition through hardware triggering or timestamp alignment to obtain visible light images and raw infrared thermal imaging data of power equipment at the same time. Map the raw infrared thermal imaging data into grayscale images for feature extraction and retain the raw data containing absolute temperature information. S2. Online coarse registration without calibration plate based on geometric constraints: extract significant line segment sets from visible light images and infrared grayscale images respectively; construct line segment geometric descriptors and establish matching line segment pairs between the two line segment sets; use the random sampling consistency algorithm to eliminate mismatches, solve the homography transformation matrix, and project and transform the infrared grayscale image to the visible light coordinate system to obtain a spatially aligned infrared image; S3 Constructs a dual-stream feature fusion network based on spatial attention mechanism: input the visible light image into the backbone network to extract multi-scale texture feature maps; input the registered infrared image into a lightweight convolutional network to generate a single-channel spatial attention mask; apply the spatial attention mask to the visible light feature map using Hadamard product to obtain the fused feature map; S4 Multi-scale Target Detection and Fault Region Regression: Input the fused feature map into the detection head to regress the target bounding box, equipment category probability and target confidence of power equipment at different scales; S5 Dual Threshold Fault Judgment and Temperature Inversion: Based on the detected target bounding box, its coordinates are mapped back to the coordinate system of the original infrared thermal imaging data, the temperature matrix of the corresponding area is extracted and the highest temperature is calculated; a confidence threshold and a temperature alarm threshold are set, and fault judgment and output are performed by combining the target confidence and the highest temperature.

[0012] Furthermore, in step S1, the synchronous acquisition is performed at the microsecond level; the original infrared thermal imaging data is linearly mapped to an 8-bit grayscale image for feature extraction.

[0013] Furthermore, in step S2, an improved line detection algorithm, including the LSD algorithm or Hough transform, is used to extract the set of significant line segments; the line segment geometric descriptor includes the line segment length, direction angle, midpoint coordinates, and relative geometric relationships of neighboring line segments; matching line segment pairs are established using geometric hashing or nearest neighbor search strategies.

[0014] Furthermore, in step S3, the lightweight convolutional network consists of two convolutional layers: the first layer uses a 3x3 convolutional kernel in conjunction with the ReLU activation function to extract local thermal gradient features of the infrared image; the second layer uses a 1x1 convolutional kernel to compress the number of channels to 1, and generates a normalized spatial attention mask through the Sigmoid activation function.

[0015] Furthermore, in step S3, the formula for feature fusion using the Hadamard product is: ,in, This is a visible light feature map. For spatial attention masking, This indicates element-wise multiplication; It is a learnable scalar adjustment factor, initially set to 0, and automatically updated based on the gradient of the loss function during network training.

[0016] Furthermore, in step S5, the fault determination logic includes: if the target confidence level is greater than the confidence level threshold and the highest temperature is greater than the temperature alarm threshold, then it is determined to be a confirmed fault; if the target confidence level is greater than the confidence level threshold but the highest temperature is less than or equal to the temperature alarm threshold, then it is determined to be a normally operating device; if the target confidence level is less than or equal to the confidence level threshold, then it is filtered.

[0017] Furthermore, in step S5, a dynamic temperature threshold is used for fault determination: the average background temperature of the current image is calculated, and the dynamic fault threshold is set as the sum of the average background temperature and a fixed temperature rise threshold. When the highest temperature of the target area exceeds the dynamic fault threshold and the target confidence exceeds the preset visual confidence threshold, it is determined to be a relative temperature rise fault.

[0018] Furthermore, in step S3, the backbone network is CSPDarknet53 or MobileNetV3-Large; the method is deployed on an edge computing platform or a low-power NPU platform, and the neural network model is quantized using INT8 to compress the model size and improve inference speed.

[0019] Furthermore, in step S2, a registration strategy combining pre-calibration and dynamic fine-tuning is adopted: the homography matrix under the standard distance is pre-calculated as the basic transformation matrix; during the operation of the device, the homography matrix fine-tuning based on line features is only initiated when severe screen jitter is detected.

[0020] The beneficial effects of this invention are as follows: By employing a calibration-free online registration method based on line segment geometric descriptors and the RANSAC algorithm, this invention effectively utilizes the inherent straight-line contour features of power equipment, enabling the system to achieve highly robust real-time spatial alignment even in complex inspection scenarios where infrared image textures are blurred and cameras vibrate. By introducing a spatial attention fusion mechanism controlled by learnable factors, the infrared thermal image is dynamically converted into a weighted mask focusing on high-temperature regions and adaptively weighted and fused with deep visible light features. This allows the network to simultaneously retain high-resolution texture details and abnormal temperature information, significantly improving the accuracy of fault component identification and classification. By constructing a dual-logic judgment rule combining "visual confidence" and "temperature threshold," the system can cross-verify the appearance and thermal characteristics of the target, effectively filtering out interference from environmental heat sources and significantly reducing the false alarm rate, thus achieving more reliable power equipment fault location and diagnosis. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0022] A power equipment inspection method based on deep learning dual-light feature fusion according to an embodiment of the present invention includes the following steps: S1 Construct a multimodal synchronous acquisition system and data preprocessing: Establish a rigid binocular acquisition system containing a visible light camera and an infrared thermal imager. Achieve synchronous acquisition through hardware triggering or timestamp alignment to obtain visible light images and raw infrared thermal imaging data of power equipment at the same time. Map the raw infrared thermal imaging data into grayscale images for feature extraction and retain the raw data containing absolute temperature information. S2. Online coarse registration without calibration plate based on geometric constraints: extract significant line segment sets from visible light images and infrared grayscale images respectively; construct line segment geometric descriptors and establish matching line segment pairs between the two line segment sets; use the random sampling consistency algorithm to eliminate mismatches, solve the homography transformation matrix, and project and transform the infrared grayscale image to the visible light coordinate system to obtain a spatially aligned infrared image; S3 Constructs a dual-stream feature fusion network based on spatial attention mechanism: input the visible light image into the backbone network to extract multi-scale texture feature maps; input the registered infrared image into a lightweight convolutional network to generate a single-channel spatial attention mask; apply the spatial attention mask to the visible light feature map using Hadamard product to obtain the fused feature map; S4 Multi-scale Target Detection and Fault Region Regression: Input the fused feature map into the detection head to regress the target bounding box, equipment category probability and target confidence of power equipment at different scales; S5 Dual Threshold Fault Judgment and Temperature Inversion: Based on the detected target bounding box, its coordinates are mapped back to the coordinate system of the original infrared thermal imaging data, the temperature matrix of the corresponding area is extracted and the highest temperature is calculated; a confidence threshold and a temperature alarm threshold are set, and fault judgment and output are performed by combining the target confidence and the highest temperature.

[0023] Preferably, in step S1, the synchronous acquisition is at the microsecond level; the original infrared thermal imaging data is linearly mapped to an 8-bit grayscale image for feature extraction.

[0024] Preferably, in step S2, an improved line detection algorithm, including the LSD algorithm or Hough transform, is used to extract the set of significant line segments; the line segment geometric descriptor includes the line segment length, direction angle, midpoint coordinates, and relative geometric relationships of neighboring line segments; and matching line segment pairs are established using geometric hashing or nearest neighbor search strategies.

[0025] Preferably, in step S3, the lightweight convolutional network consists of two convolutional layers: the first layer uses a 3x3 convolutional kernel in conjunction with the ReLU activation function to extract local thermal gradient features of the infrared image; the second layer uses a 1x1 convolutional kernel to compress the number of channels to 1, and generates a normalized spatial attention mask through the Sigmoid activation function.

[0026] Preferably, in step S3, the formula for feature fusion using the Hadamard product is: ,in, This is a visible light feature map. For spatial attention masking, This indicates element-wise multiplication; It is a learnable scalar adjustment factor, initially set to 0, and automatically updated based on the gradient of the loss function during network training.

[0027] Preferably, in step S5, the fault determination logic includes: if the target confidence level is greater than the confidence level threshold and the highest temperature is greater than the temperature alarm threshold, then it is determined to be a confirmed fault; if the target confidence level is greater than the confidence level threshold but the highest temperature is less than or equal to the temperature alarm threshold, then it is determined to be a normally operating device; if the target confidence level is less than or equal to the confidence level threshold, then it is filtered.

[0028] Preferably, in step S5, a dynamic temperature threshold is used for fault determination: the average background temperature of the current image is calculated, and the dynamic fault threshold is set as the sum of the average background temperature and a fixed temperature rise threshold. When the highest temperature of the target area exceeds the dynamic fault threshold and the target confidence exceeds the preset visual confidence threshold, it is determined to be a relative temperature rise fault.

[0029] Preferably, in step S3, the backbone network is CSPDarknet53 or MobileNetV3-Large; the method is deployed on an edge computing platform or a low-power NPU platform, and the neural network model is quantized with INT8 to compress the model size and improve inference speed.

[0030] Preferably, in step S2, a registration strategy combining pre-calibration and dynamic fine-tuning is adopted: the homography matrix under the standard distance is pre-calculated as the basic transformation matrix; during the operation of the device, the homography matrix fine-tuning based on line features is only initiated when severe screen jitter is detected.

[0031] To facilitate understanding of the above technical solutions of the present invention, the following detailed description of the above technical solutions of the present invention will be provided through specific usage methods.

[0032] In practical application, the power equipment inspection method based on deep learning dual-light feature fusion according to the present invention includes the following steps: Step S1: Construct a multimodal synchronous acquisition system and perform data preprocessing: A rigid binocular acquisition system comprising a visible light camera and an infrared thermal imager is established. Microsecond-level synchronous acquisition is achieved through hardware triggering or timestamp alignment to obtain visible light images of power equipment at the same moment. and raw infrared thermal imaging data ; raw infrared thermal imaging data Linear mapping to 8-bit grayscale image Used for feature extraction while retaining information including absolute temperature. Used for subsequent temperature inversion.

[0033] Step S2: Online coarse registration of the uncalibrated plate based on geometric constraints: Taking advantage of the characteristic that the outlines of equipment in power scenarios (such as cabinet edges, insulator strings, and conductors) are rich in linear features, visible light images of power equipment are analyzed. and 8-bit grayscale image Extract the set of salient line segments using improved line detection algorithms (such as LSD or Hough transform). and ; Construct a line segment geometric descriptor, including line segment length, direction angle, midpoint coordinates, and relative geometric relationships with neighboring line segments; Based on the descriptor, geometric hashing or nearest neighbor search strategies are used to... and Establish matching line segment pairs between them; The RANSAC (Random Sample Consensus) algorithm is used to eliminate mismatches. By minimizing the vertical distance error of the matching line segment pairs, the homography transformation matrix of the current viewpoint is calculated. Infrared grayscale image The projection is transformed to the visible light coordinate system to obtain a spatially aligned infrared image. .

[0034] Step S3: Construct a two-stream feature fusion network based on spatial attention mechanism: Construct a deep neural network that includes a visible light backbone branch, an infrared attention branch, and a feature fusion module: (1) Visible light main branch: Visible light image Input the backbone network to extract multi-scale texture feature maps. ; (2) Infrared attention branch (concretization structure): The registered infrared image Input a lightweight convolutional network. This lightweight network consists of two convolutional layers: the first layer uses a 3x3 convolutional kernel with a ReLU activation function to extract local thermal gradient features in the infrared image and map them to an intermediate feature space; the second layer uses a 1x1 convolutional kernel to compress the number of channels to 1, generating a single-channel spatial attention mask that can represent the weights of hotspot regions. Finally, the Sigmoid activation function is used to... The values ​​are normalized to An interval is defined to represent the thermal saliency probability of a pixel. (3) Feature fusion: The spatial attention mask is applied to the visible light feature map using the Hadamard product, as shown in the following formula:

[0035] in, This indicates element-wise multiplication. This is a learnable scalar adjustment factor, initially set to 0. During network backpropagation training, Automatic updates are based on the gradient of the loss function. Its purpose is to: in the early stages of training, prioritize learning visible light texture features to avoid infrared noise interference; as training converges, By gradually increasing and adaptively introducing infrared thermal feature weights, a dynamic balance between the two modalities of features is achieved. This step enables the network to significantly enhance the feature response in high-temperature regions while preserving visible light texture.

[0036] Step S4: Multi-scale target detection and fault region regression: The fused feature map The data is input to the detection head, which regresses the bounding box of the power equipment at different scales. ), Device Category Probability and target confidence level Redundant detection boxes are removed using non-maximum suppression (NMS), and the final target detection result is output.

[0037] Step S5: Dual threshold fault determination and temperature inversion: Based on the detected target bounding box Map its coordinates back to the original infrared thermal imaging data coordinate system; extraction The temperature matrix of the corresponding region is used to calculate the highest temperature in that region. And average temperature rise; set confidence threshold and temperature alarm threshold The following decision logic will be executed: like > and > If the fault is confirmed, the faulty device ID, precise location, and highest temperature will be output. like > but <= It was determined to be a normally operating device; like <= These are considered background interference or invalid targets and are filtered out.

[0038] Example 1: Substation Inspection Robot System Based on High-Performance Edge Computing Unit This embodiment is applicable to substation ground inspection robots, which have high requirements for computing power and real-time performance, and adopts the NVIDIA Jetson Orin platform and the improved YOLOv8 algorithm.

[0039] 1. Hardware system configuration (1) Binocular acquisition module: adopts a binocular structure fixed by a rigid bracket.

[0040] Visible light camera: Hikvision industrial camera (MV-CA050-20GM), resolution 2448X2048, global shutter; Infrared thermal imager: uncooled long-wave infrared core, resolution 640X512, thermal sensitivity (NEMD) <40mk; Synchronous triggering: using FPGA to generate a 10Hz pulse signal to simultaneously trigger the hard trigger pins of both cameras, ensuring that the time synchronization error between the two images is <1ms.

[0041] (2) Computing platform: NVIDIA Jetson AGX Orin 64GB, system environment is Ubuntu 20.04+CUDA11.4+TensorRT8.5.

[0042] 2. Detailed Algorithm Flow Step S1 (Data Preprocessing): Acquire images using the GigE interface. Retain the 16-bit raw infrared radiation data (RawData) for temperature measurement, and simultaneously normalize it to an 8-bit grayscale image through linear stretching for feature extraction.

[0043] Step S2 (Line Feature Registration): Visible light images The LSD (Line Segment Detector) algorithm is used to extract high-precision line segments; Canny edge detection and Hough Transform are used to extract edge line segments from the infrared image I_ir. Filtering strategy: Remove fragmented line segments with a length of less than 30 pixels and low confidence; Matching solution: Construct a triplet descriptor based on the "midpoint-length-angle" of a line segment, perform coarse matching using geometric hashing, and then calculate the homography matrix using the RANSAC algorithm. Complete image registration.

[0044] Step S3 (Construction of Feature Fusion Network): Backbone network: CSPDarknet53 is used to extract the C3, C4, and C5 feature layers of visible light images; Attention Module: Design a lightweight 3x3 convolutional layer to read the registered infrared image and output an attention map with 1 channel. ; Fusion operation: Execute the fusion formula in the C4 feature layer (corresponding to the medium-scale target): This operation amplifies the characteristic values ​​of the high-temperature region by 1.5 to 2 times.

[0045] Step S4 (Fault Determination and Output): The fused features are input into the detection head, and the bounding box is output. Temperature inversion: Map back to 16-bit raw infrared data to calculate the highest temperature in the region. Decision logic: Set , Result: If the conditions are met, the system automatically generates an alarm log and controls the robot's gimbal to lock onto the target for continuous tracking.

[0046] 3. Performance Testing and Effect Verification To verify the practical effectiveness of this embodiment, a 30-day field test was conducted at a 500kV substation, collecting and labeling 5000 sets of image samples containing faults such as insulator overheating and loose clamps. The test results are as follows: (1) Registration accuracy: Compared with the traditional registration method based on SURF feature points, the registration method based on line segment geometric constraints adopted in this embodiment improves the registration success rate from 82% to 98.5% in weak texture infrared images, and controls the average reprojection error within 2.4 pixels, which meets the requirements of accurate positioning.

[0047] (2) Fault detection accuracy: On the same test set, the mAP (mean accuracy) of using only visible light detection is 76.4% (greatly affected by nighttime lighting), and the mAP of using only infrared detection is 68.9% (high false alarm rate). However, by using the dual-light feature fusion method in this embodiment, the mAP is improved to 94.2%.

[0048] (3) Anti-interference capability: By introducing the steps in step S3 By adjusting the factor and using the dual threshold determination in step S5, the system reduces the false alarm rate for non-faulty heat sources such as vehicle exhaust and heated stones by 85%, effectively solving the problem of excessive false alarms in traditional infrared detection.

[0049] Example 2: UAV Power Line Inspection Terminal Based on Low-Power NPU This embodiment demonstrates the broad applicability of the invention, applicable to power-sensitive drone payloads or handheld devices, and is based on the Rockchip RK3588 NPU.

[0050] 1. Hardware system configuration Computing core: It adopts the domestic RK3588 development board and integrates an NPU with a computing power of 6 TOPS; Imaging module: Lightweight dual-light pod. Visible light uses a 1080P wide-angle module, and infrared uses a 384x288 micro-frame. Communication module: The alarm results are transmitted back to the central control center via a 5G module.

[0051] 2. Detailed Algorithm Flow Step S1 (Lightweight Model Design): Considering the power consumption limitations of the drone, the backbone network in step S3 was replaced with MobileNetV3-Large. Using the RKNN-Toolkit2 toolchain, the trained PyTorch model was quantized using INT8 (8-bit Quantization), compressing the model size to less than 10MB and increasing the inference speed to 45fps.

[0052] Step S2 (Rapid Registration Optimization): Due to the high-frequency vibrations during drone flight, a pre-calibration + dynamic fine-tuning strategy is adopted. The homography matrix at the standard distance is pre-calculated at the factory. During runtime, H-matrix fine-tuning based on line features is only initiated when severe image jitter is detected (based on IMU data); otherwise, it directly uses... This significantly reduces CPU load.

[0053] Step S3 (Determination of relative temperature threshold): Drone outdoor operations are greatly affected by environmental wind speed and temperature; therefore, a relative temperature rise determination method is used instead of a fixed threshold. The average background temperature of the current image is calculated. Set dynamic fault thresholds. = +, where Set to 15℃ (refer to DL / T664-2016 Specification for Infrared Diagnosis of Live Equipment). If the target area temperature exceeds... If the visual confidence level is greater than 0.8, it is determined to be a relative temperature rise fault.

[0054] 3. Performance Indicators Description This embodiment has been specifically optimized for the power consumption and real-time performance of the UAV payload. The specific indicators are as follows: (1) Lightweight model effect: After INT8 quantization and channel pruning, the size of the fusion network model is only 6.8MB. The inference time on the RK3588 NPU is 21ms / frame, achieving real-time detection of 47FPS, which fully meets the requirements of high-speed inspection of UAVs (flight speed 5-8m / s).

[0055] (2) Temperature inversion accuracy: Within a flight range of 5-15 meters from the equipment, the temperature measurement area is located by dual-light fusion, and the temperature measurement deviation is maintained within ±1.8℃ or ±1.8% (the maximum value is taken), which meets the DL / T 664-2016 accurate temperature measurement standard.

[0056] (3) Comparative advantages: Compared with the traditional UAV pods on the market that use "infrared picture-in-picture" overlay, the present invention can directly output the fused fault semantic tags, which improves the efficiency of the inspector's image review by about 3 times.

[0057] In summary, by employing the above-mentioned technical solution of this invention, and utilizing a calibration-free online registration method based on line segment geometric descriptors and the RANSAC algorithm, the inherent straight-line contour features of power equipment are effectively utilized. This enables the system to achieve highly robust real-time spatial alignment even in complex inspection scenarios where infrared image textures are blurred and cameras vibrate. By introducing a spatial attention fusion mechanism controlled by learnable factors, the infrared thermal image is dynamically converted into a weighted mask focusing on high-temperature regions and adaptively weighted and fused with deep visible light features. This allows the network to simultaneously retain high-resolution texture details and abnormal temperature information, significantly improving the accuracy of fault component identification and classification. By constructing a dual-logic judgment rule combining "visual confidence" and "temperature threshold," the system can cross-verify the appearance and thermal characteristics of the target, effectively filtering out interference from environmental heat sources and significantly reducing the false alarm rate, thus achieving more reliable power equipment fault location and diagnosis.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A power equipment inspection method based on deep learning dual-light feature fusion, characterized in that, Includes the following steps: S1 Construct a multimodal synchronous acquisition system and data preprocessing: Establish a rigid binocular acquisition system containing a visible light camera and an infrared thermal imager. Achieve synchronous acquisition through hardware triggering or timestamp alignment to obtain visible light images and raw infrared thermal imaging data of power equipment at the same time. Map the raw infrared thermal imaging data into grayscale images for feature extraction and retain the raw data containing absolute temperature information. S2 Online coarse registration without calibration plate based on geometric constraints: extracting sets of significant line segments from visible light images and infrared grayscale images respectively; Construct a geometric descriptor for line segments and establish matching line segment pairs between two sets of line segments; use the random sampling consistency algorithm to eliminate mismatches, solve the homography transformation matrix, and project and transform the infrared grayscale image to the visible light coordinate system to obtain a spatially aligned infrared image; S3 constructs a two-stream feature fusion network based on spatial attention mechanism: the visible light image is input into the backbone network to extract multi-scale texture feature maps; the registered infrared image is input into a lightweight convolutional network to generate a single-channel spatial attention mask; The spatial attention mask is applied to the visible light feature map using the Hadamard product to obtain the fused feature map; S4 Multi-scale Target Detection and Fault Region Regression: Input the fused feature map into the detection head to regress the target bounding box, equipment category probability and target confidence of power equipment at different scales; S5 Dual Threshold Fault Judgment and Temperature Inversion: Based on the detected target bounding box, its coordinates are mapped back to the coordinate system of the original infrared thermal imaging data, the temperature matrix of the corresponding area is extracted and the highest temperature is calculated; a confidence threshold and a temperature alarm threshold are set, and fault judgment and output are performed by combining the target confidence and the highest temperature.

2. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S1, the synchronous acquisition is at the microsecond level; the original infrared thermal imaging data is linearly mapped to an 8-bit grayscale image for feature extraction.

3. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S2, an improved line detection algorithm, including the LSD algorithm or Hough transform, is used to extract the set of significant line segments; the line segment geometric descriptor includes the line segment length, direction angle, midpoint coordinates, and relative geometric relationship of neighboring line segments; matching line segment pairs are established using geometric hashing or nearest neighbor search strategy.

4. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S3, the lightweight convolutional network consists of two convolutional layers: the first layer uses a 3x3 convolutional kernel in conjunction with the ReLU activation function to extract the local thermal gradient features of the infrared image; The second layer uses a 1x1 convolution kernel to compress the number of channels to 1, and generates a normalized spatial attention mask through a Sigmoid activation function.

5. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1 or 4, characterized in that, In step S3, the formula for feature fusion using the Hadamard product is as follows: ,in, This is a visible light feature map. For spatial attention masking, This indicates element-wise multiplication; It is a learnable scalar adjustment factor, initially set to 0, and automatically updated based on the gradient of the loss function during network training.

6. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S5, the fault determination logic includes: if the target confidence level is greater than the confidence level threshold and the highest temperature is greater than the temperature alarm threshold, then it is determined to be a confirmed fault; if the target confidence level is greater than the confidence level threshold but the highest temperature is less than or equal to the temperature alarm threshold, then it is determined to be a normally operating device; if the target confidence level is less than or equal to the confidence level threshold, then it is filtered.

7. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S5, a dynamic temperature threshold is used to determine the fault: the average background temperature of the current image is calculated, and the dynamic fault threshold is set as the sum of the average background temperature and a fixed temperature rise threshold. When the highest temperature of the target area exceeds the dynamic fault threshold and the target confidence exceeds the preset visual confidence threshold, it is determined to be a relative temperature rise fault.

8. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S3, the backbone network is CSPDarknet53 or MobileNetV3-Large; the method is deployed on an edge computing platform or a low-power NPU platform, and the neural network model is quantized with INT8 to compress the model size and improve inference speed.

9. The power equipment inspection method based on deep learning dual-light feature fusion according to claim 1, characterized in that, In step S2, a registration strategy combining pre-calibration and dynamic fine-tuning is adopted: the homography matrix under the standard distance is pre-calculated as the basic transformation matrix; During device operation, homography matrix fine-tuning based on line features is only initiated when severe screen jitter is detected.

Images