Miniature unmanned aerial vehicle power line inspection method and system based on stereoscopic thermal imaging

Abstract

The application discloses a micro unmanned aerial vehicle power line inspection method and system based on stereoscopic thermal imaging, the method comprising: acquiring a thermal infrared image sequence of a left thermal imaging camera and a right thermal imaging camera; pre-processing the thermal infrared image sequence image, and calculating the geometric centroid coordinates of each high-temperature area; obtaining matched feature points based on the geometric centroid coordinates of each high-temperature area; obtaining thermal source three-dimensional position information according to the matched feature points; fusing the calculated thermal source three-dimensional position information with the positioning information and the attitude information of the unmanned aerial vehicle, and adjusting the unmanned aerial vehicle to maintain a safe inspection distance; the system comprises a micro unmanned aerial vehicle flight platform, an on-board computing module and a stereoscopic vision acquisition module, and solves the technical problems of high cost, heavy load of high-resolution thermal imaging equipment in power inspection, inability of monocular vision to obtain depth information and difficulty of feature matching of low-resolution thermal images in a complex background.

Description

Technical Field

[0001] This application relates to the technical field of power line inspection, and in particular to a method and system for power line inspection using micro-UAVs based on stereoscopic thermal imaging. Background Technology

[0002] With the rapid development of the global economy and the advancement of digital transformation, electricity demand continues to rise, and the scale of the power grid is expanding daily. The security and reliability of power transmission networks are directly related to the national economy and people's livelihood. In the power system, transmission lines and substation equipment are exposed to the natural environment for a long time, enduring harsh weather conditions such as wind, sun, rain, and icing, while also bearing continuous high voltage and high current loads. These factors make power equipment prone to problems such as insulation aging, poor connector contact, and conductor corrosion and breakage. These defects often manifest as localized abnormal temperature increases (i.e., overheating) in the early stages. If not detected and dealt with in time, overheating can lead to equipment burning, insulation breakdown, and subsequently short circuits, tripping, or even large-scale power outages, causing huge economic losses and social impacts.

[0003] Therefore, regular condition inspections of power lines, especially thermal fault detection, are one of the core tasks of power operation and maintenance departments. Current power line inspection methods mainly include the following: Manual ground inspection: This method involves inspectors walking along the route using handheld infrared thermal imagers. This method is severely limited by terrain (such as mountains, rivers, and swamps), making it not only inefficient but also susceptible to obstructions due to the typically upward-looking perspective, hindering a comprehensive observation of the equipment status on the towers. Furthermore, inspection personnel must trek long distances, resulting in high labor intensity and potential safety risks in certain extreme environments.

[0004] Manned helicopter inspection: This method utilizes manned helicopters equipped with expensive electro-optical pods (including high-resolution visible light cameras, infrared cameras, and ultraviolet cameras) for inspections. It is highly efficient and can quickly cover long distances. However, its drawbacks are also significant: First, the cost is extremely high, reaching thousands of dollars per flight hour; second, the safety risks are high, as helicopters need to fly close to high-voltage lines at low altitudes, requiring highly skilled pilots, and the consequences of an accident could be disastrous; finally, due to the high speed of helicopters and limitations imposed by safe flight distances, they often only perform "scanning" inspections, limiting their ability to detect minute thermal defects.

[0005] Conventional drone inspection: In recent years, with the maturity of drone technology, multi-rotor drones have been widely used in power line inspection. Compared with manned helicopters, drones are lower in cost, more flexible, and pose no risk of personnel injury. However, existing drone thermal imaging inspection solutions mainly face the following technical bottlenecks: Lack of depth perception: The vast majority of commercial inspection drones are equipped with only a monocular thermal infrared camera. While monocular vision can generate thermal images and identify high-temperature points, it cannot directly obtain the precise distance (depth information) of the heat source relative to the drone. This lack of depth information leads to two serious problems: First, it is difficult to accurately locate the three-dimensional spatial coordinates of the fault point, and can only rely on the drone's own GPS coordinates for rough estimation, which makes subsequent maintenance and positioning difficult; second, the drone cannot autonomously judge the distance to energized equipment, and during close-range, detailed inspections, it is prone to collisions due to operational errors or environmental interference (such as gusts of wind), threatening the safety of the power grid.

[0006] The payload-cost trade-off of high-resolution equipment: To obtain clear thermal images for traditional image processing algorithms (such as edge detection and feature point matching), high-resolution thermal imaging modules (e.g., 640x512 pixels or higher) are typically required. These devices are large, heavy, power-hungry, and expensive (usually tens of thousands of yuan or more). Miniature drones (MAVs) have limited payload capacity (typically less than 500 grams) to carry such heavy equipment. However, MAVs, due to their small size, are best suited for precise inspections inside towers or in confined spaces.

[0007] The challenge of processing low-resolution images: To adapt to micro-drones, only lightweight, low-cost, low-resolution (e.g., 32x32 or 80x60 pixels) thermal sensors can be used. However, images from these sensors are extremely blurry, lacking texture detail, and the target object often occupies only a few pixels of dot-like area. Traditional stereo matching algorithms based on grayscale correlation (Block Matching) or feature points (SIFT, SURF, ORB) completely fail on such low-resolution and textureless images. How to achieve reliable stereo matching on extremely low-resolution thermal images to recover depth information is a major challenge for current technology.

[0008] To address the aforementioned problems, this invention proposes a method and system for inspecting power lines using micro-UAVs based on stereoscopic thermal imaging vision. This system utilizes a pair of specially calibrated and algorithmically optimized low-resolution thermal imaging cameras, combined with an innovative polygon geometric feature matching algorithm, to achieve real-time detection and three-dimensional localization of overheating faults in power equipment on a computationally limited micro-UAV platform, filling the gap in low-cost, lightweight stereoscopic thermal imaging inspection technology. Summary of the Invention

[0009] This application provides a power line inspection method based on stereoscopic thermal imaging vision to solve the above-mentioned technical problems.

[0010] In view of this, this application provides a power line inspection method based on stereoscopic thermal imaging vision, including the following steps: S1. Control the micro-drone equipped with a stereo thermal imaging camera to fly along a predetermined trajectory of the power line and acquire the thermal infrared image sequence of the left and right thermal imaging cameras in real time. S2. Perform time synchronization alignment and image preprocessing on the thermal infrared image sequence, filter background noise using a preset temperature threshold, segment high-temperature regions in the left and right images, and calculate the geometric centroid coordinates and average temperature value of each high-temperature region. S3. Based on the geometric centroid coordinates of each high-temperature region, construct a candidate feature point set; according to the distribution of the number of feature points in the candidate feature point set, use a polygon matching algorithm based on shape context to determine the correspondence between the high-temperature regions in the left and right images, and obtain the successfully matched feature points; S4. Calculate the disparity based on the difference in horizontal coordinates of the successfully matched feature points in the left and right images, and combine the baseline length and focal length of the stereo camera to calculate the vertical depth distance of the target heat source relative to the UAV camera, thereby obtaining the three-dimensional position information of the heat source. S5. The calculated three-dimensional location information of the heat source is fused with the positioning and attitude information of the UAV to generate the global spatial coordinates of the overheating fault point of the power equipment, and the flight control command of the UAV is dynamically adjusted according to the distance information to maintain a safe inspection distance.

[0011] Preferably, in step S2, the specific steps for time synchronization alignment and image preprocessing of the thermal infrared image sequence include: S21. Receive the left image frame stream generated by the left thermal imaging camera and the right image frame stream generated by the right thermal imaging camera in real time, and record the acquisition timestamp of each frame image respectively. S22. Using the nearest neighbor timestamp matching strategy, for the currently processed left image frame, the frame with the smallest absolute value of timestamp difference in the right image frame buffer queue is searched as the corresponding right image frame to form a stereo image pair. S23. Perform non-uniformity correction on the stereo image pair and set an overheating fault detection threshold. , to extract pixel temperature values ​​from the image Set the pixel value as the background value and retain The pixels are used as foreground hotspots; S24. Perform connected component analysis on the foreground hot spot to identify several independent heat source patches. The gray-scale weighted moment method was used to calculate the heat source patch for each heat source patch. subpixel-level centroid and the average temperature of all pixels within the patch. .

[0012] Preferably, in step S3, the polygon matching algorithm considers the number of feature points in the left and right images. and For different combinations of cases, a hierarchical matching strategy is executed, the hierarchical matching strategy including: First-level matching strategy: If the number of feature points in the left image... = 1 and the number of feature points in the right image When = 1, it is directly determined that the unique feature point in the left image and the unique feature point in the right image form a matching point pair; Second-level matching strategy: If = If the value is greater than 1, then construct the left feature polygon using all feature points of the left image as vertices. Construct a right feature polygon using all feature points of the right image as vertices. Then proceed to the polygon similarity calculation process; Third-level matching strategy: If and ,set up Selecting from images with a large number of feature points For each point, generate all possible subsets of candidate feature points, construct a virtual polygon for each subset of candidate feature points, calculate the similarity between each subset and the feature polygon in another image, and select the subset with the highest similarity as the best matching combination. Fourth-level matching strategy: If or If no valid matching target is detected in the current frame, the depth calculation for the current frame will be terminated.

[0013] Preferably, the specific steps for calculating the polygon similarity include: S31. Calculate the geometric center coordinates of all vertices of the polygon to be matched, subtract the geometric center coordinates from the coordinates of each vertex in the polygon, and transform the polygon into a local coordinate system with the geometric center as the origin. S32. Then connect the geometric center with each normalized vertex to form a set of eigenvectors; S33. Calculate the polar angle of each feature vector in the feature vector set relative to the coordinate axis of the camera imaging plane, and sort the vertices in clockwise or counterclockwise order; S34. Calculate the root mean square error (RMSE) between the eigenvector angle sequence of the polygon in the left figure and the eigenvector angle sequence of the candidate polygon in the right figure. S35. Select the vertex combination with the smallest RMSE and verify the average temperature difference of the corresponding vertices. Check if the temperature is less than the preset temperature tolerance threshold; if it is, then it is confirmed as the final matching result.

[0014] Preferably, in step S4, the formula for calculating the vertical depth distance of the target heat source relative to the UAV camera is:

[0015] in The depth distance of the target heat source. This is the equivalent focal length (in pixels) of a thermal imaging camera. The baseline lengths of the left and right thermal imaging cameras. and These are the horizontal pixel coordinates of the matched feature points in the left and right images, respectively. This represents the disparity value. The equivalent focal length Based on the camera's field of view and image pixel width The calculation yields the following formula: .

[0016] Preferably, the baseline length The value range is based on the minimum safe inspection distance. and the estimated width of the target object Determined, maximum baseline length The following constraints must be met:

[0017] in, The field of view of the thermal imaging camera; the baseline length on a micro-UAV platform. Set to 0.1-0.3 meters.

[0018] Another aspect of this application provides a power line inspection system based on stereoscopic thermal imaging vision for implementing the above-described method, comprising: Micro UAV flight platform: includes airframe, power propulsion system and underlying flight controller, used to carry mission payload and perform flight missions; The stereo thermal imaging acquisition module includes two identical low-resolution thermal imaging cameras with parallel optical axes and mounted horizontally with a fixed baseline, used to simultaneously acquire thermal radiation images of power lines and equipment. Airborne computing processing unit: electrically connected to the stereo thermal imaging acquisition module and the flight controller, and equipped with a memory and a processor, the processor being programmed to execute the power line inspection method as described in any one of claims 1 to 6; Multi-source sensing and positioning module: including GNSS global navigation satellite system receiver, laser rangefinder altimeter and 2D lidar, used to acquire the UAV's geographical location, relative altitude and environmental obstacle information.

[0019] Preferably, the resolution of the low-resolution thermal imaging camera is [resolution value missing]. Pixels, field of view is The frame rate is not less than 7Hz; the airborne computing processing unit asynchronously reads the data streams of the two thermal imaging cameras through the USB interface or UART interface, and performs timestamp alignment at the software layer.

[0020] Preferably, the system further includes an air-to-ground communication module, which uses the TCP Fast-Open protocol and data compression algorithm to transmit the three-dimensional coordinates, temperature data and corresponding thermal images of the overheating fault point calculated by the airborne computing processing unit to the ground monitoring terminal in real time; the underlying flight controller automatically adjusts the UAV attitude through the position control loop according to the heat source distance information output by the airborne computing processing unit to maintain a safe distance of more than 2 meters from the power line equipment.

[0021] A non-volatile computer-readable storage medium containing computer-executable instructions, which, when executed by a processor, cause the processor to perform the power line inspection method based on stereo thermal imaging vision described in this application.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Employing a low-resolution thermal sensor, compared to traditional high-resolution thermal imagers, results in a significant reduction in weight and cost, allowing the system to be integrated into micro-UAVs and greatly lowering the barrier to entry for inspection equipment. A polygon matching algorithm based on sparse centroids eliminates reliance on image texture, enabling accurate target matching in extremely low-resolution thermal images with low computational complexity, making it suitable for real-time operation on embedded platforms. Stereo vision recovers the depth information of the heat source, not only for precise fault location but also providing crucial obstacle avoidance data for UAVs, enabling them to operate safely near power lines and overcoming the limitations of monocular thermal imaging's "see but not touch" capability. The algorithm fully considers the inconsistency in point counts caused by occlusion and false detections, using a combined optimization strategy to find the optimal subset, effectively reducing the false matching rate and adapting to complex and ever-changing inspection environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a step diagram of a power line inspection method based on stereoscopic thermal imaging vision provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall hardware architecture and sensor layout of a power line inspection system based on stereoscopic thermal imaging vision provided in this application embodiment; Figure 3 This is a flowchart of the core algorithm of a power line inspection method based on stereoscopic thermal imaging vision provided in this application embodiment; Figure 4 This is a schematic diagram of the stereoscopic vision epipolar geometry principle used in a power line inspection method based on stereoscopic thermal imaging vision provided in this application embodiment. Figure 5 This is a logical diagram of the polygon matching algorithm of a power line inspection method based on stereo thermal imaging vision provided in this application embodiment, under different numbers of feature points. Figure 6 This is a geometric diagram illustrating the polygon normalization and feature vector angle matching of a power line inspection method based on stereo thermal imaging vision provided in this embodiment of the application. Figure 7 This is an analysis curve showing the influence of the baseline length on the depth measurement range and theoretical error of a power line inspection method based on stereoscopic thermal imaging vision provided in this embodiment of the application. Figure 8 This image shows the effect verification of the power line inspection method and system based on stereoscopic thermal imaging vision provided in this application embodiment in detecting and locating multiple heat source targets in the Gazebo simulation environment and real flight experiment. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0026] For easier understanding, please refer to Figures 1 to 8 This application provides a power line inspection method based on stereoscopic thermal imaging vision, including the following steps: S1. Control the micro-drone equipped with a stereo thermal imaging camera to fly along a predetermined trajectory of the power line and acquire the thermal infrared image sequence of the left and right thermal imaging cameras in real time. S2. Perform time synchronization alignment and image preprocessing on the thermal infrared image sequence, filter background noise using a preset temperature threshold, segment high-temperature regions in the left and right images, and calculate the geometric centroid coordinates and average temperature value of each high-temperature region. Specifically, in step S2, the specific steps for time synchronization alignment and image preprocessing of the thermal infrared image sequence include: S21. Receive the left image frame stream generated by the left thermal imaging camera and the right image frame stream generated by the right thermal imaging camera in real time, and record the acquisition timestamp of each frame image respectively. S22. Using the nearest neighbor timestamp matching strategy, for the currently processed left image frame, the frame with the smallest absolute value of timestamp difference in the right image frame buffer queue is searched as the corresponding right image frame to form a stereo image pair. S23. Perform non-uniformity correction on the stereo image pair and set an overheating fault detection threshold. , to extract pixel temperature values ​​from the image Set the pixel value as the background value and retain The pixels are used as foreground hotspots; S24. Perform connected component analysis on the foreground hot spot to identify several independent heat source patches. The gray-scale weighted moment method was used to calculate the heat source patch for each heat source patch. subpixel-level centroid and the average temperature of all pixels within the patch. .

[0027] S3. Based on the geometric centroid coordinates of each high-temperature region, construct a candidate feature point set; according to the distribution of the number of feature points in the candidate feature point set, use a polygon matching algorithm based on shape context to determine the correspondence between the high-temperature regions in the left and right images, and obtain the successfully matched feature points; Specifically, in step S3, the polygon matching algorithm considers the number of feature points in the left and right images. and For different combinations of cases, a hierarchical matching strategy is executed, the hierarchical matching strategy including: First-level matching strategy: If the number of feature points in the left image... = 1 and the number of feature points in the right image When = 1, it is directly determined that the unique feature point in the left image and the unique feature point in the right image form a matching point pair; Second-level matching strategy: If = If the value is greater than 1, then construct the left feature polygon using all feature points of the left image as vertices. Construct a right feature polygon using all feature points of the right image as vertices. Then proceed to the polygon similarity calculation process; Third-level matching strategy: If and ,set up Selecting from images with a large number of feature points For each point, generate all possible subsets of candidate feature points, construct a virtual polygon for each subset of candidate feature points, calculate the similarity between each subset and the feature polygon in another image, and select the subset with the highest similarity as the best matching combination. Fourth-level matching strategy: If or If no valid matching target is detected in the current frame, the depth calculation for the current frame is terminated. In detail, the specific steps for calculating the polygon similarity include: S31. Calculate the geometric center coordinates of all vertices of the polygon to be matched, subtract the geometric center coordinates from the coordinates of each vertex in the polygon, and transform the polygon into a local coordinate system with the geometric center as the origin. S32. Then connect the geometric center with each normalized vertex to form a set of eigenvectors; S33. Calculate the polar angle of each feature vector in the feature vector set relative to the coordinate axis of the camera imaging plane, and sort the vertices in clockwise or counterclockwise order; S34. Calculate the root mean square error (RMSE) between the eigenvector angle sequence of the polygon in the left figure and the eigenvector angle sequence of the candidate polygon in the right figure. S35. Select the vertex combination with the smallest RMSE and verify the average temperature difference of the corresponding vertices. Is the temperature less than the preset temperature tolerance threshold? If it is less than the threshold, then the result is confirmed as the final match. S4. Calculate the disparity based on the difference in horizontal coordinates of the successfully matched feature points in the left and right images, and combine the baseline length and focal length of the stereo camera to calculate the vertical depth distance of the target heat source relative to the UAV camera, thereby obtaining the three-dimensional position information of the heat source. Specifically, in step S4, the formula for calculating the vertical depth distance of the target heat source relative to the UAV camera is as follows:

[0028] in The depth distance of the target heat source. This is the equivalent focal length (in pixels) of a thermal imaging camera. The baseline lengths of the left and right thermal imaging cameras. and These are the horizontal pixel coordinates of the matched feature points in the left and right images, respectively. This represents the disparity value. The equivalent focal length Based on the camera's field of view and image pixel width The calculation yields the following formula: .

[0029] In detail, the baseline length The value range is based on the minimum safe inspection distance. and the estimated width of the target object Determined, maximum baseline length The following constraints must be met:

[0030] in, The field of view of the thermal imaging camera; the baseline length on a micro-UAV platform. Set to 0.1-0.3 meters.

[0031] S5. The calculated three-dimensional location information of the heat source is fused with the positioning and attitude information of the UAV to generate the global spatial coordinates of the overheating fault point of the power equipment, and the flight control command of the UAV is dynamically adjusted according to the distance information to maintain a safe inspection distance.

[0032] Another aspect of this application provides a power line inspection system based on stereoscopic thermal imaging vision for implementing the above-described method, comprising: Micro UAV flight platform: includes airframe, power propulsion system and underlying flight controller, used to carry mission payload and perform flight missions; The stereo thermal imaging acquisition module includes two identical low-resolution thermal imaging cameras with parallel optical axes and mounted horizontally with a fixed baseline, used to simultaneously acquire thermal radiation images of power lines and equipment. Airborne computing processing unit: electrically connected to the stereo thermal imaging acquisition module and the flight controller, and equipped with a memory and a processor, the processor being programmed to execute the power line inspection method as described in any one of claims 1 to 6; Multi-source sensing and positioning module: including GNSS global navigation satellite system receiver, laser rangefinder altimeter and 2D lidar, used to acquire the UAV's geographical location, relative altitude and environmental obstacle information.

[0033] Preferably, the resolution of the low-resolution thermal imaging camera is [resolution value missing]. Pixels, field of view is The frame rate is not less than 7Hz; the airborne computing processing unit asynchronously reads the data streams of the two thermal imaging cameras through the USB interface or UART interface, and performs timestamp alignment at the software layer.

[0034] Preferably, the system further includes an air-to-ground communication module, which uses the TCP Fast-Open protocol and data compression algorithm to transmit the three-dimensional coordinates, temperature data and corresponding thermal images of the overheating fault point calculated by the airborne computing processing unit to the ground monitoring terminal in real time; the underlying flight controller automatically adjusts the UAV attitude through the position control loop according to the heat source distance information output by the airborne computing processing unit to maintain a safe distance of more than 2 meters from the power line equipment.

[0035] A non-volatile computer-readable storage medium containing computer-executable instructions, which, when executed by a processor, cause the processor to perform the power line inspection method based on stereo thermal imaging vision described in this application.

[0036] Example 2: System Hardware Platform Construction: like Figure 2 As shown, this embodiment constructs a prototype power inspection system based on micro unmanned aerial vehicles (MAVs).

[0037] Flight platform: A DJI F450 quadcopter frame with a 450mm wheelbase is used, a size that balances payload capacity with flexibility for both indoor and outdoor flight. The power system uses a 2212 motor paired with a 9450 propeller.

[0038] Flight control: The core controller uses the Pixhawk 4 flight controller, running the PX4 open-source firmware, and is responsible for attitude stabilization, motor control, and low-level safety protection (such as low battery return-to-home).

[0039] Onboard computing unit: Equipped with an Intel NUC mini PC (i7-8559U processor, 16GB RAM), running Ubuntu 18.04 operating system and ROS (Robot Operating System) Melodic version. This unit is the "brain" of the system, responsible for running all vision algorithms, data fusion, and high-level task planning.

[0040] Stereo thermal imaging module: This is the core sensing component of the system. It uses two Teraranger EVO Thermal 33 thermal imaging cameras.

[0041] Resolution: Pixels. Although the resolution is low, the data volume is extremely small, which is beneficial for high-frequency processing. Field of view (FoV): .

[0042] Installation method: The two cameras are fixed on a 3D-printed rigid bracket, with the optical axes strictly parallel and the horizontal baseline length... It is set to 0.2 meters.

[0043] Data interface: Connects to the onboard computer via USB interface.

[0044] Auxiliary sensors: RPLIDAR A2: A 360-degree 2D LiDAR used to acquire environmental contours and assist in SLAM positioning.

[0045] Garmin Lidar-Lite V3: A single-point laser rangefinder, mounted downwards, providing high-precision ground altitude data.

[0046] BlueFOX-MLC200wC: Optical flow camera for horizontal velocity estimation in GPS-free environments.

[0047] Ublox Neo-M8N: GNSS receiver that provides outdoor latitude and longitude coordinates.

[0048] Example 3: Thermal Radiation Imaging Model and Parameter Settings: Thermal imaging cameras determine temperature by detecting the infrared radiation energy emitted by an object. According to the Stefan-Boltzmann law:

[0049] in Radiant exitance ( ), The Stefan-Boltzmann constant is... This refers to absolute temperature. Emissivity, ranging from 0 to 1. In this embodiment, the camera has a preset emissivity. = 0.95. This applies to most non-metallic electrical materials (such as ceramic insulators and aged rubber sheaths). For highly reflective metals (such as new aluminum wire), the low emissivity may result in a measured temperature lower than the actual temperature. However, in fault detection scenarios, hot spots are often accompanied by oxidation or carbonization, resulting in higher emissivity. Furthermore, this system primarily focuses on relative temperature differences rather than absolute temperature measurement accuracy; therefore, this model is sufficient for fault screening requirements. The raw data output by the camera is... The temperature matrix is ​​in degrees Celsius.

[0050] Example 4: Image Processing and Feature Extraction Process: like Figure 3 As shown, the image processing workflow mainly runs within the ROS nodes of the onboard computer: Data synchronization: Data streams from the left and right cameras arrive asynchronously. The program maintains two double-ended queues to store the most recently received left and right image frames, respectively. When a left image frame is received, the program iterates through the right image queue, searching for the timestamp difference. The frame with the smallest difference is selected. If a match is found, the frame is sent to the next level as a stereo image pair; otherwise, it is discarded.

[0051] Threshold filtering and segmentation: Background Removal: Setting an Overheat Threshold This threshold is set based on the normal operating temperature of power equipment (typically <90°C) and can effectively shield normal lines, towers, and environmental background.

[0052] Binarization: like Otherwise, it is 0. Connected component analysis: Traverse the binary image, merge adjacent "1" pixels, and extract the connected components. A connected component (Blob).

[0053] Centroid Extraction: To improve the positional accuracy of feature points and combat quantization errors caused by low resolution, temperature-weighted centroid calculation is employed.

[0054] in It is the first in the connected domain The coordinates of a pixel. It is its temperature value. The final output is the feature point set of the left image. and right image feature point set .

[0055] Example 5: A polygon matching algorithm based on the generalized Hough transform: After extracting feature points, the core challenge lies in determining which point in the left image corresponds to which point in the right image (stereo matching). Due to the lack of texture, region matching methods such as SAD or SSD cannot be used. Inspired by the shape feature-based object recognition in the Generalized Hough Transform (GHT), this invention proposes a polygon matching algorithm. The algorithm logic is as follows: Figure 5 As shown: Input: Left image point set (size ), right image point set (size ).

[0056] Processing branch: Case A ( =1, =1): Directly assume that the two points match.

[0057] Case B ( ): respectively and Construct polygons using points as vertices and .

[0058] Normalization (e.g.) Figure 6 ): Calculate the geometric center of a polygon Transform all vertex coordinates into relative coordinates .

[0059] Feature description: Calculate each relative vector Magnitude and phase angle .

[0060] Similarity metric: Sort the vertices of the left and right polygons by phase angle and calculate the root mean square error (RMSE) of the phase angles of the corresponding vertices. If Threshold and the average temperature difference of the corresponding points If the match is successful, then the match is successful.

[0061] Case C ( ): Assumption .

[0062] This means that the left image may have detected more noise, or the right image may have missed the target (e.g., it was occluded).

[0063] Subset traversal: from Selected from All combinations of points ( (Types). For each combination, construct a virtual polygon, and follow the method of Case B. Perform a matching calculation to determine the RMSE.

[0064] The combination with the smallest RMSE is selected as the final match. Unselected left image points are marked as no match.

[0065] The advantage of this method is that it utilizes the relative geometrical positional relationships (topology) between multiple points, rather than the appearance features of a single point, thus exhibiting strong robustness to blurring and deformation at low resolution.

[0066] Example 5: Depth Estimation Model and Error Analysis According to polar geometry (e.g.) Figure 4 ),depth Parallax The relationship is:

[0067] For the Teraranger camera in this system: Image width Pixel.

[0068] Field of view .

[0069] equivalent focal length Pixel.

[0070] Error analysis: due to It is a discrete pixel difference, and Very small, minute changes in parallax can cause A huge jump. For example, when hour: like pixels, then .

[0071] like pixels, then .

[0072] like pixels, then .

[0073] like pixels, then .

[0074] It can be seen that the resolution decreases with increasing distance. There is no discernible intermediate value between 3.6m and 5.4m. However, for the purposes of "preliminary screening" and "safe obstacle avoidance" in power line inspections: Obstacle avoidance: As long as the system detects an obstacle... (i.e., distance) This will trigger an alarm or hover, which is sufficient to prevent collisions.

[0075] Location: When hovering at close range (e.g., 2-3m), the parallax is relatively large (e.g., 5-6 pixels), and the depth resolution is relatively high (about 0.4m), which is sufficient to locate the position of the insulator string where the fault is located. Figure 7 This demonstrates the impact of baseline length on the measurement range. A longer baseline results in a larger near-range blind zone (due to excessive parallax exceeding the field of view), but higher accuracy at long ranges. Considering the size limitations of the micro-UAV (450mm frame diameter) and the safe inspection distance (2m), this invention selects $b=0.2m$ as the optimal solution, ensuring a parallax of approximately 5.4 pixels at 2m, within the camera's optimal observation sweet spot.

[0076] Example 6: Simulation and Experimental Verification Gazebo simulation: A simulation scene was built that includes power transmission towers and simulated heat sources (red spheres). Figure 8 The heat source temperature was set to 150°C. The drone moved within a range of 2m to 4m from the heat source. Data recordings are shown in the table below:

[0077] Real-world flight testing: An outdoor simulated test platform was built, using a heating resistor as the heat source. The drone performed autonomous hovering detection. Real-world data showed that due to thermal noise and airflow turbulence in the real environment, the measurement standard deviation increased (approximately 0.006m at 2.5m and 0.048m at 3.5m), but the overall error remained within 5%. The system successfully guided the drone to hover at a distance of approximately 2.6 meters from the heat source, verifying the feasibility of closed-loop control. The experiment also revealed that when the target distance exceeded 4 meters, the detection rate significantly decreased because the target occupied only one pixel or less in the image. This confirms the rationale for positioning this system as a "micro-drone close-range fine-grained inspection system."

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A power line inspection method based on stereoscopic thermal imaging vision, characterized in that, Includes the following steps: S1. Control the micro-drone equipped with a stereo thermal imaging camera to fly along a predetermined trajectory of the power line and acquire the thermal infrared image sequence of the left and right thermal imaging cameras in real time. S2. Perform time synchronization alignment and image preprocessing on the thermal infrared image sequence, filter background noise using a preset temperature threshold, segment high-temperature regions in the left and right images, and calculate the geometric centroid coordinates and average temperature value of each high-temperature region. S3. Construct a set of candidate feature points based on the geometric centroid coordinates of each high-temperature region; Based on the distribution of the number of feature points in the candidate feature point set, a polygon matching algorithm based on shape context is used to determine the correspondence between the high-temperature regions in the left and right images, and to obtain the successfully matched feature points. S4. Calculate the disparity based on the difference in horizontal coordinates of the successfully matched feature points in the left and right images, and combine the baseline length and focal length of the stereo camera to calculate the vertical depth distance of the target heat source relative to the UAV camera, thereby obtaining the three-dimensional position information of the heat source. S5. The calculated three-dimensional location information of the heat source is fused with the positioning and attitude information of the UAV to generate the global spatial coordinates of the overheating fault point of the power equipment. The flight control commands of the UAV are dynamically adjusted according to the distance information to maintain a safe inspection distance.

2. The power line inspection method based on stereoscopic thermal imaging vision according to claim 1, characterized in that, In step S2, the specific steps for time synchronization alignment and image preprocessing of the thermal infrared image sequence include: S21. Receive the left image frame stream generated by the left thermal imaging camera and the right image frame stream generated by the right thermal imaging camera in real time, and record the acquisition timestamp of each frame image respectively. S22. Using the nearest neighbor timestamp matching strategy, for the currently processed left image frame, the frame with the smallest absolute value of timestamp difference in the right image frame buffer queue is searched as the corresponding right image frame to form a stereo image pair. S23. Perform non-uniformity correction on the stereo image pair and set an overheating fault detection threshold. , to extract pixel temperature values ​​from the image Set the pixel value as the background value and retain The pixels are used as foreground hotspots; S24. Perform connected component analysis on the foreground hot spot to identify several independent heat source patches. The gray-scale weighted moment method was used to calculate the heat source patch for each heat source patch. subpixel-level centroid and the average temperature of all pixels within the patch. .

3. The power line inspection method based on stereoscopic thermal imaging vision according to claim 1, characterized in that, In step S3, the polygon matching algorithm considers the number of feature points in the left and right images. and For different combinations of cases, a hierarchical matching strategy is executed, the hierarchical matching strategy including: First-level matching strategy: If the number of feature points in the left image... = 1 and the number of feature points in the right image When = 1, it is directly determined that the unique feature point in the left image and the unique feature point in the right image form a matching point pair; Second-level matching strategy: If = If the value is greater than 1, then construct the left feature polygon using all feature points of the left image as vertices. Construct a right feature polygon using all feature points of the right image as vertices. Then proceed to the polygon similarity calculation process; Third-level matching strategy: If and ,set up Selecting from images with a large number of feature points For each point, generate all possible subsets of candidate feature points, construct a virtual polygon for each subset of candidate feature points, calculate the similarity between each subset and the feature polygon in another image, and select the subset with the highest similarity as the best matching combination. Fourth-level matching strategy: If or If no valid matching target is detected in the current frame, the depth calculation for the current frame will be terminated.

4. The power line inspection method based on stereoscopic thermal imaging vision according to claim 3, characterized in that, The specific steps for calculating polygon similarity include: S31. Calculate the geometric center coordinates of all vertices of the polygon to be matched, subtract the geometric center coordinates from the coordinates of each vertex in the polygon, and transform the polygon into a local coordinate system with the geometric center as the origin. S32. Then connect the geometric center with each normalized vertex to form a set of eigenvectors; S33. Calculate the polar angle of each feature vector in the feature vector set relative to the coordinate axis of the camera imaging plane, and sort the vertices in clockwise or counterclockwise order; S34. Calculate the root mean square error (RMSE) between the eigenvector angle sequence of the polygon in the left figure and the eigenvector angle sequence of the candidate polygon in the right figure. S35. Select the vertex combination with the smallest RMSE and verify the average temperature difference of the corresponding vertices. Check if the temperature is less than the preset temperature tolerance threshold; if it is, then it is confirmed as the final matching result.

5. The power line inspection method based on stereoscopic thermal imaging vision according to claim 1, characterized in that, In step S4, the formula for calculating the vertical depth distance of the target heat source relative to the UAV camera is: in The depth distance of the target heat source. This is the equivalent focal length (in pixels) of a thermal imaging camera. The baseline lengths of the left and right thermal imaging cameras. and These are the horizontal pixel coordinates of the matched feature points in the left and right images, respectively. This represents the disparity value. The equivalent focal length Based on the camera's field of view and image pixel width The calculation yields the following formula: 。 6. The power line inspection method based on stereoscopic thermal imaging vision according to claim 5, characterized in that, The baseline length The value range is based on the minimum safe inspection distance. and the estimated width of the target object Determined, maximum baseline length The following constraints must be met: in, The field of view of the thermal imaging camera; the baseline length on a micro-UAV platform. Set to 0.1-0.3 meters.

7. A miniature unmanned aerial vehicle (UAV) power line inspection system based on stereoscopic thermal imaging vision, characterized in that, include: Micro UAV flight platform: including airframe, power propulsion system and underlying flight controller, used to carry mission payload and perform flight missions; The stereo thermal imaging acquisition module includes two identical low-resolution thermal imaging cameras with parallel optical axes and mounted horizontally with a fixed baseline, used to simultaneously acquire thermal radiation images of power lines and equipment. Airborne computing processing unit: electrically connected to the stereo thermal imaging acquisition module and the flight controller, and equipped with a memory and a processor, the processor being programmed to execute the power line inspection method as described in any one of claims 1 to 6; Multi-source sensing and positioning module: including GNSS global navigation satellite system receiver, laser rangefinder altimeter and 2D lidar, used to acquire the UAV's geographical location, relative altitude and environmental obstacle information.

8. The micro-UAV power line inspection system based on stereoscopic thermal imaging vision according to claim 7, characterized in that, The resolution of the low-resolution thermal imaging camera is Pixels, field of view is The frame rate is not less than 7Hz; the airborne computing processing unit asynchronously reads the data streams of the two thermal imaging cameras through the USB interface or UART interface, and performs timestamp alignment at the software layer.

9. The micro-UAV power line inspection system based on stereoscopic thermal imaging vision according to claim 7, characterized in that, The system also includes an air-to-ground communication module, which uses the TCP Fast-Open protocol and data compression algorithm to transmit the three-dimensional coordinates, temperature data and corresponding thermal images of the overheating fault point calculated by the airborne computing unit to the ground monitoring terminal in real time; the underlying flight controller automatically adjusts the UAV attitude through the position control loop according to the heat source distance information output by the airborne computing unit to maintain a safe distance of more than 2 meters from the power line equipment.

10. A non-volatile computer-readable storage medium containing computer-executable instructions, characterized in that, When executed by a processor, the computer-executable instructions cause the processor to perform the power line inspection method based on stereoscopic thermal imaging vision as described in any one of claims 1 to 6.

Images