Line inspection method and system based on unmanned aerial vehicle visual navigation

Abstract

The application relates to a line inspection method and system based on unmanned aerial vehicle visual navigation, and relates to the technical field of line inspection.The application comprises the following steps: performing enhancement processing on power equipment line images collected by an inspection unmanned aerial vehicle in an inspection area to obtain enhanced equipment line images; performing multi-scale feature extraction on the enhanced equipment line images to obtain equipment line feature maps, and performing feature fusion on the equipment line feature maps to obtain a fused feature map; performing target detection analysis on the fused feature map to obtain line hidden danger data and obstacle position data, performing labeling processing on the line hidden danger data to obtain inspection report data; updating a current flight path of the inspection unmanned aerial vehicle based on the obstacle position data to obtain obstacle avoidance path data, and adjusting the flight attitude parameters of the inspection unmanned aerial vehicle according to the obstacle avoidance path data, so that the technical problems that manual inspection has been difficult to meet the growing safety monitoring demand, the inspection cycle is long, and the coverage range is limited are solved.

Description

Technical Field

[0001] This invention relates to the field of line inspection technology, and in particular to a line inspection method and system based on UAV visual navigation. Background Technology

[0002] As a critical infrastructure for energy transmission in modern society, the safe and stable operation of power lines directly affects the normal functioning of industrial production and residential life. Traditional power line inspections mainly rely on manual climbing of towers or observation using simple tools such as binoculars. This method is not only inefficient but also poses significant safety hazards, especially in complex terrain environments such as mountainous areas and areas spanning rivers, where inspectors face multiple risks including working at heights and adverse weather conditions. With the continuous expansion of the power grid and the extension of line length, manual inspections can no longer meet the ever-increasing demand for safety monitoring. Problems such as long inspection cycles and limited coverage are becoming increasingly prominent, urgently requiring the exploration of more efficient and safer inspection technologies.

[0003] In recent years, the rapid development of drone technology has brought new solutions to power line inspection, offering advantages such as mobility, flexibility, and relatively low cost compared to traditional methods. However, current drone inspection still faces several technical bottlenecks: First, the quality of visual images is severely affected by factors such as changes in lighting and weather conditions, resulting in insufficient contrast and blurred details in the acquired line images, making it difficult to accurately identify potential line hazards; second, the ability to identify and avoid obstacles in complex environments is insufficient, as obstacles such as trees, buildings, and birds can easily pose a threat to flight safety, and existing navigation systems are slow to react when dealing with such dynamic obstacles; third, there is a lack of effective multi-scale feature extraction mechanisms, leading to inconsistent accuracy in identifying defects and anomalies of different sizes, and early hazards such as small cracks and rust spots are easily missed. Summary of the Invention

[0004] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides a route inspection method based on UAV visual navigation, comprising the following steps:

[0006] Images of power equipment and lines collected by inspection drones in the inspection area are enhanced to obtain enhanced images of the equipment and lines; wherein, the inspection area includes power equipment and the power lines connecting the power equipment.

[0007] Multi-scale feature extraction is performed on the enhanced equipment line image to obtain an equipment line feature map, and feature fusion is performed on the equipment line feature map to obtain a fused feature map;

[0008] The fused feature map is subjected to target detection analysis to obtain line hazard data and obstacle location data. The line hazard data is then labeled to obtain inspection report data.

[0009] The current flight path of the inspection drone is updated based on the obstacle location data to obtain obstacle avoidance path data, and the flight attitude parameters of the inspection drone are adjusted according to the obstacle avoidance path data.

[0010] Furthermore, the enhancement processing of the power equipment and line images collected by the inspection drone in the inspection area to obtain enhanced equipment and line images includes:

[0011] The images of power equipment lines collected by the inspection drone in the inspection area are processed by brightness partitioning, and the images of power equipment lines are divided into multiple brightness sub-regions to obtain a brightness distribution map.

[0012] Brightness equalization calculations are performed on each brightness sub-region in the brightness distribution map to obtain an enhanced device line image.

[0013] Furthermore, the step of performing multi-scale feature extraction on the enhanced equipment circuit image to obtain an equipment circuit feature map includes:

[0014] The enhanced equipment circuit image is divided into multiple scales to obtain a multi-layer image group;

[0015] Edge contours are extracted for each scale layer in the multi-layer image group to identify the edge information of power equipment at different scales, resulting in a multi-scale contour map. Geometric feature statistics are then performed on the multi-scale contour map to obtain a feature distribution map.

[0016] The feature distribution map is divided into multiple feature sub-regions to obtain a regional feature map. Spatial relationship calculations are then performed on the feature sub-regions in the regional feature map to obtain a device circuit feature map.

[0017] Furthermore, spatial relationship calculations are performed on the feature sub-regions in the aforementioned region feature map to obtain the equipment circuit feature map, including:

[0018] Centroid coordinates are extracted from each feature sub-region in the region feature map to obtain a set of equipment center points. Equipment type is labeled for each center point in the set of equipment center points to obtain an equipment location label map including tower location points, insulator location points and conductor connection points.

[0019] Based on the equipment location marking map, the Euclidean distance between adjacent power equipment is calculated to obtain an equipment spacing table. The electrical connection relationship between power equipment is determined according to the equipment spacing table to obtain an equipment association matrix. The equipment association matrix records the connection status between towers and insulators, and between insulators and conductors.

[0020] Based on the device association matrix, topological connection segments between devices are drawn on the regional feature map to obtain a topological connection map. The topological connection map is then overlaid with the device location annotation map to obtain a line feature map.

[0021] Furthermore, feature fusion is performed on the device circuit feature map to obtain a fused feature map, including:

[0022] The importance of the features in the equipment line feature map is evaluated, and an importance score is obtained by analyzing the criticality of different feature sub-regions in power line inspection.

[0023] The equipment line feature map is weighted based on the importance score to obtain a weighted feature map. Feature overlay calculation is then performed on the weighted feature map to add features at the same location at different scales to obtain a preliminary fused feature map.

[0024] Redundant features with high correlation and little impact on the inspection results are removed from the preliminary fusion feature map to obtain the fusion feature map.

[0025] Furthermore, the target detection analysis performed on the fused feature map to obtain line hazard data and obstacle location data includes:

[0026] The fused feature map is divided into multiple equipment area maps according to the distribution of power equipment in the inspection area, and the area division results are obtained. The edge pixel statistics of each equipment area map in the area division results are obtained to obtain the edge pixel distribution.

[0027] Based on the edge pixel distribution, the device region map is contour extracted to identify the outline of the power equipment in the device region map, and the outline of the power equipment is morphologically dilated to obtain a dilated outline map.

[0028] The pixel values ​​in the expansion contour map are compared with the preset range of hidden danger feature pixel values ​​to obtain suspected hidden danger areas. The suspected hidden danger areas are then marked to obtain line hidden danger data.

[0029] Based on the line hazard data, obstacle search is performed on the fused feature map to obtain obstacle location data.

[0030] Furthermore, based on the line hazard data, obstacle search is performed on the fused feature map to obtain obstacle location data, including:

[0031] Extract the pixel center coordinates of the line hazard data, and use the pixel center coordinates as the center and a preset search radius to delineate a circular boundary in the fused feature map to obtain the obstacle search area;

[0032] The obstacle search area is segmented into foreground pixels and background pixels according to a grayscale threshold to obtain a foreground segmentation map. Connected regions are marked on the foreground pixels in the foreground segmentation map to obtain the obstacle candidate area.

[0033] The centroid coordinates of the candidate obstacle region are calculated to obtain the obstacle pixel coordinates. The obstacle pixel coordinates are then transformed by combining the flight position parameters of the inspection UAV to obtain the obstacle position data.

[0034] Furthermore, the current flight path of the inspection drone is updated based on the obstacle location data to obtain obstacle avoidance path data, including:

[0035] The obstacle location data is used to calculate the safe distance, and the minimum obstacle avoidance distance is determined according to the size and type of the obstacle to obtain the safe area. The safe area is then gridded to obtain the path planning map.

[0036] The path planning map is sampled, and path nodes are arranged at preset intervals within a safe area to obtain a node distribution map. Connectivity detection is performed on adjacent nodes in the node distribution map to obtain a feasible path.

[0037] The feasible paths are calculated for path cost. The flight distance and turning angle of the current flight path of the inspection drone are comprehensively analyzed to obtain a path score. Based on the path score, the feasible paths are ranked to obtain the optimal path.

[0038] The optimal path is smoothed by rounding off sharp corners and turns to obtain obstacle avoidance path data.

[0039] Furthermore, adjusting the flight attitude parameters of the inspection drone based on the obstacle avoidance path data includes:

[0040] The obstacle avoidance path data is decomposed into target attitude values ​​in three directions: heading angle, pitch angle, and roll angle, to obtain an attitude command set. The attitude command set is then divided into time sequences to obtain an action sequence table including attitude adjustment order and transition time.

[0041] Based on the action sequence list, the difference of the current attitude of the UAV is calculated to obtain the attitude compensation value, and the power is allocated to the attitude compensation value to generate flight attitude parameters.

[0042] This invention also provides a route inspection system based on UAV visual navigation, comprising:

[0043] The enhancement module is used to enhance the images of power equipment and lines collected by the inspection drone in the inspection area, so as to obtain enhanced images of the equipment and lines.

[0044] The extraction module is used to perform multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map, and to perform feature fusion on the equipment line feature map to obtain a fused feature map;

[0045] The analysis module is used to perform target detection analysis on the fused feature map to obtain line hazard data and obstacle location data, and to annotate the line hazard data to obtain inspection report data;

[0046] The update module is used to update the current flight path of the inspection drone based on the obstacle location data to obtain obstacle avoidance path data, and adjust the flight attitude parameters of the inspection drone according to the obstacle avoidance path data.

[0047] This invention provides a line inspection method based on UAV visual navigation, comprising the following steps: enhancing images of power equipment lines collected by the inspection UAV in the inspection area to obtain enhanced equipment line images; extracting multi-scale features from the enhanced equipment line images to obtain equipment line feature maps, and fusing features from the equipment line feature maps to obtain fused feature maps; performing target detection analysis on the fused feature maps to obtain line hazard data and obstacle location data, and annotating the line hazard data to obtain inspection report data; updating the current flight path of the inspection UAV based on the obstacle location data to obtain obstacle avoidance path data, and adjusting the flight attitude parameters of the inspection UAV according to the obstacle avoidance path data. This method solves the technical problems that manual inspection can no longer meet the growing safety monitoring needs, and that inspection cycles are long and coverage is limited. It achieves simultaneous hazard identification and obstacle location, improving the overall efficiency of inspection operations. By automatically generating fully annotated inspection report data, the tedious manual sorting and recording steps required in traditional methods are eliminated. Inspection personnel only need to review and confirm the system output results, significantly reducing their workload. The structured presentation of the report data also facilitates subsequent statistical analysis and trend judgment, providing data support for preventive maintenance of power equipment. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the steps of a route inspection method based on UAV visual navigation in one embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a line inspection system based on UAV visual navigation in one embodiment of the present invention;

[0051] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] The following describes in detail, with reference to the accompanying drawings, a method for line inspection based on UAV visual navigation according to an embodiment of the present invention.

[0054] Figure 1 This invention provides a method for route inspection based on UAV visual navigation, comprising the following steps:

[0055] Step S1: Enhance the images of power equipment lines collected by the inspection drone in the inspection area to obtain enhanced images of the equipment lines; wherein, the inspection area includes power equipment and the power lines connecting the power equipment.

[0056] Specifically, the high-definition cameras carried by the inspection drones continuously photograph the power equipment and power lines within the inspection area. Due to the complex and variable on-site environment, the original images often suffer from underexposure or overexposure. To address this issue, the system first uses a histogram equalization algorithm to remap the grayscale distribution of the image, dispersing pixel values ​​that were originally concentrated in a certain range across the entire value range, thus making details in dark areas clearer. For example, when the drone photographs an insulator string under backlight conditions, the cracks are almost invisible in the shadow areas of the original image. After equalization processing, these subtle textures become visible. Simultaneously, considering that smog can cause images to appear washed out and blurry, the system also employs a dark channel dehazing method. By calculating the minimum color channel value of a local area of ​​the image, it estimates the atmospheric light intensity and transmittance, thereby restoring a clear scene. This ensures that even images of conductors and towers taken in low visibility conditions can present a relatively ideal visual effect after processing, creating a basic condition for subsequent analysis.

[0057] Step S2: Perform multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map, and perform feature fusion on the equipment line feature map to obtain a fused feature map.

[0058] Specifically, after obtaining the enhanced image of the equipment line, the system feeds it into an improved residual network structure with several convolutional layers of varying depths. The shallow convolutional kernels are set to 3×3, primarily responsible for capturing small rust spots and cracks on the conductor surface, as the shallow network has a small receptive field and is sensitive to local textures. As the network progresses to deeper layers, the kernels gradually increase to 5×5 or even 7×7, at which point the network begins to focus on larger targets such as the overall shape of insulators and the structure of towers. Each set of convolution and pooling operations outputs a feature map of the equipment line. For example, the feature map output from the second layer contains rich edge information, while the feature map output from the fifth layer contains more abstract semantic content. The subsequent fusion process is crucial; the system first enlarges the deep feature maps using deconvolution operations to match the shallow feature maps. Figure 1 The same size is used, and then they are directly stitched together according to the channel dimension, or they are merged using a weighted summation method. The weight coefficients are adjusted based on the detection results on the validation set. For example, in a real-world scenario, when a transformer and a small terminal block appear in the same image, the fused feature map retains the edge details of the terminal block captured in the shallow layer, while also utilizing the overall positional information of the transformer from a deeper understanding. This way, no target of any size will be missed during subsequent target detection.

[0059] Step S3: Perform target detection analysis on the fused feature map to obtain line hazard data and obstacle location data, and label the line hazard data to obtain inspection report data.

[0060] Specifically, the next step is for the object detection network to process the fused feature map. To simultaneously obtain data on line hazards and obstacle locations, a practical approach is to use a detection model with two independent prediction branches. This model's backbone network shares the previously obtained fused features; one branch is specifically responsible for predicting various line hazards, such as insulator damage or broken conductor strands; the other branch focuses on identifying obstacles like trees and cranes. During inference, the network performs a dense sliding scan on the feature map, predicting a bounding box, a class label, and a confidence score for each potential target. Thus, the raw output contains the location and type information of all detected targets. Labeling the line hazard data is not simply a matter of ticking a box. The system automatically captures this raw data and then calls a drawing module to precisely select the location of the hazard on the original inspection image with a prominent red rectangle, adding a text description next to it, such as "Missing equipotential ring - 95% confidence." After visually labeling all potential hazards, the program will package these labeled images, the corresponding hazard type list, geographical location and other metadata into a standardized electronic document. This structured document, which can be directly delivered to maintenance personnel, is the final inspection report data required.

[0061] Step S4: Update the current flight path of the inspection drone based on the obstacle location data to obtain obstacle avoidance path data, and adjust the flight attitude parameters of the inspection drone according to the obstacle avoidance path data.

[0062] Specifically, after obtaining obstacle location data, the flight control system first projects these coordinate points onto a pre-set 3D grid map. Each grid cell is 0.5m x 0.5m x 0.5m in size, and grid cells occupied by obstacles are marked as impassable. Then, an improved A* algorithm is invoked to search for a new path in the grid map. The algorithm starts from the drone's current position and extends towards the next target point, automatically avoiding marked grid cells. For example, if a tree is detected 12 meters ahead, the algorithm calculates two options: shifting 6 meters to the left and then moving forward, or increasing altitude by 4 meters to pass over the tree canopy. By comparing path length and energy consumption, the optimal option is selected. This replanned flight trajectory is the obstacle avoidance path data. After obtaining the obstacle avoidance path data, the flight control module decomposes the trajectory into a series of waypoints, with adjacent waypoints spaced approximately 2 meters apart. Each waypoint corresponds to a set of target attitude parameters, including pitch angle, roll angle, yaw angle, and flight speed. For example, to perform a left turn maneuver, the yaw angle will be adjusted from the current 0 degrees to -15 degrees, while the roll angle will become 8 degrees to coordinate with the turn, and the speed will be reduced from 5 meters per second to 3 meters per second to ensure stability. These parameters will be sent to the motor drive module through the PID controller, and the rotational speed of the four propellers will change accordingly, so that the aircraft can fly according to the obstacle avoidance path data and complete attitude adjustment.

[0063] In a specific embodiment, the step of enhancing the images of power equipment lines collected by the inspection drone in the inspection area to obtain enhanced equipment line images includes:

[0064] The images of power equipment lines collected by the inspection drone in the inspection area are processed by brightness partitioning, and the images of power equipment lines are divided into multiple brightness sub-regions to obtain a brightness distribution map.

[0065] Brightness equalization calculations are performed on each brightness sub-region in the brightness distribution map to obtain an enhanced device line image.

[0066] Specifically, after the power equipment line image is transmitted, the program first converts the image from the RGB color space to the HSV color space, extracting the V channel, or luminance channel, for separate processing. Then, it uses a sliding window method to move along the luminance channel, setting the window size to 64×64 pixels and the step size to 32 pixels each time, ensuring that adjacent windows overlap by half. The area covered by each window is defined as a luminance sub-region. The program calculates the average gray value of all pixels within this sub-region. If the average value is between 0 and 85, it is marked as a dark area; 85 to 170 is marked as a midtone; and 170 to 255 is marked as a bright area. For example, if the upper left corner of the image captures a backlit tower structure, that area has an average gray value of only 52 and will be marked as a dark area; if the lower right corner shows an insulator directly exposed to sunlight, the average gray value can reach 198, and it will be classified as a bright area. After all sub-regions are marked, a luminance distribution map of the same size as the original image is generated. On this map, different locations are labeled with different values ​​to indicate the luminance attributes of the corresponding areas.

[0067] The subsequent brightness equalization calculation is performed independently for each region, with the greatest enhancement applied to dark areas, followed by midtones, and the least to bright areas. Specifically, the program creates a separate histogram for each brightness sub-region, with the horizontal axis representing gray levels from 0 to 255 and the vertical axis representing the number of pixels with that gray level. Taking the aforementioned dark area as an example, its histogram shows that most pixels are concentrated in the narrow range of 30 to 70. At this point, the cumulative distribution function is used to remap these pixel values ​​to a wider range. The mapping formula is that the new gray level equals 255 multiplied by the cumulative probability of the current gray level. After this transformation, pixels that were originally clustered around 50 will be dispersed to the range of 50 to 120, and the details in the dark areas will be revealed. For bright areas, the opposite strategy is used, appropriately compressing the distribution range of high gray levels to avoid overexposure, while standard histogram equalization is sufficient for midtone areas. After all sub-regions have been processed, the program will perform bilinear interpolation at the overlapping boundaries to smooth the transition and prevent obvious block marks. Finally, the processed luminance channel is merged with the original H and S channels and converted back to RGB space, thus obtaining an enhanced device line image with a more reasonable overall luminance distribution.

[0068] In a specific embodiment, the step of performing multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map includes:

[0069] The enhanced equipment circuit image is divided into multiple scales to obtain a multi-layer image group;

[0070] Edge contours are extracted for each scale layer in the multi-layer image group to identify the edge information of power equipment at different scales, resulting in a multi-scale contour map. Geometric feature statistics are then performed on the multi-scale contour map to obtain a feature distribution map.

[0071] The feature distribution map is divided into multiple feature sub-regions to obtain a regional feature map. Spatial relationship calculations are then performed on the feature sub-regions in the regional feature map to obtain a device circuit feature map.

[0072] Specifically, after the enhanced equipment circuit image is sent in, it first undergoes a Gaussian pyramid transformation to create a multi-layer image group. The original image is used as layer 0, maintaining a resolution of 1920×1080. Then, the original image is downsampled by a factor of 2 to generate layer 1, which becomes 960×540. Layer 1 is then downsampled again to obtain layer 2, which is 480×270, and so on, until layer 4, forming a total of 5 layers with different resolutions. Before downsampling, each image layer is smoothed using a 5×5 Gaussian kernel convolution to avoid aliasing. The advantage of this multi-layer image group is that the coarse bottom layer shows the overall structure of the tower, while the fine top layer preserves small details such as bolts and cracks.

[0073] After obtaining the multi-layer image set, the Canny edge detection operator is run on each layer, but the parameters used for different layers are slightly different. Layers 0 and 1 process high-resolution images, with the Canny operator's low threshold set to 50 and high threshold set to 150, which can detect very fine rust edges on the surface of the conductors; for low-resolution layers such as layers 3 and 4, the low threshold is increased to 80 and the high threshold to 200, mainly capturing obvious structural boundaries such as transformer casings and crossarms. After each layer of the image is processed by Canny, a binary edge map is output, with white pixels representing detected edge points and black pixels representing the background. The edge maps of the 5 layers are aligned and superimposed according to the original image size to form a multi-scale contour map, which contains both the coarse outline of large objects and the fine edges of small objects. Then the program traverses each continuous edge on the multi-scale contour map and calculates its geometric parameters, including contour length, bounding box area, aspect ratio, and roundness. For example, if a long, thin outline of 128 pixels with a bounding box of 15×95 and an aspect ratio of 6.3 is detected, it can be basically determined that this is the edge of a conductor; if an outline of 586 pixels in length, a bounding box of 180×210, and a circularity of 0.68 is found, it is most likely the outer outline of an insulator. These geometric parameters are encoded into feature vectors and stored in the corresponding spatial locations to form a feature distribution map. Each pixel in the map, in addition to brightness information, also carries the geometric statistics of the outline to which it belongs.

[0074] After the feature distribution map is generated, the watershed algorithm is used for region segmentation. The algorithm treats the similarity between feature vectors as height values; areas with clustered similar features are low-lying regions, while areas with significant differences are watersheds. The segmentation process starts from the seed point with the smallest gradient and expands outwards, stopping at boundaries where gradients change abruptly. This ensures that feature points belonging to the same device component are grouped into a single feature sub-region. For example, an insulator string might appear as a continuous region on the feature distribution map, with similar contour lengths, roundness, and other parameters. The watershed algorithm will identify this as a separate piece. Adjacent crossarms, due to their completely different aspect ratios and bounding box dimensions, will be grouped into another sub-region. After all sub-regions are labeled, the region feature map is obtained, where each connected component represents a potential device component.

[0075] Finally, in the spatial relationship calculation stage, the program reads the centroid coordinates of each feature sub-region in the regional feature map and establishes a spatial adjacency matrix. The element in the i-th row and j-th column of the matrix records the center distance and relative azimuth angle between the i-th and j-th sub-regions. For example, if sub-region 3 is found to be 2.3 meters directly above sub-region 5, this positional relationship conforms to the typical structure of an insulator suspended below a crossarm, and a high correlation weight is assigned to this pair of sub-regions. Simultaneously, the topological connectivity between sub-regions is calculated; if the boundaries of two regions are in direct contact, the connection flag is set to 1. By comprehensively considering the three dimensions of distance, azimuth, and topological connectivity, each feature sub-region obtains a relationship vector describing its surrounding environment. These relationship vectors are concatenated with the previously extracted geometric features to ultimately form the equipment line feature map. This feature map contains both the morphological information of each equipment component and records the spatial layout relationships between them.

[0076] In a specific embodiment, spatial relationship calculations are performed on the feature sub-regions in the region feature map to obtain a device line feature map, including:

[0077] Centroid coordinates are extracted from each feature sub-region in the region feature map to obtain a set of equipment center points. Equipment type is labeled for each center point in the set of equipment center points to obtain an equipment location label map including tower location points, insulator location points and conductor connection points.

[0078] Based on the equipment location marking map, the Euclidean distance between adjacent power equipment is calculated to obtain an equipment spacing table. The electrical connection relationship between power equipment is determined according to the equipment spacing table to obtain an equipment association matrix. The equipment association matrix records the connection status between towers and insulators, and between insulators and conductors.

[0079] Based on the device association matrix, topological connection segments between devices are drawn on the regional feature map to obtain a topological connection map. The topological connection map is then overlaid with the device location annotation map to obtain a line feature map.

[0080] Specifically, each feature sub-region on the region feature map is a connected set of pixels. The program first traverses these connected regions and calculates the arithmetic mean of the coordinates of all pixels within each region. The point formed by the mean of the x-coordinate and the mean of the y-coordinate is the centroid of that region. If a feature sub-region contains pixels in the range of (520, 340) to (580, 410), the calculated centroid is approximately around (550, 375), and this coordinate is recorded in the device center point set. Dozens of such centroid points are extracted from the entire image, each corresponding to a potential device component.

[0081] After obtaining the set of equipment center points, the original feature sub-regions corresponding to each center point are analyzed one by one, and the equipment type is determined through morphological parameters. The feature sub-regions of poles and towers are usually large and vertically elongated rectangles with an aspect ratio exceeding 3. If a region with an area exceeding 15,000 pixels and an aspect ratio of 3.8 is detected, it is basically confirmed to be a pole or tower structure, and its centroid is marked as the pole or tower location point. The feature sub-regions of insulators are relatively small, with a circularity generally between 0.6 and 0.8. When a region with an area of ​​2,800 pixels and a circularity of 0.71 is found, its centroid is marked as the insulator location point. Conductors are special; their feature sub-regions are extremely thin and long, with an aspect ratio often exceeding 10, and although the region area is not small, the width is only a few to a dozen pixels. In such cases, they are marked as conductor connection points. Once all center points are marked, an equipment location marking map will be generated. This map uses different colors or symbols to indicate the centroid positions of various types of equipment. For example, blue dots represent tower locations, green triangles represent insulator locations, and red short lines represent conductor connection points.

[0082] Next, the distance between the devices needs to be calculated. The program will randomly select two center points from the device location map, read their coordinates and record them as (x1, y1) and (x2, y2) respectively, and then apply the Euclidean distance formula to calculate the distance between the two points, which is the square root of (x1-x2)² plus the square root of (y1-y2). For example, in a real-world scenario, a tower is located at (450, 280), and an insulator is located nearby at (520, 380). The calculated Euclidean distance is approximately 116 pixels. Considering the conversion ratio between image resolution and actual shooting distance, this 116 pixels corresponds to approximately 2.3 meters on-site. After calculating the distances between all device pairs, the distances will be entered into a device spacing table. The table is a two-dimensional matrix, with rows and columns corresponding to different device center points, and the matrix elements storing the distance values.

[0083] Using the equipment spacing table, electrical connections can be inferred because some equipment in actual lines must be connected within a specific distance range. For example, the typical spacing between an insulator and a tower is 1.5 to 3.5 meters. If the equipment spacing table shows a distance of 2.1 meters between a tower and an insulator, falling within this reasonable range, it can be preliminarily determined that the two are connected. However, simply looking at the distance is not enough; the relative position must also be checked. The insulator should be diagonally below or directly below the tower. This can be verified by comparing the y-coordinates of the two points. The determination between conductors and insulators is similar. The distance from the conductor connection point to the insulator location is usually 0.8 to 2 meters, and the conductor should be at the lower end of the insulator. Only when these conditions are met is the corresponding position marked as 1 in the equipment association matrix to indicate a connection. The equipment association matrix is ​​a sparse matrix, with most elements being 0. Only equipment pairs with a confirmed connection are filled with 1. For example, a 1 in the 3rd row and 7th column of the matrix indicates that equipment 3 and equipment 7 are electrically connected.

[0084] Finally, the topology is drawn. The program scans all positions with a value of 1 in the device association matrix, finds the coordinates of the corresponding two device center points, and then connects these two points with a straight line segment on the region feature map. For example, if the association matrix shows a connection between the tower location (450, 280) and the insulator location (520, 380), a line segment will be drawn from (450, 280) to (520, 380). The line segment color can be set to cyan for easy distinction. After all connections are drawn, a topology connection diagram is formed, which clearly shows the physical connection paths between various device components. Then, the topology connection diagram and the previous device location annotation diagram are overlaid. The overlay operation is a bitwise OR operation between the two images. The device location annotation diagram provides the location markers for various devices, and the topology connection diagram adds the connecting line segments. The merged image shows both where each device is located and how they are connected. This is the final device line feature map, containing complete device spatial layout and topology connection information.

[0085] In a specific embodiment, feature fusion is performed on the device circuit feature map to obtain a fused feature map, including:

[0086] The importance of the features in the equipment line feature map is evaluated, and an importance score is obtained by analyzing the criticality of different feature sub-regions in power line inspection.

[0087] The equipment line feature map is weighted based on the importance score to obtain a weighted feature map. Feature overlay calculation is then performed on the weighted feature map to add features at the same location at different scales to obtain a preliminary fused feature map.

[0088] Redundant features with high correlation and little impact on the inspection results are removed from the preliminary fusion feature map to obtain the fusion feature map.

[0089] Specifically, after obtaining the equipment line feature map, an importance analysis is performed on each feature sub-region. This analysis mainly considers three indicators. The first is the equipment failure frequency, which is calculated from the historical inspection database based on the probability of problems with various types of equipment. Insulators, due to long-term exposure, are prone to damage and have a failure rate of up to 12%. The probability of broken conductor strands is about 6%, while the failure rate of the relatively robust tower body is only about 2%. The second indicator is the degree of hazard. Broken conductor strands, once they occur, directly cause a power outage, and the hazard level is set to the highest level, level 5. Insulator damage will cause discharge but will not immediately interrupt the power supply, so it is level 4. Localized corrosion of the tower has little impact in the short term and is set to level 2. The third is the detection difficulty coefficient. Small cracks and rust spots are inconspicuous on the image and are easily missed, so the detection difficulty coefficient is set to 3.5. The detection difficulty of obvious anomalies such as deformation of large equipment is only 1.2. The program will calculate the comprehensive score of each feature sub-region by weighting and summing these three indicators. The weight ratio is tentatively set as 40% for failure frequency, 35% for hazard level, and 25% for detection difficulty. Taking the characteristic sub-region related to conductors as an example, a failure rate of 6% multiplied by 0.4 equals 0.024, a hazard level of 5 multiplied by 0.35 equals 1.75, and a detection difficulty of 3.5 multiplied by 0.25 equals 0.875. The combined score of these three factors is 2.649. After normalization and mapping to the 0-1 range, this value becomes 0.88, which is the importance score of this characteristic sub-region. The score calculated for the insulator region is approximately 0.76, the tower region's score is 0.43, and the background region is almost zero.

[0090] Next, weights are assigned based on importance scores. The program reads the feature sub-region number to which each pixel belongs in the device line feature map, then looks up the corresponding score from the previously calculated scoring table, and directly uses this score as the weight coefficient for that pixel. For example, the point at coordinates (680, 420) belongs to feature sub-region number 7. Looking up the table, region number 7 has a score of 0.88, so the weight of this pixel is set to 0.88. After assigning weights to each pixel in the entire image, a weighted feature map is generated. From a data structure perspective, this image has the same size as the original image, but each pixel has an additional weight value in addition to the original feature vector. Obtaining the weighted feature map is not enough. Don't forget that several layers of feature maps with different resolutions were generated during the multi-scale extraction. Now, these layers need to be merged.

[0091] The fusion operation first aligns the dimensions by enlarging the coarse, low-resolution feature maps to the same size as the finest layer using bilinear interpolation. For example, if the top-level feature map is 1920×1080 and the third layer was originally only 480×270, it will also become 1920×1080 after interpolation. Then, the feature maps are stacked pixel by pixel; specifically, the feature vectors from different layers at the same coordinate position are added together according to their corresponding weights. For example, at coordinates (800, 600), the eigenvector of the top-level feature map is [0.65, 0.32, 0.18] with a weight of 0.88, the second-level feature vector is [0.41, 0.58, 0.23] with a weight of 0.76, and the third-level feature vector is [0.29, 0.44, 0.51] with a weight of 0.43. The weighted sum is [0.65×0.88+0.41×0.76+0.29×0.43, 0.32×0.88+0.58×0.76+0.44×0.43, 0.18×0.88+0.23×0.76+0.51×0.43]. The calculated new eigenvector is approximately [1.01, 0.91, 0.56]. Once all pixels have been processed in this way, a preliminary fused feature map is obtained. This map integrates information from multiple scales, preserving details while taking into account the overall structure.

[0092] However, the initial fused feature map contains a lot of redundant content that needs to be cleaned up. The program first calculates the Pearson correlation coefficient between each dimension of the feature vector. Assuming the feature vector has 32 dimensions, a 32×32 correlation coefficient matrix will be generated. The element in the i-th row and j-th column of the matrix represents the correlation between the i-th and j-th features in the entire image. If the correlation coefficient between the 5th and 12th dimensions is found to be as high as 0.94, it means that the information carried by these two dimensions is highly redundant, and it is enough to keep only one. Next, it is necessary to determine which of these two dimensions has a greater impact on the inspection results. The method is to remove one dimension at a time and run the target detection again to see how much the accuracy of hazard identification decreases. The test found that after removing the 5th dimension, the accuracy dropped from 89.2% to 87.6%, while after removing the 12th dimension, it only dropped to 88.8%. Obviously, the 5th dimension is more important, so the 12th dimension is deleted. A similar filtering process will traverse all highly correlated dimension pairs, and finally the 32-dimensional feature vector will be reduced to about 18 dimensions. Another type of redundancy comes from spatial distribution. If two feature sub-regions are adjacent in an image and their feature vectors are almost identical, it's due to over-segmentation. In this case, they are merged into one region. For example, if two adjacent regions both correspond to the edge features of a conductor and have a feature vector similarity of 0.96, merging them reduces the amount of data and avoids redundant calculations. Through these elimination and merging operations, the redundant components in the initial fused feature map are significantly reduced, leaving only the key features that are truly useful for the inspection task. The final fused feature map has a data volume that is compressed by about 35% compared to the initial version, but contains a higher effective information density. This improves speed without sacrificing accuracy when performing subsequent object detection. The entire fusion process is essentially about finding a balance between information integrity and computational efficiency. It highlights key regions through importance assessment, uses multi-scale overlay to balance details and the overall picture, and controls the data scale through redundancy elimination. The final fused feature map is a high-quality feature representation that has been carefully selected and integrated.

[0093] In a specific embodiment, the step of performing target detection analysis on the fused feature map to obtain line hazard data and obstacle location data includes:

[0094] The fused feature map is divided into multiple equipment area maps according to the distribution of power equipment in the inspection area, and the area division results are obtained. The edge pixel statistics of each equipment area map in the area division results are obtained to obtain the edge pixel distribution.

[0095] Based on the edge pixel distribution, the device region map is contour extracted to identify the outline of the power equipment in the device region map, and the outline of the power equipment is morphologically dilated to obtain a dilated outline map.

[0096] The pixel values ​​in the expansion contour map are compared with the preset range of hidden danger feature pixel values ​​to obtain suspected hidden danger areas. The suspected hidden danger areas are then marked to obtain line hidden danger data.

[0097] Based on the line hazard data, obstacle search is performed on the fused feature map to obtain obstacle location data.

[0098] Specifically, after the fused feature map is input, it is first segmented into regions based on the previously marked device location information. The program reads the bounding rectangle coordinates of each device in the device location annotation map, and then extracts the corresponding image blocks from the fused feature map according to these coordinate ranges. For example, if the annotation map shows that a certain tower occupies a rectangular area from (320, 150) to (580, 720), this 260×570 pixel range is extracted from the fused feature map to form a device region map. The same operation is performed on components such as conductors and insulators, with each device forming a separate image. To avoid information loss due to the boundaries being too close to the devices, the extraction is extended outward by 15 to 20 pixels as a buffer. After all devices are separated into independent image blocks, the region segmentation result is obtained, containing a dozen to dozens of device region maps.

[0099] After obtaining each device region map, the program analyzes the distribution of edge pixels. First, it uses the Sobel operator to perform convolutions in both the horizontal and vertical directions. The horizontal convolution kernel is [-1, 0, 1; -2, 0, 2; -1, 0, 1], and the vertical kernel is transposed. Then, it calculates the gradient magnitude at each pixel location. Pixels with gradient magnitudes exceeding a certain threshold, such as 80, are marked as edge points. The program then analyzes the spatial distribution of these edge points across the entire device region map. For example, if an insulator's device region map is 180×240 pixels, Sobel processing reveals 4823 pixels with gradients exceeding the threshold. These edge pixels are mainly concentrated at the junction of the insulator's outer contour and inner skirts, with particularly high density in the upper and middle parts of the region map and sparser at the bottom. This distribution characteristic is recorded to form edge pixel distribution data, including parameters such as the total number of edge points, the coordinate range of dense areas, and the average spacing.

[0100] When extracting a complete contour based on edge pixel distribution, a chain code tracking algorithm is used. The program starts from the position with the highest edge pixel density, finds an edge point as the starting point, and then searches for the next edge point in an 8-neighborhood in a clockwise direction. For each point found, a direction code relative to the previous point is recorded, with 0 to 7 representing the eight directions: right, upper right, up, upper left, left, lower left, down, and lower right, respectively. During tracking, if three or more consecutive points extend in the same direction, the segment can be considered a straight line; if the direction code changes frequently, it indicates a curve. For example, when tracking the outer contour of an insulator, the starting point is at (90, 120), the next point is at (91, 120) with a direction code of 0, the next point is at (92, 121) with a direction code of 1, then (92, 122) with a code of 2, and so on until returning to the starting point to form a closed contour. The entire contour sequence contains more than 500 points and their corresponding direction codes. After extracting all the closed contours, the outlines of the power equipment in the equipment area map are identified. There are multiple contours in a single image. The main contour corresponds to the shape of the equipment, while the smaller internal contours correspond to the parts or texture details.

[0101] After contour extraction, morphological dilation is performed to thicken the contour lines and connect minor gaps or breaks. The structuring element used in the dilation operation is a 3×3 or 5×5 square. The program iterates through each point on the contour, placing the structuring element around that point. If any pixel within the structuring element's coverage area belongs to the contour, the center point is also marked as a contour point. For example, originally there was no contour point at position (150, 200), but the surrounding 3×3 area (151, 201) is a contour point. After dilation, (150, 200) will also be included in the contour. After 2 to 3 dilation iterations, the original 1-pixel-wide contour line will expand to 5 to 7 pixels wide, forming a dilated contour map. In this map, the device boundaries become more continuous and full, and some contour breaks caused by uneven lighting or noise are repaired.

[0102] After obtaining the expansion contour map, the program begins locating potential hazards. It reads preset ranges of hazard characteristic pixel values, calculated based on a large number of historical hazard samples. For example, pixel values ​​in damaged insulator areas are typically between 35 and 65, as shadows appear at the damaged area, resulting in lower brightness, while the edge gradient abnormally increases to over 150. Conductor corrosion is characterized by significantly higher R-channel values ​​than G and B channels, with typical pixel values ​​being [158, 82, 71], a reddish-orange hue. The program iterates through each pixel within the expansion contour map, checking its brightness, color components, and gradient magnitude to see if they fall within a specific hazard characteristic range. If a continuous area of ​​over 30 pixels is found at position (95, 186) in the insulator area map, with brightness values ​​between 41 and 58 and an edge gradient reaching 167, it is preliminarily identified as a potential hazard area. After marking all eligible areas, small noise spots are filtered out, retaining connected regions larger than 20 pixels; these regions are considered genuine hazards.

[0103] When marking the location of suspected hazard areas, the local coordinates must be converted back to full-map coordinates because the equipment area map is a small patch extracted from the fused feature map, and an offset needs to be added back. For example, if the top left corner of an insulator equipment area map is at (820, 340) on the full map, and a hazard is found at (95, 186) within that area map, then the full-map coordinates would be (820 + 95, 340 + 186), which equals (915, 526). Each hazard is also appended with a category label, confidence level, and the corresponding equipment number, ultimately generating line hazard data. The data structure is as follows: Hazard Number 001, Type: Insulator Damage, Coordinates: (915, 526), ​​Confidence Level: 0.87, Equipment Number: INS-23, Discovery Time: Current Timestamp. All hazard data is packaged into a list or database record for use in subsequent report generation.

[0104] Obstacle search is relatively independent. The program searches the fused feature map for large objects that are not electrical equipment. During the search, the identified equipment areas are first masked off on the map. The remaining areas, if showing continuous gradient changes and texture features, are considered obstacles such as trees and buildings. Specifically, connected component analysis is performed in the non-equipment areas. Connected components with an area exceeding 5000 pixels are extracted, and their shape features are calculated. Trees are typically irregular clumps with an aspect ratio close to 1 and high texture complexity, while buildings have regular rectangular outlines with straight edges. Each candidate target is labeled based on these morphological features, and its centroid coordinates and circumscribed rectangle dimensions are recorded. This information constitutes obstacle location data, with a format similar to: Obstacle ID OBS-05, Type: Tree, Centroid (1250, 680), Circumscribed Rectangle Width 180 Height 220, Distance from nearest equipment tower #34: approximately 12 meters. With this obstacle location data, the subsequent path planning module can avoid these areas in advance, ensuring the drone's flight safety.

[0105] In a specific embodiment, obstacle search is performed on the fused feature map based on the line hazard data to obtain obstacle location data, including:

[0106] Extract the pixel center coordinates of the line hazard data, and use the pixel center coordinates as the center and a preset search radius to delineate a circular boundary in the fused feature map to obtain the obstacle search area;

[0107] The obstacle search area is segmented into foreground pixels and background pixels according to a grayscale threshold to obtain a foreground segmentation map. Connected regions are marked on the foreground pixels in the foreground segmentation map to obtain the obstacle candidate area.

[0108] The centroid coordinates of the candidate obstacle region are calculated to obtain the obstacle pixel coordinates. The obstacle pixel coordinates are then transformed by combining the flight position parameters of the inspection UAV to obtain the obstacle position data.

[0109] Specifically, the line hazard data records the boundary coordinates of each hazard area. The program first iterates through these hazard areas, calculating the arithmetic mean of all pixel coordinates in each area to determine the center location. For example, a damaged insulator area contains pixels ranging from (912, 523) to (928, 541). The average horizontal coordinate is approximately 920, and the average vertical coordinate is 532. This (920, 532) is the pixel center coordinate of the hazard. After obtaining this center coordinate, the search area needs to be drawn on the fused feature map. The preset search radius is usually set to 150 to 200 pixels. This value is determined based on the actual situation on site. Too small a radius may miss obstacles, while too large a radius will increase invalid calculations. A circle with (920, 532) as the center and a radius of 180 pixels is drawn. The boundary equation of the circle is (x-920)² + (y-532)² = 180². The program will iterate through all pixels on the fused feature map, marking the points that satisfy this equation to form a circular outline. The area inside the outline is the obstacle search area. In practice, it is not necessary to actually draw a circle. Just calculate the distance of each pixel to the center point. Pixels less than or equal to 180 are included in the search area. In this way, a potential problem point will be surrounded by more than 100,000 pixels to be searched.

[0110] After the obstacle search area is determined, foreground segmentation begins. This step separates obstacles such as trees and hanging objects from the background of the sky and the tower. The program first converts the color image within the search area to grayscale. The conversion formula is: grayscale value equals 0.299 times the R channel, 0.587 times the G channel, and 0.114 times the B channel. This weighting is set based on the human eye's sensitivity to different colors. After conversion, each pixel has only a grayscale value between 0 and 255. Then, a threshold is set to divide the pixels into two categories. The threshold is usually calculated automatically using Otsu's method. Otsu's method tries every value between 0 and 255 and calculates the inter-group variance after dividing the pixels into two groups with this value as the boundary. The value with the largest variance is the optimal threshold. If the calculated threshold is 132, then pixels with a grayscale value greater than 132 in the search area are marked as foreground pixels, and those less than or equal to 132 are background pixels. Foreground pixels generally correspond to targets with higher brightness or complex textures, while background pixels are mostly the sky or a uniform surface. After all pixels are marked, a foreground segmentation map is generated. In this map, the foreground area is displayed in white and the background area is black, making it easy to see where obstacles exist.

[0111] White pixels in the foreground segmentation map are often not connected into a single area, but scattered across several locations, requiring connectivity analysis to classify them. The program employs a two-pass scanning algorithm. The first pass scans line by line from the top left corner. When a white pixel is encountered, its left and top pixels are checked. If the left or top pixel is also white and already labeled, the current pixel is labeled with the same tag; otherwise, a new tag is assigned. During the scan, if a pixel is found to have left and top pixels belonging to different tags, it indicates that these two tags are actually connected, and their equivalence relationship is recorded. After the first pass, a set of temporary tags and their equivalence pairs are obtained. During the second pass, these temporary tags are merged into the final tags based on the equivalence relationships. For example, if there are three white pixel clusters in the search area, the first pass labels them 1, 2, and 3 respectively. Later, it is found that tags 1 and 2 are actually connected. In the second pass, all pixels labeled 2 are changed to tag 1, leaving only two independent connected regions labeled 1 and 3. Each connected region represents a candidate obstacle region. The program counts the number of pixels contained in each region. Small spots with an area of ​​less than 500 pixels are directly filtered out as noise, and only the real obstacle candidates are retained.

[0112] For each obstacle candidate region, its centroid position needs to be calculated. The method is similar to finding the coordinates of the hazard center: sum the x-coordinates of all white pixels in the region and divide by the total number of pixels to obtain the x-coordinate of the centroid; the y-coordinate is calculated similarly. Assuming a candidate region contains 6823 pixels, the sum of their x-coordinates is 6458920, and the sum of their y-coordinates is 3625184. Therefore, the centroid coordinates are (6458920 / 6823, 3625184 / 6823), approximately (946, 531). These coordinates represent the center position of the obstacle in the pixel coordinate system of the fused feature map and are denoted as the obstacle pixel coordinates. However, pixel coordinates are not very useful for subsequent path planning and need to be converted to actual spatial coordinates.

[0113] Coordinate system transformation requires the UAV's flight position parameters, including current GPS latitude and longitude, altitude, camera pitch angle, and focal length. The program first projects the pixel coordinates onto a normalized image plane based on the camera's intrinsic parameter matrix. This matrix includes focal lengths fx and fy, principal point coordinates cx and cy, and normalized coordinates x' equal to (x-cx) / fx, y' equal to (y-cy) / fy. Taking the previous example, assuming the camera focal length fx = 1800 pixels, fy = 1800 pixels, and the principal point is at (960, 540), then the normalized coordinates corresponding to (946, 531) are x' = (946-960) / 1800 approximately -0.0078, and y' = (531-540) / 1800 approximately -0.0050. Then, a rotation matrix is ​​constructed based on the camera's pitch angle. For example, a pitch angle of -25 degrees indicates the camera is tilted downwards by 25 degrees. The rotation matrix is ​​a 3×3 matrix, and its specific values ​​involve trigonometric calculations. Multiplying the normalized coordinates by the rotation matrix yields the direction vector in the camera coordinate system. Then, based on the drone's altitude and ground elevation data, the intersection point of the ray and the ground is calculated. This intersection point is the obstacle's position in the ground coordinate system. Finally, the drone's current GPS coordinates are added as an offset to obtain the obstacle's absolute latitude and longitude coordinates, such as 31.2456°N, 121.5382°E, and 125 meters above sea level. This set of latitude, longitude, and altitude data constitutes the obstacle's position data and is packaged into a structured record with a format similar to: Obstacle ID OBS-12, Type: Tree, Pixel Coordinates (946, 531), Geographic Coordinates (31.2456°N, 121.5382°E, 125m), Horizontal Distance from Drone: Approximately 15.3 meters, Azimuth: 23 degrees East of North. After all candidate obstacle areas are processed through this process, an obstacle location dataset containing several records is generated. Each record describes the spatial location and basic attributes of an obstacle in detail. This data is directly sent to the flight control system for path planning and obstacle avoidance decisions, ensuring that the drone can avoid these dangerous areas in advance during subsequent inspections.

[0114] In a specific embodiment, the current flight path of the inspection drone is updated based on the obstacle location data to obtain obstacle avoidance path data, including:

[0115] The obstacle location data is used to calculate the safe distance, and the minimum obstacle avoidance distance is determined according to the size and type of the obstacle to obtain the safe area. The safe area is then gridded to obtain the path planning map.

[0116] The path planning map is sampled, and path nodes are arranged at preset intervals within a safe area to obtain a node distribution map. Connectivity detection is performed on adjacent nodes in the node distribution map to obtain a feasible path.

[0117] The feasible paths are calculated for path cost. The flight distance and turning angle of the current flight path of the inspection drone are comprehensively analyzed to obtain a path score. Based on the path score, the feasible paths are ranked to obtain the optimal path.

[0118] The optimal path is smoothed by rounding off sharp corners and turns to obtain obstacle avoidance path data.

[0119] Specifically, after acquiring obstacle location data, the program first extracts the geometric parameters and classification information of each obstacle to calculate the safety distance. For slender obstacles like utility poles, the distance is calculated by adding 2 meters to the diameter; for high-voltage lines, an additional 4 meters is added due to sag; and for trees, the distance is calculated by adding 5 meters to the outer diameter to account for crown sway. For example, if a high-voltage line has a measured diameter of 0.8 meters and a maximum sag of 1.2 meters, the safety distance is 0.8 + 4 = 4.8 meters. A cylindrical no-fly zone with a radius of 4.8 meters is then generated with the cable's centerline as the axis. After all obstacles are processed, a series of no-fly zones are marked in space; areas outside these zones where flight is permitted are called safe zones. Next, this safe zone is divided into a cubic grid at 1.5-meter intervals. Each grid is labeled with a Boolean value: cells falling within the no-fly zone are marked as 0, and cells that can fly are marked as 1. Finally, a three-dimensional array is formed to store these 0s and 1s; this is the path planning map.

[0120] Once the path planning map is in place, sampling points need to be added. The program scans the entire 3D array, placing a path node every 3 to 4 cells with a value of 1. The node position is taken as the geometric center of the corresponding cell. For example, if cell (15, 22, 9) is marked as 1, its actual coordinates are (22.5 meters, 33 meters, 13.5 meters high), then a node with the number N126 is created there. Hundreds of nodes may be scattered throughout the space. The node numbers and coordinates are recorded to form a node distribution map, with a data format similar to a point set, where each record contains a node ID and three coordinate values ​​(X, Y, Z). Then, it's necessary to determine which nodes can be directly connected. This is done by taking two nodes, such as N126 and N134, drawing a straight line between them, and taking sampling points every 0.3 meters along the line. The corresponding grid cells for these sampling points are checked to see if they are all marked as 1. If a sampling point (25.8, 35.2, 14.1) has a corresponding grid cell value of 0, it means the straight line will pass through an obstacle, and the two nodes are not connected. Only when all sampling points are in the safe grid can an edge be drawn between these two points on the node distribution map to indicate that a direct flight is possible. After all node pairs have been checked, nodes connected by edges form a feasible path network.

[0121] Once the feasible path network is obtained, the cost of each route needs to be calculated. This requires retrieving the parameters of the UAV's current flight path as a benchmark. The current flight path stores the original inspection route. The program reads the total flight distance of this original route, assuming it's 128.5 meters, and then counts the number of turning points and the turning angles. For example, the original route has 5 turning points with angles of 35 degrees, 28 degrees, 42 degrees, 31 degrees, and 38 degrees. Now, to evaluate the deviation of each alternative path in the feasible path network from the original route, Dijkstra's algorithm is used to search from the starting point to the target point, resulting in an alternative path: starting point - N134 - N168 - N205 - target point. The total length is calculated by adding up the distances of each segment: 94.3 meters, which is 34.2 meters shorter than the original route's 128.5 meters. Then, the turning angle of this alternative route is calculated. A turn is considered valid if the angle between two adjacent segments exceeds 25 degrees. The angle between segments N134 to N168 and N168 to N205 is 46 degrees, and the angle between segments N168 to N205 and N205 to the target point is 39 degrees, for a total of two turning points. Next, the flight distance and turning angle are analyzed together. In terms of distance, 94.3 minus 128.5 gives a distance saving of 34.2 meters, which is a positive gain. 1.5 points are added for each meter saved. In terms of turning, the number of turns on the original route is 5, minus the number of turns on the alternative route is 2, resulting in 3 fewer turns. However, the turning angle of the new route is larger, with an average turning angle of (46+39) / 2 equaling 42.5 degrees, which is 7.7 degrees higher than the average turning angle of the original route (35+28+42+31+38) / 5 equaling 34.8 degrees. Sharp turns increase flight risk, and 0.5 points are deducted for each degree increase. The path score is calculated as follows: a base score of 1000, plus distance savings multiplied by 1.5, plus the reduction in turning times multiplied by 10, minus the angle increment multiplied by 0.5. Substituting the data, we get 1000 + 34.2 × 1.5 + 3 × 10 - 7.7 × 0.5, which is approximately 1077.45 points. Using the same method, all alternative paths in the feasible path network are scored. Some paths, although short, have sharp turns and low scores, while others are smooth but involve long detours and are also not ideal. All paths are sorted from highest to lowest score, and the path with the highest score is the optimal path, which shortens the flight distance and controls the difficulty of turning.

[0122] After selecting the optimal path, curve optimization is required. The original path is a broken line with sharp angles at turning points. The program finds each turning point and selects two positions 5 meters in front and behind as the endpoints of an arc. The radius of the arc is determined by the size of the turning angle; a 30-degree turning angle uses an arc with a radius of 6 meters, and a 60-degree turning angle uses an arc with a radius of 12 meters. For example, at node N168, a 46-degree turn is made, and an arc with a radius of 10 meters is used to connect the two straight lines before and after it. The center of the arc is calculated using geometric relationships, and then dense waypoints are generated on the arc by interpolation at 0.5-meter intervals. After all turning points are replaced, the original broken line becomes a smooth curve with arc transitions. The coordinate sequence of waypoints on this curve is the obstacle avoidance path data. Each waypoint records information such as position, speed, and orientation for flight control use.

[0123] In a specific embodiment, adjusting the flight attitude parameters of the inspection drone based on the obstacle avoidance path data includes:

[0124] The obstacle avoidance path data is decomposed into target attitude values ​​in three directions: heading angle, pitch angle, and roll angle, to obtain an attitude command set. The attitude command set is then divided into time sequences to obtain an action sequence table including attitude adjustment order and transition time.

[0125] Based on the action sequence list, the difference of the current attitude of the UAV is calculated to obtain the attitude compensation value, and the power is allocated to the attitude compensation value to generate flight attitude parameters.

[0126] Specifically, after the obstacle avoidance path data is transmitted, the flight control program reads the coordinates of each waypoint one by one and calculates the required flight attitude based on the spatial relationship between two adjacent waypoints. Assuming the current waypoint WP082 coordinates are (68.3, 42.7, 21.5), and the next waypoint is WP083 coordinates (75.8, 48.2, 23.1), the horizontal projection direction of the line connecting the two points is the heading angle. By calculating atan2 (48.2-42.7, 75.8-68.3) using the arctangent function, the heading angle is approximately 36 degrees, indicating that the nose should be facing 36 degrees east of north. The pitch angle is determined by the ratio of altitude change to horizontal distance. The altitude difference is 23.1 - 21.5 = 1.6 meters, and the horizontal distance is approximately 9.2 meters (√(75.8 - 68.3)² + (48.2 - 42.7)²). The pitch angle is approximately 10 degrees (arctan(1.6 / 9.2)), meaning the aircraft needs to slightly pitch up. The roll angle is mainly used for turning. If the path involves a change of direction, centripetal force is generated by tilting the aircraft. With a turning radius of 8 meters and a speed of 4 meters per second, the roll angle, calculated using the formula arctan(v² / rg), is approximately 11 degrees. Turning left results in a positive 11 degrees, and turning right results in a negative 11 degrees. These three angle values ​​combined constitute the target attitude value for the flight segment. After calculating all waypoint pairs, an attitude command set is formed. Each command includes the segment number, heading angle, pitch angle, and roll angle.

[0127] Once the attitude command set is obtained, the execution order and time allocation must be considered, because attitude adjustments in the three directions cannot be completed simultaneously and must be prioritized. Generally, the heading angle is adjusted first to ensure correct direction, then the pitch angle is adjusted to control altitude, and finally the roll angle is finely adjusted to maintain balance. A transition time of 0.3 to 0.5 seconds is left between each action to allow the aircraft to stabilize. For example, when switching from flight segment S12 to S13, the heading angle needs to change from 28 degrees to 36 degrees, the pitch angle from 5 degrees to 10 degrees, and the roll angle from 0 degrees to 11 degrees. The action sequence table will record it as follows: the first step of adjusting the heading angle from 28 to 36 degrees takes 0.8 seconds, the second step of adjusting the pitch angle from 5 to 10 degrees takes 0.6 seconds, and the third step of adjusting the roll angle from 0 to 11 degrees takes 0.4 seconds, requiring a total of 1.8 seconds to complete the entire set of actions. After the actions of all flight segments are ordered and timed, a detailed action sequence table is formed, which clearly lists the start time, duration, and target value of each attitude adjustment action.

[0128] Before executing a maneuver, the difference between the current actual attitude and the target attitude must be measured. The IMU (Inertial Measurement Unit) on the drone outputs the current yaw, pitch, and roll angles in real time. For example, if the readings at a certain moment are 31 degrees yaw, 7 degrees pitch, and 2 degrees roll, while the target attitude is 36 degrees yaw, 10 degrees pitch, and 11 degrees roll, the attitude compensation values ​​are calculated as 5 degrees, 3 degrees, and 9 degrees respectively. These three differences tell the system how much rotation is needed in each direction. Then, the angle compensation is converted into actual motor movements. Quadcopter drones achieve attitude control by adjusting the speed of their four propellers. Increasing the yaw angle requires the two diagonally opposite motors to accelerate while the other two decelerate; adjusting the pitch angle relies on the differential speed of the front and rear motors; and controlling the roll angle relies on the differential speed of the left and right motors. The program calculates the speed increment for each motor based on the compensation value. For example, to increase the yaw angle by 5 degrees, motors 1 and 3 each increase their speed by 200 rpm, while motors 2 and 4 each decrease by 200 rpm. To increase the pitch angle by 3 degrees, the first two motors accelerate by 150 rpm, while the last two decelerate by 150 rpm. To roll by 9 degrees, the left motor increases its speed by 400 rpm, while the right motor decreases by 400 rpm. These speed commands are combined to form flight attitude parameters. The four values ​​correspond to the target speeds of the four motors. After being sent to the ESC module, the motors operate at the new speeds, and the aircraft attitude changes accordingly, gradually approaching the flight state required by the obstacle avoidance path.

[0129] The above describes a route inspection method based on UAV visual navigation in an embodiment of the present invention. The following describes a route inspection system based on UAV visual navigation in an embodiment of the present invention. Please refer to [link to relevant documentation]. Figure 2 One embodiment of the UAV-based visual navigation-based line inspection system of the present invention includes:

[0130] Enhancement module 21 is used to enhance the images of power equipment lines collected by the inspection drone in the inspection area to obtain enhanced images of the equipment lines.

[0131] The extraction module 22 is used to perform multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map, and to perform feature fusion on the equipment line feature map to obtain a fused feature map;

[0132] Analysis module 23 is used to perform target detection analysis on the fused feature map to obtain line hazard data and obstacle location data, and to annotate the line hazard data to obtain inspection report data;

[0133] The update module 24 is used to update the current flight path of the inspection drone based on the obstacle position data to obtain obstacle avoidance path data, and adjust the flight attitude parameters of the inspection drone according to the obstacle avoidance path data.

[0134] In this embodiment, the specific implementation of each module in the above system embodiment is described in the above method embodiment, and will not be repeated here.

Claims

1. A method for line inspection based on UAV visual navigation, characterized in that, Includes the following steps: Images of power equipment and lines collected by inspection drones in the inspection area are enhanced to obtain enhanced images of the equipment and lines; wherein, the inspection area includes power equipment and the power lines connecting the power equipment. Multi-scale feature extraction is performed on the enhanced equipment line image to obtain an equipment line feature map, and feature fusion is performed on the equipment line feature map to obtain a fused feature map; The fused feature map is subjected to target detection analysis to obtain line hazard data and obstacle location data. The line hazard data is then labeled to obtain inspection report data. The current flight path of the inspection drone is updated based on the obstacle location data to obtain obstacle avoidance path data, and the flight attitude parameters of the inspection drone are adjusted according to the obstacle avoidance path data. The step of performing multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map includes: The enhanced equipment circuit image is divided into multiple scales to obtain a multi-layer image group; Edge contours are extracted for each scale layer in the multi-layer image group to identify the edge information of power equipment at different scales, resulting in a multi-scale contour map. Geometric feature statistics are then performed on the multi-scale contour map to obtain a feature distribution map. The feature distribution map is divided into multiple feature sub-regions to obtain a regional feature map. Spatial relationship calculations are then performed on the feature sub-regions in the regional feature map to obtain a device circuit feature map. The feature map of the equipment circuit is fused to obtain a fused feature map, including: The importance of the features in the equipment line feature map is evaluated, and an importance score is obtained by analyzing the criticality of different feature sub-regions in power line inspection. The equipment line feature map is weighted based on the importance score to obtain a weighted feature map. Feature overlay calculation is then performed on the weighted feature map to add features at the same location at different scales to obtain a preliminary fused feature map. Redundant features with high correlation and little impact on the inspection results are removed from the preliminary fusion feature map to obtain the fusion feature map.

2. The route inspection method based on UAV visual navigation according to claim 1, characterized in that, The process of enhancing the images of power equipment lines collected by the inspection drone in the inspection area to obtain enhanced images of the equipment lines includes: The images of power equipment lines collected by the inspection drone in the inspection area are processed by brightness partitioning, and the images of power equipment lines are divided into multiple brightness sub-regions to obtain a brightness distribution map. Brightness equalization calculations are performed on each brightness sub-region in the brightness distribution map to obtain an enhanced device line image.

3. The route inspection method based on UAV visual navigation according to claim 1, characterized in that, Spatial relationship calculations are performed on the feature sub-regions in the region feature map to obtain the equipment circuit feature map, including: Centroid coordinates are extracted from each feature sub-region in the region feature map to obtain a set of equipment center points. Equipment type is labeled for each center point in the set of equipment center points to obtain an equipment location label map including tower location points, insulator location points and conductor connection points. Based on the equipment location marking map, the Euclidean distance between adjacent power equipment is calculated to obtain an equipment spacing table. The electrical connection relationship between power equipment is determined according to the equipment spacing table to obtain an equipment association matrix. The equipment association matrix records the connection status between towers and insulators, and between insulators and conductors. Based on the device association matrix, topological connection segments between devices are drawn on the regional feature map to obtain a topological connection map. The topological connection map is then overlaid with the device location annotation map to obtain a line feature map.

4. The route inspection method based on UAV visual navigation according to claim 1, characterized in that, The target detection analysis of the fused feature map yields line hazard data and obstacle location data, including: The fused feature map is divided into multiple equipment area maps according to the distribution of power equipment in the inspection area, and the area division results are obtained. The edge pixel statistics of each equipment area map in the area division results are obtained to obtain the edge pixel distribution. Based on the edge pixel distribution, the device region map is contour extracted to identify the outline of the power equipment in the device region map, and the outline of the power equipment is morphologically dilated to obtain a dilated outline map. The pixel values ​​in the expansion contour map are compared with the preset range of hidden danger feature pixel values ​​to obtain suspected hidden danger areas. The suspected hidden danger areas are then marked to obtain line hidden danger data. Based on the line hazard data, obstacle search is performed on the fused feature map to obtain obstacle location data.

5. The route inspection method based on UAV visual navigation according to claim 4, characterized in that, Based on the aforementioned line hazard data, obstacle search is performed on the fused feature map to obtain obstacle location data, including: Extract the pixel center coordinates of the line hazard data, and use the pixel center coordinates as the center and a preset search radius to delineate a circular boundary in the fused feature map to obtain the obstacle search area; The obstacle search area is segmented into foreground pixels and background pixels according to a grayscale threshold to obtain a foreground segmentation map. Connected regions are marked on the foreground pixels in the foreground segmentation map to obtain the obstacle candidate area. The centroid coordinates of the candidate obstacle region are calculated to obtain the obstacle pixel coordinates. The obstacle pixel coordinates are then transformed by combining the flight position parameters of the inspection UAV to obtain the obstacle position data.

6. The route inspection method based on UAV visual navigation according to claim 1, characterized in that, The current flight path of the inspection drone is updated based on the obstacle location data to obtain obstacle avoidance path data, including: The obstacle location data is used to calculate the safe distance, and the minimum obstacle avoidance distance is determined according to the size and type of the obstacle to obtain the safe area. The safe area is then gridded to obtain the path planning map. The path planning map is sampled, and path nodes are arranged at preset intervals within a safe area to obtain a node distribution map. Connectivity detection is performed on adjacent nodes in the node distribution map to obtain a feasible path. The feasible paths are calculated for path cost. The flight distance and turning angle of the current flight path of the inspection drone are comprehensively analyzed to obtain a path score. Based on the path score, the feasible paths are ranked to obtain the optimal path. The optimal path is smoothed by rounding off sharp corners and turns to obtain obstacle avoidance path data.

7. The route inspection method based on UAV visual navigation according to claim 1, characterized in that, Adjusting the flight attitude parameters of the inspection drone based on the obstacle avoidance path data includes: The obstacle avoidance path data is decomposed into target attitude values ​​in three directions: heading angle, pitch angle, and roll angle, to obtain an attitude command set. The attitude command set is then divided into time sequences to obtain an action sequence table including attitude adjustment order and transition time. Based on the action sequence list, the difference of the current attitude of the UAV is calculated to obtain the attitude compensation value, and the power is allocated to the attitude compensation value to generate flight attitude parameters.

8. A line inspection system based on UAV visual navigation, characterized in that, The method for performing the UAV-based visual navigation-based line inspection method according to any one of claims 1 to 7 includes: The enhancement module is used to enhance the images of power equipment and lines collected by the inspection drone in the inspection area, so as to obtain enhanced images of the equipment and lines. The extraction module is used to perform multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map, and to perform feature fusion on the equipment line feature map to obtain a fused feature map; The analysis module is used to perform target detection analysis on the fused feature map to obtain line hazard data and obstacle location data, and to annotate the line hazard data to obtain inspection report data; The update module is used to update the current flight path of the inspection drone based on the obstacle location data to obtain obstacle avoidance path data, and adjust the flight attitude parameters of the inspection drone according to the obstacle avoidance path data. The step of performing multi-scale feature extraction on the enhanced equipment line image to obtain an equipment line feature map includes: The enhanced equipment circuit image is divided into multiple scales to obtain a multi-layer image group; Edge contours are extracted for each scale layer in the multi-layer image group to identify the edge information of power equipment at different scales, resulting in a multi-scale contour map. Geometric feature statistics are then performed on the multi-scale contour map to obtain a feature distribution map. The feature distribution map is divided into multiple feature sub-regions to obtain a regional feature map. Spatial relationship calculations are then performed on the feature sub-regions in the regional feature map to obtain a device circuit feature map. The feature map of the equipment circuit is fused to obtain a fused feature map, including: The importance of the features in the equipment line feature map is evaluated, and an importance score is obtained by analyzing the criticality of different feature sub-regions in power line inspection. The equipment line feature map is weighted based on the importance score to obtain a weighted feature map. Feature overlay calculation is then performed on the weighted feature map to add features at the same location at different scales to obtain a preliminary fused feature map. Redundant features with high correlation and little impact on the inspection results are removed from the preliminary fusion feature map to obtain the fusion feature map.

Images