Unmanned aerial vehicle inspection path planning and navigation method for power transmission line based on visual intelligence

Abstract

The application discloses a power transmission line unmanned aerial vehicle inspection path planning navigation method based on visual intelligence, relates to the technical field of unmanned aerial vehicle inspection, and comprises the following steps: collecting a visible light image sequence and an infrared thermal imaging image sequence of a power transmission line operation environment; performing feature point extraction and matching on the visible light image sequence to generate a three-dimensional point cloud model containing the spatial positions of power transmission towers and poles and the direction of the power transmission lines; registering the infrared thermal imaging image sequence and the three-dimensional point cloud model to generate a composite perception map fusing geometric structures and thermal distribution states; calling a pre-trained visual navigation strategy network to analyze the composite perception map, generating a global flight trajectory and a local obstacle avoidance path segment; and generating a navigation instruction set containing flight attitude angles, flight speeds and shooting trigger points according to the connection relationship between the global flight trajectory and the local obstacle avoidance path segment. The method realizes the fusion of line geometry and thermal distribution information, and improves the accuracy and comprehensiveness of the inspection path.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) inspection technology, specifically a visual intelligence-based UAV inspection path planning and navigation method for power transmission lines. Background Technology

[0002] Unmanned aerial vehicle (UAV) inspection of power transmission lines is an important means to ensure the safe and stable operation of the power system. Existing UAV inspection path planning and navigation methods mostly rely on single-type image data. Some methods only collect visible light images of power transmission lines, generate line geometric information through image processing, and plan the flight path based on this. Other methods collect infrared thermal imaging images separately to detect abnormal equipment temperatures, but do not correlate the temperature information with the spatial geometric information of the line.

[0003] Existing technical solutions have shortcomings. When planning paths using only visible light images, they cannot perceive the surface temperature distribution of equipment, making it difficult to accurately cover temperature anomalies and leading to omissions in inspections. Furthermore, single infrared thermal imaging images lack spatial coordinate support for the route, making it impossible to accurately locate anomalies. Additionally, path planning only considers single-dimensional information, resulting in insufficient obstacle avoidance accuracy and path rationality. Moreover, existing path planning methods often employ traditional algorithms, making it difficult to dynamically generate paths that balance global inspection and local obstacle avoidance based on composite information. Navigation commands also fail to comprehensively cover key parameters such as flight attitude, speed, and image capture triggering.

[0004] How to effectively integrate line geometry information with equipment thermal distribution information, accurately obtain the spatial location of temperature anomalies, generate navigation paths that take into account both global trajectory and local obstacle avoidance, and output comprehensive and accurate UAV inspection navigation commands has become an urgent problem to be solved in the current UAV inspection path planning and navigation technology for power transmission lines. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art;

[0006] To this end, the present invention proposes a visual intelligence-based unmanned aerial vehicle (UAV) inspection path planning and navigation method for power transmission lines, comprising:

[0007] The visible light image sequence and infrared thermal image sequence of the working environment of the transmission line are collected. The visible light image sequence is used to identify the physical shape of the line towers and insulators, and the infrared thermal image sequence is used to sense the temperature distribution of the equipment surface.

[0008] Feature point extraction and matching processing are performed on the visible light image sequence to generate a three-dimensional point cloud model containing the spatial location of the transmission towers and the direction of the conductors. The three-dimensional point cloud model is used to construct the geometric skeleton of the line structure.

[0009] The infrared thermal imaging image sequence is registered with the three-dimensional point cloud model to generate a composite sensing map that integrates geometric structure and thermal distribution state. The composite sensing map includes the spatial coordinates of temperature anomaly points and temperature gradient change data.

[0010] The pre-trained visual navigation strategy network is invoked to perform flight path analysis on the composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task.

[0011] Based on the connection between the global flight trajectory and the local obstacle avoidance path segment, a set of UAV inspection path planning and navigation instructions is generated, which includes flight attitude angle, flight speed and shooting trigger point.

[0012] Furthermore, feature point extraction and matching processing is performed on the visible light image sequence to generate a three-dimensional point cloud model containing the spatial location of the transmission towers and the orientation of the conductors, including:

[0013] Each frame of the visible light image sequence is subjected to edge enhancement and texture filtering. Corner points and line segments whose corner feature response values ​​are greater than a first preset threshold and whose line segment lengths are greater than a second preset threshold are extracted as candidate feature points.

[0014] By using a cross-frame feature point matching algorithm, a spatial correspondence is established between candidate feature points in two adjacent frames of images, generating a set of feature point matching pairs.

[0015] Based on the set of feature point matching pairs and the flight attitude sensor data of the UAV, the coordinate values ​​of the feature points in three-dimensional space are calculated using the principle of triangulation.

[0016] Clustering of feature point coordinates belonging to the same transmission tower or conductor, and fitting to generate a three-dimensional contour point set of the transmission tower and a spatial curve equation of the conductor;

[0017] By integrating the three-dimensional contour point set of all transmission towers with the spatial curve equation of the conductors, a three-dimensional point cloud model describing the overall structure of the line is constructed.

[0018] Further, the infrared thermal imaging image sequence is registered with the three-dimensional point cloud model to generate a composite sensing map that integrates geometric structure and thermal distribution, including:

[0019] The infrared thermal imaging image sequence is subjected to non-uniformity correction and temperature calibration to generate a standardized temperature matrix;

[0020] Key points with obvious planar or cylindrical features are extracted from the three-dimensional point cloud model and used as reference control points for registration.

[0021] The temperature values ​​in the standardized temperature matrix are mapped to the key points and their neighborhood space of the three-dimensional point cloud model through the feature projection algorithm, and a correlation table between temperature data and geometric coordinates is established.

[0022] Spatial interpolation is performed on the temperature data in the association table to generate temperature field distribution data covering the entire surface of the three-dimensional point cloud model.

[0023] The temperature field distribution data is overlaid with the three-dimensional point cloud model to generate the composite sensing map that simultaneously includes geometric coordinates, structure type, and temperature attributes.

[0024] Furthermore, the step of invoking a pre-trained visual navigation strategy network to perform flight path analysis on the composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task, including:

[0025] The composite perception map is input into the encoder part of the visual navigation strategy network to extract the topological features of the transmission line, the temperature distribution features of the temperature anomaly area, and the contour features of obstacles in the environment.

[0026] The topological features, temperature distribution features, and obstacle contour features are input into the decoder part of the visual navigation strategy network to generate an initial flight waypoint sequence.

[0027] The initial flight waypoint sequence is subjected to connectivity verification. If there is a situation where the line of sight between waypoints is blocked by obstacles, a transition waypoint is inserted in the blocked area to generate a corrected flight waypoint sequence.

[0028] Based on the corrected flight waypoint sequence, the global flight trajectory of the UAV in the long-distance cruise segment and the local obstacle avoidance path segment in the close-range fine observation segment are calculated respectively.

[0029] The start and end points of the global flight trajectory are smoothly connected to the endpoints of the local obstacle avoidance path segments to form a seamless and complete flight path.

[0030] Furthermore, the initial waypoint sequence undergoes connectivity verification. If the line of sight between waypoints is obstructed by obstacles, transition waypoints are inserted within the obstructed areas to generate a corrected waypoint sequence, including:

[0031] A line-of-sight vector is constructed using two adjacent waypoints in the initial flight waypoint sequence as endpoints;

[0032] Using a ray casting algorithm, it is detected whether the line-of-sight vector collides or interferes with the outline of an obstacle in the composite perception map;

[0033] If a collision interference is detected, the position coordinates and normal direction of the collision point are calculated, and a no-fly zone is delineated with the collision point as the center and a preset safety distance as the radius.

[0034] On the boundary of the no-fly zone, select the tangent point with the smallest angle to the line-of-sight vector as the transition waypoint, and insert the transition waypoint between the original adjacent waypoints;

[0035] Traverse all adjacent waypoint pairs in the initial waypoint sequence, repeatedly perform line-of-sight detection and transition waypoint insertion operations until all lines of sight are unobstructed, and output the corrected waypoint sequence.

[0036] Furthermore, based on the connection relationship between the global flight trajectory and the local obstacle avoidance path segment, a UAV inspection path planning and navigation instruction set is generated, including flight attitude angles, flight speeds, and shooting trigger points, comprising:

[0037] Extract the geometric parameters of the straight and turning segments of the global flight trajectory, and calculate the pitch and roll angles of the UAV in cruise mode based on the geometric parameters, which are used as the reference flight attitude angles;

[0038] The flight speed of the UAV is dynamically adjusted based on the rate of curvature change of the local obstacle avoidance path segment, so that the flight speed is inversely proportional to the rate of curvature change of the path segment.

[0039] From the temperature anomaly points and key structural points in the composite sensing map, target points that need to be photographed are selected, and the best shooting position that can be directly facing the target points is determined on the global flight trajectory and local obstacle avoidance path segment.

[0040] Set the shooting trigger point at each of the optimal shooting positions, and configure the corresponding shutter trigger timing and gimbal pointing angle;

[0041] The calculated flight attitude angle, the adjusted flight speed, and all the shooting trigger points and their parameters are encoded into the UAV inspection path planning and navigation instruction set.

[0042] Furthermore, from the temperature anomalies and key structural points in the composite sensing map, target points that need to be focused on for imaging are selected, and the optimal imaging position directly facing the target points is determined on the global flight trajectory and local obstacle avoidance path segments, including:

[0043] Traverse all temperature anomaly points in the composite sensing map, mark points whose temperature values ​​exceed the preset fault threshold as primary target points, and mark points whose temperature values ​​are within the warning range as secondary target points;

[0044] All insulator string points, conductor crimping points, and hardware connection points of the transmission towers are extracted from the three-dimensional point cloud model as key structural points.

[0045] The primary target points, secondary target points, and structural key points are deduplicated and merged to form a unified set of target points.

[0046] For each target point in the target point set, the intersection of the inverted cone with the target point as the center and the preset field of view as the half-aperture angle with the global flight trajectory and the local obstacle avoidance path segment is calculated, and the position closest to the target point in the intersection set is taken as the best shooting position.

[0047] If a target point cannot find a valid intersection on any path, then the target point is removed from the set of target points.

[0048] Furthermore, the visual navigation policy network is trained through the following steps:

[0049] Acquire composite sensing map samples of multiple historical inspection tasks and corresponding manual operation trajectory records. The composite sensing map samples include spatial information of line structure, temperature anomalies and obstacles, and the manual operation trajectory records are used as training labels.

[0050] A neural network model consisting of an encoder, an attention mechanism module, and a decoder is constructed. The encoder is used to extract features from the composite sensing map, the attention mechanism module is used to enhance the feature weights of temperature anomaly areas and obstacles, and the decoder is used to generate a predicted waypoint sequence.

[0051] The composite sensing map samples are input into the neural network model, and the mean square error between the predicted waypoint sequence output by the model and the trajectory sequence of the corresponding training label is calculated as the initial loss function.

[0052] Based on the initial loss function, a penalty term is introduced to impose additional losses on waypoints that collide with obstacles in the predicted trajectory and waypoints that are missed in observing key target points, thus forming a multi-objective joint loss function;

[0053] Using the backpropagation algorithm, the parameters of the neural network model are updated based on the multi-objective joint loss function until the average error between the predicted trajectory and the training label on the validation set is lower than a preset convergence threshold, thus completing the training and obtaining the pre-trained visual navigation policy network.

[0054] Furthermore, it also includes a path replanning step based on real-time visual feedback:

[0055] During the flight of the UAV according to the UAV inspection path planning and navigation instruction set, the current field of view image collected by the airborne camera is acquired in real time;

[0056] The current field-of-view image is compared with the prior image at the corresponding location in the composite perception map to identify the displacement of newly added unknown obstacles or existing obstacles.

[0057] If an unknown obstacle or obstacle displacement is detected, the visual navigation strategy network is invoked again to generate a temporary avoidance path segment, starting from the current drone position and targeting the nearest path node.

[0058] The temporary avoidance path segment is spliced ​​and replaced with the original planned path to generate an inspection path planning and navigation instruction set that is updated in real time.

[0059] Control the drone to switch and execute the real-time updated inspection path planning and navigation instruction set until it bypasses the obstacle and returns to the original planned path.

[0060] Further, the step of comparing the feature differences between the current field-of-view image and the prior image at the corresponding location in the composite sensing map to identify the displacement of newly added unknown obstacles or existing obstacles includes:

[0061] Extract local prior images of the area surrounding the current UAV location from the composite sensing map. The local prior images include known towers, power lines, and background environmental features.

[0062] Scale-invariant feature transformation is performed on the current field-of-view image and the local prior image respectively, and feature descriptors of the two sets of images are extracted.

[0063] The feature descriptor matching algorithm is used to calculate the feature matching logarithm and matching error between two sets of images.

[0064] If the number of matching pairs is lower than a preset matching threshold or the matching error is higher than a preset error threshold, then it is determined that there is a visually inconsistent region in the current field of view.

[0065] Connectivity analysis and contour fitting are performed on the visually inconsistent regions to identify the bounding rectangles of suspected new obstacles or existing obstacles that have been displaced, and their position coordinates and size information are output.

[0066] Compared with the prior art, the beneficial effects of the present invention are:

[0067] The infrared thermal imaging sequence and visible light image sequence collected during power transmission line inspections are matched after feature point extraction. A 3D point cloud model containing the spatial location of transmission towers and the direction of conductors is then registered to generate a composite sensing map that integrates the geometric structure of the transmission line and the thermal distribution status of equipment. This composite sensing map includes the spatial coordinates of temperature anomalies and temperature gradient change data. Compared to conventional single-image data applications, this method can simultaneously acquire information on the line's geometric framework and the equipment's thermal distribution, achieving precise correspondence between temperature anomalies and their spatial locations. This avoids anomaly location errors due to a lack of geometric information and inspection omissions due to a lack of thermal distribution information. It ensures that the inspection path accurately covers all key detection areas, enabling simultaneous progress in geometric structure inspection and temperature anomaly detection.

[0068] A pre-trained visual navigation strategy network is invoked to perform flight path analysis on a composite perception map that integrates geometric structure and thermal distribution. This generates a global flight trajectory and local obstacle avoidance path segments for the inspection task. Based on the connection between the two, a drone inspection path planning and navigation command set is generated, including flight attitude angles, flight speed, and shooting trigger points. Compared with conventional path planning algorithms, this approach can achieve reasonable global flight trajectory planning, ensuring the comprehensiveness of the inspection task. At the same time, it accurately generates local obstacle avoidance path segments to cope with complex obstacles in the working environment. The navigation commands cover key parameters throughout the entire flight process without additional manual intervention, making drone flight smoother, reducing flight deviation and collision risks. Furthermore, precise trigger shooting ensures the integrity and relevance of the inspection image data. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the steps of the visual intelligence-based UAV inspection path planning and navigation method for power transmission lines described in this invention.

[0070] Figure 2 A flowchart for generating a 3D point cloud model;

[0071] Figure 3 A flowchart for flight path analysis and processing;

[0072] Figure 4 A heatmap showing the correlation of training data for a visual navigation strategy network;

[0073] Figure 5 This is a graph showing the changes in flight speed and attitude angle parameters during different flight phases of a drone inspection. Detailed Implementation

[0074] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] See Figure 1 The system collects visible light image sequences and infrared thermal image sequences of the operating environment of the transmission line. The visible light image sequences are used to identify the physical morphology of the line towers and insulators, while the infrared thermal image sequences are used to sense the temperature distribution on the equipment surface. Feature point extraction and matching processing is performed on the visible light image sequences to generate a 3D point cloud model containing the spatial location of the transmission towers and the direction of the conductors. This 3D point cloud model is used to construct the geometric skeleton of the line structure. The infrared thermal image sequences and the 3D point cloud model are registered to generate a composite perception map that integrates the geometric structure and thermal distribution. This composite perception map includes the spatial coordinates of temperature anomalies and temperature gradient change data. A pre-trained visual navigation strategy network is invoked to perform flight path analysis processing on the composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task. Based on the connection relationship between the global flight trajectory and the local obstacle avoidance path segments, a drone inspection path planning and navigation instruction set containing flight attitude angles, flight speeds, and shooting trigger points is generated.

[0076] In one embodiment of the present invention, see [reference] Figure 2This paper describes a method for extracting and matching feature points from visible light image sequences to generate a three-point cloud model containing the spatial locations of transmission towers and the orientation of conductors. The method includes the following steps: Edge enhancement and texture filtering are performed on each frame of the visible light image sequence. This process aims to suppress image noise and enhance the edge information of linear structures such as towers and conductors. On the preprocessed image, the corner feature response value of each pixel is calculated, and all continuous line segments with a length greater than a second preset threshold are detected. Pixels with corner feature response values ​​greater than a first preset threshold, along with the endpoints of line segments that meet the length condition, are collectively marked as candidate feature points. Using a cross-frame feature point matching algorithm, such as descriptor-based brute-force matching or a FLANN matcher, the most similar spatial correspondence points for candidate feature points from the previous frame are found in the subsequent frame between two temporally adjacent frames, thus establishing a set of feature point matching pairs. Based on the feature point matching pairs and the flight attitude sensor data from the synchronously recorded UAV flight control data, the precise coordinates of each pair of matched feature points in three-dimensional space are calculated using the triangulation principle. The flight attitude sensor data includes the position and attitude angle of the UAV when capturing each frame of the image. It can be understood that triangulation calculations rely on the pixel coordinates of the feature points in the two frames, the camera's intrinsic parameter matrix, and the relative motion relationship between the two frames. All three-dimensional points obtained through triangulation are clustered according to their spatial distribution and geometric relationships. Feature points belonging to the same transmission tower are clustered into a single point set, and a surface fitting algorithm is used to generate a three-dimensional contour point set for the transmission tower. Feature points belonging to the same conductor are used to fit the spatial curve equation representing its catenary morphology. Integrating the three-dimensional contour point sets of all transmission towers and the spatial curve equations of all conductors, a complete three-dimensional point cloud model describing the geometry of the transmission line corridor is constructed. In some embodiments, the corner feature response values... Calculated using the following formula:

[0077]

[0078] in: It is the second-order differential matrix of the local window of the image. It is a matrix The determinant, It is a matrix traces, It is an empirical constant with a value ranging from 0.04 to 0.06.

[0079] In practice, the infrared thermal imaging image sequence is registered with a 3D point cloud model to generate a composite sensing map that integrates geometric structure and thermal distribution. This process includes the following steps: Non-uniformity correction and temperature calibration are performed on the infrared thermal imaging image sequence. Non-uniformity correction eliminates fixed pattern noise caused by inconsistencies in the responses of different pixels of the infrared detector. Temperature calibration converts the grayscale values ​​output by the detector into physical quantities reflecting the true temperature of the object's surface according to calibration parameters. The result is a series of standardized temperature matrices, where each element represents the actual temperature at the corresponding location in the image. Key points with obvious planar or cylindrical features are extracted from the constructed 3D point cloud model. Optionally, these key points can be the center point of the tower foot plane of the transmission tower, the end point of the crossarm, or the point where the conductor connects to the insulator. These points will serve as reference control points for aligning the 2D thermal image data with the 3D geometric model. The feature projection algorithm maps temperature values ​​from a standardized temperature matrix to key points and their neighborhood in a 3D point cloud model. Based on the relative pose of the infrared and visible light cameras and their imaging models, the algorithm assigns the temperature values ​​of corresponding pixels in the temperature matrix to their corresponding key points in the 3D point cloud model, establishing a correlation table between temperature data and geometric coordinates. Since the sampling resolution and spatial coverage of infrared thermal images and visible light images may not be entirely consistent, spatial interpolation is performed on the temperature data in the correlation table, such as using Kriging interpolation or inverse distance weighted interpolation algorithms, to generate continuous temperature field distribution data covering the entire surface of the 3D point cloud model.

[0080] In one embodiment of the present invention, a pre-trained visual navigation strategy network is invoked to perform flight path analysis on a composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task. This process includes multiple steps. (See also...) Figure 3 The composite sensing map is input into the encoder part of the visual navigation strategy network. The encoder, typically composed of a convolutional neural network, is used to extract and compress spatial information from the map layer by layer, outputting a high-dimensional feature vector containing features of the power transmission line topology, temperature distribution features of temperature anomaly areas, and the contour features of obstacles in the environment. These high-dimensional feature vectors are then input into the decoder part of the visual navigation strategy network. The decoder, typically composed of fully connected layers or a recurrent neural network, is used to decode the abstract feature vectors into a series of ordered spatial coordinate points, i.e., generating the initial flight waypoint sequence.

[0081] In practice, the initial flight waypoint sequence undergoes connectivity verification. If the line of sight between waypoints is obstructed by obstacles, transition waypoints are inserted within the obstructed area to generate a corrected flight waypoint sequence. The process is as follows: Using two adjacent waypoints in the initial flight waypoint sequence as endpoints, a directed line segment, i.e., the line-of-sight vector, is constructed from the starting point to the ending point. Using a ray casting algorithm, it is detected whether the line-of-sight vector collides with the contours of obstacles in the composite perception map. The algorithm emits a ray from the starting point along the direction of the line-of-sight vector and calculates the intersection points of the ray with all triangular faces (obtained by triangulation of the 3D point cloud model) in the scene. In essence, the ray casting algorithm traverses all possible intersecting faces to find the nearest collision point. If a collision is detected, the position coordinates of the collision point and the normal direction of the collision face are calculated. A spherical no-fly zone is then delineated in 3D space with the collision point as the center and a preset safety distance as the radius. On the boundary sphere of the no-fly zone, calculate all points tangent to the line-of-sight vector, and select the point with the smallest angle to the line-of-sight vector as a transition waypoint. Insert this newly generated transition waypoint between two existing adjacent waypoints. Iterate through all adjacent waypoint pairs in the initial flight waypoint sequence, repeating the line-of-sight detection and transition waypoint insertion operations for each pair until the line-of-sight vector formed by any two adjacent waypoints in the sequence is verified to be unobstructed. The output at this point is the corrected flight waypoint sequence. In some embodiments, the preset safety distance needs to comprehensively consider the physical size of the UAV, positioning error, and flight control margin. Calculate the collision point. The formula can be expressed as:

[0082]

[0083] in: Represents the starting coordinates of the line-of-sight vector. It is a normalized view direction vector. It is the minimum non-negative intersection point distance parameter obtained after the ray intersects with all obstacle patches.

[0084] In practice, based on the corrected waypoint sequence, the global flight trajectory of the UAV during the long-distance cruise segment and the local obstacle avoidance path segment during the close-range fine observation segment are calculated separately. The global flight trajectory is calculated by smoothly fitting a series of far-distance waypoints used for route connections using spline curves, aiming to achieve efficient cruise flight. The local obstacle avoidance path segment is planned using more refined curves for areas with dense waypoints and close to targets or obstacles to ensure flight stability and accuracy. It can be understood that the local obstacle avoidance path segment usually has a higher pathpoint density and stricter dynamic constraints. The start and end points of the global flight trajectory are smoothly connected to the endpoints of the local obstacle avoidance path segment. The smooth connection is achieved by ensuring the continuity of the first derivative (velocity) and second derivative (acceleration) of the trajectory at the connection point, thus forming a complete flight path that is seamlessly connected both geometrically and kinematically.

[0085] In one embodiment of the present invention, a drone inspection path planning and navigation instruction set, including flight attitude angles, flight speed, and shooting trigger points, is generated based on the connection relationship between the global flight trajectory and local obstacle avoidance path segments. This process involves multiple calculation and configuration steps. Geometric parameters of the straight and turning segments of the global flight trajectory are extracted, including the length and direction vector of the straight segments and the radius of curvature and central angle of the turning segments. Based on these geometric parameters, the pitch and roll angles required for the drone to maintain trajectory tracking during cruise are calculated, and the calculation results serve as the reference flight attitude angles. The drone's flight speed is dynamically adjusted according to the rate of curvature change of the local obstacle avoidance path segments, making the flight speed inversely proportional to the rate of curvature change of the path segments. That is, the flight speed is reduced in sections with large or drastic path curvature changes, while a higher flight speed is allowed in sections with gentle paths. Target points requiring focused shooting are selected from temperature anomaly points and key structural points in the composite sensing map, and the optimal shooting position directly facing the target points is determined on the global flight trajectory and local obstacle avoidance path segments. At each determined optimal shooting location, a shooting trigger point is set, and the corresponding shutter trigger timing and gimbal pointing angle are configured. The gimbal pointing angle is calculated in real time based on the target point's position relative to the UAV. The calculated flight attitude angle sequence, the adjusted flight speed sequence, and all shooting trigger points and their parameters are organized according to a predefined instruction encoding format to ultimately generate a UAV inspection path planning and navigation instruction set that the UAV flight control system can directly parse and execute. In some embodiments, flight speed... With path curvature change rate The inverse proportional relationship can be defined by a function:

[0086]

[0087] in: It is the planned flight speed at a specific point on a local obstacle avoidance path segment. It is the preset baseline cruising speed. It is an adjustment coefficient greater than zero. It is the absolute value of the rate of change of curvature of the path at that point.

[0088] In practical implementation, target points requiring focused photography are selected from temperature anomaly points and key structural points in the composite sensing map. The optimal shooting position directly facing the target point is determined on the global flight trajectory and local obstacle avoidance path segments. The selection and determination process follows these methods: All points with temperature attributes in the composite sensing map are traversed. Points with temperature values ​​exceeding a preset fault threshold are marked as primary target points, and points with temperature values ​​within a preset warning range are marked as secondary target points. All insulator string points, conductor crimping points, and hardware connection points of transmission towers are extracted from the 3D point cloud model and designated as structural critical points. The resulting sets of primary, secondary, and structural critical points are deduplicated and merged. Deduplication eliminates points that completely overlap or are extremely close in spatial location, resulting in a unified set of target points without duplicate elements. It is understood that a single location may simultaneously be a temperature anomaly point and a structural critical point. For each target point in the target point set, an inverted cone with its opening facing away from the center is constructed, centered on the target point and with a preset field of view as its semi-apex angle. This inverted cone represents all possible spatial directions from which the UAV camera can clearly capture the target point. The inverted cone is then spatially intersected with the global flight path and local obstacle avoidance path segments. The intersection point closest to the target point among all intersection points is taken as the optimal shooting position. Optionally, the intersection operation can be performed by calculating whether there is a solution between the path parameter equation and the cone surface equation. If, after the above intersection operation, a target point is found to have no valid intersection with any segment of the global flight path or local obstacle avoidance path (i.e., no path point can satisfy the field of view requirement for shooting the target point), this target point is removed from the final target point set, and no specific shooting action is planned for it. In some embodiments, the size of the preset field of view depends on the lens focal length and sensor size of the camera mounted on the UAV. It can be understood that determining the optimal shooting position ensures that the target point is stably located in the center area of ​​the camera image as the UAV passes along the flight path.

[0089] In one embodiment of the present invention, the visual navigation strategy network is trained through a process including data preparation, model building, loss function design, and iterative optimization. Multiple sets of composite perception map samples from historical inspection tasks and corresponding manual operation trajectory records are acquired. Each set of samples contains a complete composite perception map sample, which contains spatial information about the geometric structure, temperature anomaly distribution, and obstacles of a specific route segment. The corresponding manual operation trajectory record is generated by experienced operators operating a drone to inspect the same route segment in a simulated or real environment. This trajectory record serves as the target label for model learning. A neural network model consisting of an encoder, an attention mechanism module, and a decoder is constructed. The encoder typically employs a convolutional neural network structure to extract and compress the spatial features of the input composite perception map samples layer by layer. The attention mechanism module is integrated between the encoder and decoder to enhance the feature weights of temperature anomaly areas and obstacle areas in the composite perception map samples. The decoder typically consists of fully connected layers or a recurrent neural network to decode the encoded and weighted feature vectors to generate a predicted waypoint sequence.

[0090] In practice, composite sensing map samples are input into the constructed neural network model. The model's forward propagation outputs a predicted waypoint sequence. The mean squared error (MSE) between this predicted waypoint sequence and the corresponding manually operated trajectory records with training labels is calculated, and this MSE serves as the initial loss function. Based on the initial loss function, penalty terms tailored to specific flight safety and inspection task requirements are introduced. An additional collision penalty is applied to waypoints in the predicted trajectory that collide with obstacles, and an observation omission penalty is applied to waypoints in the predicted trajectory that fail to provide close-up observation of key target points (such as high-temperature anomalies). The initial loss function and these penalty terms are weighted and summed to form a multi-objective joint loss function used to guide network training. It can be understood that the introduction of penalty terms is to integrate business rules and prior knowledge into model training, rather than simply fitting trajectory coordinates. Using the backpropagation algorithm, gradients are calculated based on the multi-objective joint loss function, and the trainable parameters of the neural network model, including convolutional kernel weights, fully connected layer weights, and bias terms, are updated. This process is iteratively performed on a training set composed of multiple composite sensing map samples. Model training continues until the average positional error between the predicted trajectory and the validation set labels on an independent validation set falls below a preset convergence threshold. At this point, training is complete, resulting in a pre-trained visual navigation policy network that can be used for practical path planning. In some embodiments, a multi-objective joint loss function... The structure is as follows:

[0091]

[0092] in: It is the mean squared error loss between the predicted waypoint sequence and the manually operated trajectory record. This is a collision penalty, the value of which is calculated based on the probability that the predicted waypoint will fall into the obstacle area. This is an observation omission penalty, the value of which is calculated based on the average observation distance between the predicted waypoint and the critical target point. and It is a hyperparameter used to balance the weights of various losses.

[0093] To illustrate the composition of the training data, refer to Table 1, which shows a set of exemplary training dataset samples, where composite perception map samples and manual operation trajectory records appear in pairs. Optionally, the training data should cover line scenarios with different voltage levels, different terrain environments, and different defect types to ensure that the trained visual navigation strategy network has broad generalization capabilities.

[0094] Table 1: Sample Table of Training Dataset for Visual Navigation Strategy Network

[0095] Sample ID Line type Number of temperature anomalies Number of obstacles Number of manual operation trajectory points S001 220kV double circuit on the same tower 3 5 150 S002 500kV single circuit 1 2 120 S003 110kV overhead line 0 8 90 S004 330kV double circuit on the same tower 2 4 180

[0096] Understandably, the training set typically needs to reach thousands or even tens of thousands of sample pairs to effectively train a robust visual navigation policy network. In each training iteration, a batch of samples is randomly selected from the dataset and input into the network for forward computation and backpropagation.

[0097] See Figure 4 The study presented the linear correlation between three core variables (temperature anomalies, obstacles, and the number of trajectory points). Temperature anomalies showed a strong positive correlation with the number of trajectory points (correlation coefficient 0.67), indicating that in inspection tasks, the more temperature anomalies there are, the more manually generated trajectory points appear, reflecting the need for more refined path planning by the UAV to cover and observe these critical defect points. Obstacles showed a moderate negative correlation with the number of trajectory points (correlation coefficient -0.55), indicating that an increase in the number of obstacles leads to a decrease in the number of manually generated trajectory points. This may be because complex obstacle environments prompt operators to adopt simpler and more efficient obstacle avoidance strategies rather than increasing the number of path nodes. Temperature anomalies showed a weak negative correlation with obstacles (correlation coefficient -0.16), indicating that these two factors were almost independent in the training data, with significant differences in their spatial distribution and their impact mechanism on path planning.

[0098] In one embodiment of the present invention, the path replanning step based on real-time visual feedback is the core mechanism for the UAV to cope with dynamic environmental changes during the execution of a preset inspection task. During the flight of the UAV according to the UAV inspection path planning navigation instruction set, the onboard computer acquires the current field-of-view image captured by the onboard camera in real time via a data link. The current field-of-view image is compared with the prior image at the corresponding location in the composite sensing map. This comparison aims to identify newly added unknown obstacles in the flight environment or situations where existing obstacles have shifted relative to the composite sensing map record. It can be understood that the prior image refers to the visible light or infrared image data corresponding to the same spatial location used when constructing the composite sensing map. If an unknown obstacle or obstacle displacement is identified, the path planning system will use the current UAV position as the starting point for replanning, the nearest next path node on the original planned path as the temporary target point, re-call the visual navigation strategy network, and input a local environment representation incorporating the latest obstacle information to generate a temporary avoidance path segment. In some embodiments, the nearest next path node can be the waypoint on the original global flight trajectory or local obstacle avoidance path segment that is closest to the current UAV position and located ahead of it. The newly generated temporary avoidance path segment is spliced ​​and replaced with the original planned path. Specifically, the temporary avoidance path segment covers the section of the original instruction set from the current starting point to the nearest path node, thus generating a real-time updated inspection path planning and navigation instruction set. The UAV flight control system switches to and executes this real-time updated inspection path planning and navigation instruction set until the UAV successfully bypasses the newly identified obstacle and reaches the original path node. Afterward, the UAV resumes executing the original, unaffected subsequent inspection path planning and navigation instruction set, thereby ensuring the continuity of the inspection mission.

[0099] In specific implementation, the current field-of-view image is compared with the prior image at the corresponding location in the composite sensing map to identify newly added unknown obstacles or the displacement of existing obstacles. The specific process is as follows: Local prior images of the area surrounding the current UAV location are extracted from the composite sensing map. These local prior images contain feature information about poles, wires, and the background environment at the corresponding viewpoint, based on historical data. Scale-invariant feature transformation (SIN) is performed on both the current field-of-view image and the extracted local prior images. SIN processing extracts local image feature descriptors that are invariant to rotation, scaling, and brightness changes. Using a feature descriptor matching algorithm, such as nearest neighbor matching, the number of matching pairs and the matching error between the feature descriptors of the current field-of-view image and the feature descriptors of the local prior images are calculated. The matching error is typically measured by the distance between the descriptors of the matching pairs. If the calculated number of matching pairs is lower than a preset matching threshold, or the average matching error is higher than a preset error threshold, it is determined that there is a visually inconsistent region in the current field of view that is inconsistent with historical knowledge. Optionally, the preset matching threshold and preset error threshold need to be calibrated experimentally for a specific inspection environment. Connectivity analysis and contour fitting are performed on the identified visual inconsistency regions. Connectivity analysis aggregates spatially adjacent differing pixels into regions, while contour fitting further extracts the boundaries of these regions. Finally, the bounding rectangles of suspected newly added obstacles or displaced existing obstacles are identified, and their position coordinates and dimensions are output. In some embodiments, the criteria for determining visual inconsistencies are... It can be represented as:

[0100]

[0101] in: It is the number of feature matching logs. It is a preset matching threshold. This represents the logical "OR" operation. It is the average descriptor distance between all successfully matched feature point pairs. It is a preset error threshold. When the logical expression When true, a visual inconsistency determination is triggered.

[0102] See Figure 5During different flight phases of the UAV inspection path planning and navigation, the flight speed and attitude angle parameters exhibit a dynamic change pattern that is highly coupled with the mission objective. In specific operations, the flight speed (m / s), as the core control variable on the left longitudinal axis, reaches a peak of 8.5 m / s during the cruise phase to ensure the efficiency of long-distance inspections; it significantly decreases to 6.2 m / s and 3.8 m / s during the obstacle approach and obstacle avoidance phases, respectively, reserving sufficient decision-making and execution space for precise obstacle avoidance operations through deceleration; and it rebounds to 7.1 m / s and 8.0 m / s during the recovery phase and the inspection completion phase, respectively, demonstrating a smooth transition from emergency avoidance to resumed cruise. Flight attitude angles (°) are key response quantities on the right longitudinal axis. Changes in pitch and roll angles directly correspond to the operational intent during flight phases: During obstacle avoidance, the pitch angle reaches a peak of 7.1°, and the roll angle rises to 4.8° simultaneously, achieving active obstacle avoidance through significant attitude adjustments; During cruise and inspection completion phases, both pitch and roll angles remain at low levels (pitch angle 2.5° / 2.0°, roll angle 1.0° / 1.0°) to ensure flight stability and consistency of the shooting field of view; During obstacle approach and recovery phases, the attitude angles exhibit a transitional characteristic from cruise to obstacle avoidance and then from obstacle avoidance back to cruise (pitch angle 3.8° / 4.1°, roll angle 2.5° / 2.9°), reflecting the attitude adjustment strategy of the UAV under different mission states. During parameter configuration, the linkage between flight speed and attitude angle follows the core principles of "curvature-speed inverse ratio" and "mission-attitude matching": in the obstacle avoidance phase with large curvature, the speed is reduced to match the high attitude adjustment requirements; in the cruise phase with gentle curvature, the speed is increased to match the low attitude adjustment requirements, thereby achieving a dynamic balance between flight efficiency and operational safety.

[0103] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A visual intelligence-based method for UAV inspection path planning and navigation of power transmission lines, characterized in that, The method includes: The visible light image sequence and infrared thermal image sequence of the working environment of the transmission line are collected. The visible light image sequence is used to identify the physical shape of the line towers and insulators, and the infrared thermal image sequence is used to sense the temperature distribution of the equipment surface. Feature point extraction and matching processing are performed on the visible light image sequence to generate a three-dimensional point cloud model containing the spatial location of the transmission towers and the direction of the conductors. The three-dimensional point cloud model is used to construct the geometric skeleton of the line structure. The infrared thermal imaging image sequence is registered with the three-dimensional point cloud model to generate a composite sensing map that integrates geometric structure and thermal distribution state. The composite sensing map includes the spatial coordinates of temperature anomaly points and temperature gradient change data. The pre-trained visual navigation strategy network is invoked to perform flight path analysis on the composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task. Based on the connection between the global flight trajectory and the local obstacle avoidance path segment, a set of UAV inspection path planning and navigation instructions is generated, which includes flight attitude angle, flight speed and shooting trigger point.

2. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 1, characterized in that, Feature point extraction and matching are performed on the visible light image sequence to generate a three-dimensional point cloud model containing the spatial location of the transmission towers and the orientation of the conductors, including: Each frame of the visible light image sequence is subjected to edge enhancement and texture filtering. Corner points and line segments whose corner feature response values ​​are greater than a first preset threshold and whose line segment lengths are greater than a second preset threshold are extracted as candidate feature points. By using a cross-frame feature point matching algorithm, a spatial correspondence is established between candidate feature points in two adjacent frames of images, generating a set of feature point matching pairs. Based on the set of feature point matching pairs and the flight attitude sensor data of the UAV, the coordinate values ​​of the feature points in three-dimensional space are calculated using the principle of triangulation. Clustering of feature point coordinates belonging to the same transmission tower or conductor, and fitting to generate a three-dimensional contour point set of the transmission tower and a spatial curve equation of the conductor; By integrating the three-dimensional contour point set of all transmission towers with the spatial curve equation of the conductors, a three-dimensional point cloud model describing the overall structure of the line is constructed.

3. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 2, characterized in that, The infrared thermal imaging image sequence is registered with the three-dimensional point cloud model to generate a composite sensing map that integrates geometric structure and thermal distribution status, including: The infrared thermal imaging image sequence is subjected to non-uniformity correction and temperature calibration to generate a standardized temperature matrix; Key points with obvious planar or cylindrical features are extracted from the three-dimensional point cloud model and used as reference control points for registration. The temperature values ​​in the standardized temperature matrix are mapped to the key points and their neighborhood space of the three-dimensional point cloud model through the feature projection algorithm, and a correlation table between temperature data and geometric coordinates is established. Spatial interpolation is performed on the temperature data in the association table to generate temperature field distribution data covering the entire surface of the three-dimensional point cloud model. The temperature field distribution data is overlaid with the three-dimensional point cloud model to generate the composite sensing map that simultaneously includes geometric coordinates, structure type, and temperature attributes.

4. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 3, characterized in that, The process involves calling a pre-trained visual navigation strategy network to perform flight path analysis on the composite perception map, generating a global flight trajectory and local obstacle avoidance path segments for the inspection task, including: The composite perception map is input into the encoder part of the visual navigation strategy network to extract the topological features of the transmission line, the temperature distribution features of the temperature anomaly area, and the contour features of obstacles in the environment. The topological features, temperature distribution features, and obstacle contour features are input into the decoder part of the visual navigation strategy network to generate an initial flight waypoint sequence. The initial flight waypoint sequence is subjected to connectivity verification. If there is a situation where the line of sight between waypoints is blocked by obstacles, a transition waypoint is inserted in the blocked area to generate a corrected flight waypoint sequence. Based on the corrected flight waypoint sequence, the global flight trajectory of the UAV in the long-distance cruise segment and the local obstacle avoidance path segment in the close-range fine observation segment are calculated respectively. The start and end points of the global flight trajectory are smoothly connected to the endpoints of the local obstacle avoidance path segments to form a seamless and complete flight path.

5. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 4, characterized in that, The initial waypoint sequence undergoes connectivity verification. If the line of sight between waypoints is obstructed by obstacles, transition waypoints are inserted within the obstructed areas to generate a corrected waypoint sequence, including: A line-of-sight vector is constructed using two adjacent waypoints in the initial flight waypoint sequence as endpoints; Using a ray casting algorithm, it is detected whether the line-of-sight vector collides or interferes with the outline of an obstacle in the composite perception map; If a collision interference is detected, the position coordinates and normal direction of the collision point are calculated, and a no-fly zone is delineated with the collision point as the center and a preset safety distance as the radius. On the boundary of the no-fly zone, select the tangent point with the smallest angle to the line-of-sight vector as the transition waypoint, and insert the transition waypoint between the original adjacent waypoints; Traverse all adjacent waypoint pairs in the initial waypoint sequence, repeatedly perform line-of-sight detection and transition waypoint insertion operations until all lines of sight are unobstructed, and output the corrected waypoint sequence.

6. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 5, characterized in that, Based on the connection between the global flight trajectory and the local obstacle avoidance path segment, a set of UAV inspection path planning and navigation instructions is generated, including flight attitude angles, flight speed, and shooting trigger points, comprising: Extract the geometric parameters of the straight and turning segments of the global flight trajectory, and calculate the pitch and roll angles of the UAV in cruise mode based on the geometric parameters, which are used as the reference flight attitude angles; The flight speed of the UAV is dynamically adjusted based on the rate of curvature change of the local obstacle avoidance path segment, so that the flight speed is inversely proportional to the rate of curvature change of the path segment. From the temperature anomaly points and key structural points in the composite sensing map, target points that need to be photographed are selected, and the best shooting position that can be directly facing the target points is determined on the global flight trajectory and local obstacle avoidance path segment. Set the shooting trigger point at each of the optimal shooting positions, and configure the corresponding shutter trigger timing and gimbal pointing angle; The calculated flight attitude angle, the adjusted flight speed, and all the shooting trigger points and their parameters are encoded into the UAV inspection path planning and navigation instruction set.

7. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 6, characterized in that, From the temperature anomalies and key structural points in the composite sensing map, target points requiring focused imaging are selected, and the optimal imaging position directly facing the target points is determined on the global flight trajectory and local obstacle avoidance path segments, including: Traverse all temperature anomaly points in the composite sensing map, mark points whose temperature values ​​exceed the preset fault threshold as primary target points, and mark points whose temperature values ​​are within the warning range as secondary target points; All insulator string points, conductor crimping points, and hardware connection points of the transmission towers are extracted from the three-dimensional point cloud model as key structural points. The primary target points, secondary target points, and structural key points are deduplicated and merged to form a unified set of target points. For each target point in the target point set, the intersection of the inverted cone with the target point as the center and the preset field of view as the half-aperture angle with the global flight trajectory and the local obstacle avoidance path segment is calculated, and the position closest to the target point in the intersection set is taken as the best shooting position. If a target point cannot find a valid intersection on any path, then the target point is removed from the set of target points.

8. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 7, characterized in that, The visual navigation strategy network is trained through the following steps: Acquire composite sensing map samples of multiple historical inspection tasks and corresponding manual operation trajectory records. The composite sensing map samples include spatial information of line structure, temperature anomalies and obstacles, and the manual operation trajectory records are used as training labels. A neural network model consisting of an encoder, an attention mechanism module, and a decoder is constructed. The encoder is used to extract features from the composite sensing map, the attention mechanism module is used to enhance the feature weights of temperature anomaly areas and obstacles, and the decoder is used to generate a predicted waypoint sequence. The composite sensing map samples are input into the neural network model, and the mean square error between the predicted waypoint sequence output by the model and the trajectory sequence of the corresponding training label is calculated as the initial loss function. Based on the initial loss function, a penalty term is introduced to impose additional losses on waypoints that collide with obstacles in the predicted trajectory and waypoints that are missed in observing key target points, thus forming a multi-objective joint loss function; Using the backpropagation algorithm, the parameters of the neural network model are updated based on the multi-objective joint loss function until the average error between the predicted trajectory and the training label on the validation set is lower than a preset convergence threshold, thus completing the training and obtaining the pre-trained visual navigation policy network.

9. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 8, characterized in that, It also includes a path replanning step based on real-time visual feedback: During the flight of the UAV according to the UAV inspection path planning and navigation instruction set, the current field of view image collected by the airborne camera is acquired in real time; The current field-of-view image is compared with the prior image at the corresponding location in the composite perception map to identify the displacement of newly added unknown obstacles or existing obstacles. If an unknown obstacle or obstacle displacement is detected, the visual navigation strategy network is invoked again to generate a temporary avoidance path segment, starting from the current drone position and targeting the nearest path node. The temporary avoidance path segment is spliced ​​and replaced with the original planned path to generate an inspection path planning and navigation instruction set that is updated in real time. Control the drone to switch and execute the real-time updated inspection path planning and navigation instruction set until it bypasses the obstacle and returns to the original planned path.

10. The method for path planning and navigation of UAV inspection of power transmission lines based on visual intelligence according to claim 9, characterized in that, The step of comparing the feature differences between the current field-of-view image and the prior image at the corresponding location in the composite sensing map to identify the displacement of newly added unknown obstacles or existing obstacles includes: Extract local prior images of the area surrounding the current UAV location from the composite sensing map. The local prior images include known towers, power lines, and background environmental features. Scale-invariant feature transformation is performed on the current field-of-view image and the local prior image respectively, and feature descriptors of the two sets of images are extracted. The feature descriptor matching algorithm is used to calculate the feature matching logarithm and matching error between two sets of images. If the number of matching pairs is lower than a preset matching threshold or the matching error is higher than a preset error threshold, then it is determined that there is a visually inconsistent region in the current field of view. Connectivity analysis and contour fitting are performed on the visually inconsistent regions to identify the bounding rectangles of suspected new obstacles or existing obstacles that have been displaced, and their position coordinates and size information are output.

Images