Intelligent obstacle detection and avoidance method for power transmission line inspection unmanned aerial vehicle

Abstract

The invention discloses a power transmission line inspection unmanned aerial vehicle obstacle intelligent detection and obstacle avoidance method. The method comprises the steps that multi-source sensing data is acquired, and alignment is completed through calibration and timestamp matching; heterogeneous data preprocessing and feature enhancement; constructing a high-precision environment fusing a geometric structure and a semantic tag, mapping a two-dimensional target detection result output by the recognition network to a three-dimensional coordinate system through spatial transformation, and fusing the two-dimensional target detection result with a point cloud structure to construct a semantic occupation grid map; performing preliminary route planning according to a preset power grid topological structure and task coverage requirements, and generating a barrier-free flight path covering the whole inspection area; reinforcing learning of a dynamic obstacle avoidance strategy; track dynamic reconstruction and energy consumption optimization scheduling are carried out; the technical problems that an existing technical system has defects in the aspects of obstacle recognition accuracy, complex environment adaptability, data fusion capacity and obstacle avoidance strategy intelligence, and the requirements for high-reliability, low-energy-consumption and high-efficiency unmanned aerial vehicle power transmission line inspection are difficult to meet are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicles, and particularly relates to a power transmission line inspection unmanned aerial vehicle obstacle intelligent detection and obstacle avoidance method. BACKGROUND

[0002] With the continuous expansion of the scale of the power system and the increasing complexity of the power transmission line, in order to ensure the stable operation and power supply safety of the power transmission line, the inspection technology based on unmanned aerial vehicles (UAV) gradually becomes a new means to replace manual and helicopter inspection. Unmanned aerial vehicles have obtained wide attention and preliminary application in the field of power inspection due to their flexibility, efficiency and low risk. However, in actual operation, the detection and avoidance of obstacles are still one of the key bottlenecks restricting its wide promotion, especially in complex mountainous, forest areas, cross tower positions and other unstructured environments, the uncertainty of obstacle distribution is high, the types are complex, and higher requirements are put forward for the environmental perception and flight path control ability of unmanned aerial vehicles.

[0003] At present, the mainstream obstacle detection methods can be roughly divided into traditional image processing methods, machine learning-based methods and some early application of deep learning methods. The traditional image processing method mainly relies on edge detection, shape recognition and texture analysis and other hand-crafted feature extraction techniques, and typical algorithms include Sobel, Canny edge operator, Hough transform, etc. This kind of method is limited by image clarity and lighting conditions, and is prone to recognition errors in strong light, shadow shielding or complex background conditions, and has poor adaptability to shape changes and scale transformations of obstacles, resulting in high false alarm rate and missed detection rate, especially when there are dynamic blur, angle distortion and other problems in the image during the flight of the unmanned aerial vehicle, the robustness of the traditional method decreases significantly.

[0004] In order to overcome the above problems, researchers began to introduce support vector machines (SVM), K nearest neighbors (KNN) and other classic machine learning algorithms for obstacle recognition. By training a classifier to recognize the target area in the image, it has a certain generalization ability, and its performance depends on a large number of high-quality labeled samples, which is difficult to adapt to the situation of image distribution diversity and inconsistent obstacle shape in the actual environment, and the training and reasoning process of the model is difficult to be efficiently deployed on the resource-limited embedded platform, increasing the difficulty of system implementation.

[0005] In recent years, target detection algorithms based on deep learning have been gradually applied to unmanned aerial vehicle image recognition tasks, such as using convolutional neural networks (CNN) to achieve end-to-end obstacle recognition. However, existing methods mainly focus on feature extraction in the image domain, ignoring data fusion between multiple sensors. For example, the spatial distance information of laser radar and image semantic information cannot be effectively combined, resulting in insufficient recognition and positioning ability in three-dimensional space. Existing deep learning networks often use single-scale feature extraction, which is difficult to balance detail detection and global scene understanding. The detection ability of occluded scenes, multiple target overlaps, and small size targets is still limited. The obstacle avoidance decision module still has obvious shortcomings. Existing methods are mostly based on preset rules, such as setting fixed safety distances or simple geometric path bypass strategies, which are convenient to implement but lack flexibility and cannot dynamically respond to sudden obstacles in complex environments. Research based on reinforcement learning is still in its early stages and has problems such as unstable training, high state space dimension, and insufficient generalization ability, making it difficult to be directly deployed in actual inspection environments.

[0006] The current technical system has deficiencies in obstacle recognition accuracy, complex environment adaptability, data fusion capability, and intelligent obstacle avoidance strategy, making it difficult to meet the demand for high reliability, low energy consumption, and high efficiency of unmanned aerial vehicle power line inspection. Therefore, a complete systematic solution that integrates multi-source perception, deep visual understanding, and intelligent obstacle avoidance decision is needed to improve the autonomous flight and safe operation capabilities of unmanned aerial vehicles in power grid inspection tasks. SUMMARY

[0007] The technical problem to be solved by the present application is to provide an intelligent obstacle detection and avoidance method for power line inspection unmanned aerial vehicles, which solves the technical problems of the current technical system in obstacle recognition accuracy, complex environment adaptability, data fusion capability, and intelligent obstacle avoidance strategy, making it difficult to meet the demand for high reliability, low energy consumption, and high efficiency of unmanned aerial vehicle power line inspection.

[0008] The technical solution of the present application is:

[0009] An intelligent obstacle detection and avoidance method for power line inspection unmanned aerial vehicles, the method comprising:

[0010] Step 1, multi-source perception data acquisition, alignment is completed through calibration and timestamp matching;

[0011] Step 2, heterogeneous data preprocessing and feature enhancement: bilateral filtering is used on image data to remove noise while maintaining edge clarity; RANSAC plane fitting method is applied to point cloud data to filter out ground interference points, and voxel filtering is used to unify the spatial resolution;

[0012] Step 3, construct high-precision environment with fused geometry and semantic labels, map the two-dimensional target detection results output by the recognition network to a three-dimensional coordinate system through spatial transformation, and fuse with the point cloud structure to construct a semantic occupancy grid map;

[0013] Step 4, preliminary flight path planning according to the preset power grid topology structure and task coverage requirements, generating a flight path without obstacles and covering the entire inspection area;

[0014] Step 5, reinforcement learning dynamic obstacle avoidance strategy;

[0015] Step 6, dynamic reconstruction of flight path and optimization of energy consumption scheduling.

[0016] The multi-source perception data includes the perception system carried by the unmanned aerial vehicle, including an RGB camera, a LiDAR, a GPS, and an IMU, each of which outputs heterogeneous data at different frequencies.

[0017] The alignment method through calibration and timestamp matching includes:

[0018] Camera and LiDAR extrinsic calibration: through a calibration board or a corner reflector, feature points are extracted in the camera image, and corresponding three-dimensional points are extracted in the LiDAR point cloud, and the transformation matrix from the camera to the LiDAR is calculated by minimizing the re-projection error or point-plane error optimization;

[0019] LiDAR and IMU extrinsic calibration: obtained by using a tool platform or a calibration algorithm;

[0020] GPS and IMU coordinate system alignment: through the alignment of the coordinate origins of the two in the initial static state, the IMU drift is corrected by filtering;

[0021] The original information output by each sensor in its own coordinate system is uniformly mapped to the world coordinate system , the pixel point in the image , the corresponding space point is converted by the following formula:

[0022]

[0023] wherein is the camera intrinsic matrix, is the depth value, is the camera-to-radar extrinsic matrix, is the radar-to-GPS / IMU world coordinate system transformation, is the three-dimensional position of the pixel point in the world coordinate, the acceleration and angular velocity output by the IMU are fused through pre-integration and extended Kalman filtering (EKF), and the drift error of the GPS is corrected, and an accurate time sequence flight trajectory is constructed;

[0024] The method of synchronizing all data streams to a unified timeline includes:

[0025] Time stamping all sensors with a synchronized clock;

[0026] Linear interpolation or IMU pre-integration, for data with small time difference, including IMU and LiDAR, align low frequency data to high frequency time point by interpolation method;

[0027] Taking image frame as the main index, finding the nearest time-stamped point cloud frame and IMU state, and constructing a synchronized triple 〈image, point cloud, pose〉;

[0028] Using an extended Kalman filter (EKF) to fuse high-speed IMU data and low-speed GPS data, outputting stable and high-frequency position and attitude estimation as the time axis reference.

[0029] The method of using bilateral filtering to remove noise while maintaining edge sharpness for image data includes:

[0030] Bilateral filtering preserves image edges while achieving smoothing. The filter considers not only the spatial distance but also the pixel value difference between neighboring pixels when determining the weight of a pixel; the mathematical expression is:

[0031]

[0032] wherein is the control of spatial distance; is the control of pixel value difference; is the normalization factor; and represent the pixel position and the pixel position in the field of the current pixel to be filtered, respectively; is the filtered pixel value at position x; is the original pixel value at position ;

[0033] The method of applying RANSAC plane fitting to filter ground interference points and combining voxel filtering to unify spatial resolution includes:

[0034] Using the RANSAC algorithm to model and remove the ground plane, randomly sampling 3 points to fit a plane model Calculating the distance of all points to the plane; points with a distance below the threshold are considered "inliers", and the number of inliers is counted; repeat several iterations to select the model with the most inliers as the best plane; delete the points on the plane;

[0035] The voxel filtering method is used for downsampling processing of the point cloud, and the space is divided into fixed-size cubic voxels; the centroid of all points in each voxel is calculated or the middlemost point is selected as a representative point; other points in the voxel are discarded, and only the representative point is retained, thereby reducing the number of points and unifying the density;

[0036] The image feature enhancement processing includes histogram equalization, Gamma correction and local contrast enhancement, so that the edges and texture details are more easily extracted by the convolution network, and the density features of the point cloud data are defined as the number of points in a unit sphere:

[0037]

[0038] wherein is a point cloud set, is a current point of the cloud set, r is a neighborhood radius, is any one of the neighboring points in the point cloud set and the distance between the neighboring points is within the neighborhood radius, all sensor timestamps are aligned to a uniform time sequence to ensure the synchronization of multi-source data.

[0039] The construction method of the semantic occupancy grid map comprises the following steps: projecting the laser radar point cloud to the image plane to obtain the depth value of the corresponding pixel in the image, thereby constructing the transformation relationship from the pixel to the spatial point, for each pixel in the detection box , combining the depth value , and using the camera intrinsic matrix to inversely calculate the three-dimensional point in the camera coordinate system , and then mapping to the position in the unified world coordinate system via the extrinsic matrix, assigning a unified semantic label to the three-dimensional point set belonging to the same target detection box, and projecting the three-dimensional point set belonging to the same target detection box to the three-dimensional space grid, thereby generating the occupancy grid map with the semantic probability distribution.

[0040] The flight path generation method of step 4 comprises the following steps: in the path generation stage, the flight area is discretized into a three-dimensional discrete grid map , each grid unit corresponds to a cube in the actual space, and is attached with whether it is passable, obstacle probability and semantic type attribute information, the path planning algorithm A* runs on the grid map, takes the grid where the starting point and the ending point are located as the start node and the end node, searches for a path with the minimum cost and continuous feasibility, and the grid with high obstacle probability or semantic is marked as impassable, thereby guiding the path to automatically avoid the potential risk area, and realizing safe and efficient route planning.

[0041] The path planning algorithm A* represents the entire flight space as a three-dimensional discrete grid map Each grid cell contains obstacle occupancy probability, semantic category, and flight restriction information, defining a path cost function:

[0042]

[0043] n is the current search node. This represents the cumulative cost from the starting point to the current node. For the heuristic cost estimate from the current node to the destination; This is a path curvature penalty term that constrains the continuity and controllability of the path. This is the weighting coefficient for the curvature term, used to balance path length and flight feasibility;

[0044] During the search process, starting from the starting grid, the search expands to neighboring passable grid nodes in six or twenty-six directions, and stores visited nodes and their costs in a priority queue. If a grid has a barrier occupancy probability... If the safety threshold is exceeded, the path is marked as impassable and the branch is automatically skipped.

[0045] Use the graph search algorithm A* to find the path with the minimum cost in a spatial raster graph. The cost function is further defined as:

[0046]

[0047] in From the starting point to The cumulative cost (distance / time). From Heuristic estimation of the destination (Euclidean distance). The curvature penalty function reflects the smoothness of the trajectory. Weights are adjusted to improve trajectory controllability; path constraints include avoiding all grid cells with a probability of [missing information]. Obstacle units and minimum flight altitude; maximum angle change.

[0048] Methods for reinforcement learning-based dynamic obstacle avoidance strategies include:

[0049] An online obstacle avoidance strategy based on a deep Q-network is adopted, and the environment state is defined as follows:

[0050]

[0051] Current position For velocity vectors, The relative positions of nearby obstacles The location of the target point;

[0052] Action set Including acceleration / deceleration, left / right turn and up / down float;

[0053] The target is to maximize the strategy expected value function:

[0054]

[0055] The obstacle avoidance action at time t; The system state information at time t is represented; The reward after performing the action at time t is represented; The discount factor is;

[0056] The reward function is designed as:

[0057]

[0058] The reward function calculation result is: The current-to-nearest-obstacle distance is; The current acceleration is; The indication factor of whether close to the target point is; The weight hyperparameter is generated by continuous interaction with the simulation environment, and a highly adaptive obstacle avoidance action selection strategy for dynamic scenes is generated.

[0059] The method for trajectory dynamic reconstruction and energy consumption optimization scheduling includes: designing an energy consumption sensitive trajectory optimization, the total energy consumption E is estimated by a dynamics model, and is defined as follows:

[0060]

[0061] In the formula: , The initial and final times are; The velocity vector is; The acceleration vector is; The velocity and acceleration energy consumption coefficients are, and the trajectory smoothing constraint generates a controllable trajectory point sequence through a Bezier curve or spline interpolation , which satisfies:

[0062] ;

[0063] The i-th actual generated trajectory point after optimization is; The i-th reference trajectory point of the initial planning is; The position deviation threshold is; The angle deviation at the i-th trajectory point is; The maximum allowed value of the angle deviation is;

[0064] Ensure obstacle avoidance safety, shorten flight time and energy consumption, and achieve overall efficiency optimization.

[0065] The beneficial effects of the present application are:

[0066] The present application improves the environmental perception accuracy, path planning intelligence and overall operation efficiency of the system, specifically including:

[0067] In view of the problem of insufficient obstacle detection accuracy in complex unstructured environment, a deep learning structure combining multi-scale image feature extraction and attention mechanism is proposed, which automatically learns the spatial, texture, edge and other multi-dimensional information of obstacles through the network, and improves the recognition ability of weak targets, small foreign objects and occluded obstacles. By introducing a multi-source heterogeneous data fusion strategy, the three-dimensional point cloud of laser radar, spatial positioning information of GPS and IMU and image data are cooperatively modeled to construct a high-precision three-dimensional obstacle distribution map, realizing the unification of spatial positioning and semantic recognition.

[0068] In view of the problem that the existing path planning and obstacle avoidance strategy is rigid, has weak generalization ability and cannot dynamically adapt to new environment, the present application generates an obstacle avoidance strategy through continuous interaction between the unmanned aerial vehicle and the environment, replaces the static rule system, realizes rapid response and dynamic flight path adjustment for unknown obstacles. At the same time, in order to enhance the practicality and safety of obstacle avoidance behavior, the present application specially designs a reward function system, which integrates the success rate of obstacle avoidance, flight stability, minimization of energy consumption and other targets into the strategy learning process, so that the unmanned aerial vehicle can not only avoid obstacles, but also maintain flight efficiency and stable operation of equipment.

[0069] In view of the deficiency of the existing system in multi-task trade-off ability, the present application establishes a task-driven flight path adjustment and energy consumption control mechanism to realize coordination and unification between obstacle avoidance behavior and path optimality. Through joint analysis of the current flight state, obstacle distribution and task area, real-time optimization of flight path is realized to reduce path redundancy, shorten operation time and reduce energy consumption.

[0070] Characteristics of the present application:

[0071] The present application is based on a multi-modal perception fusion method based on unified spatial mapping mechanism: a cross-modal semantic perception model combining image-point cloud-positioning information is proposed, a unified spatial reference system is constructed by using an external parameter matrix, and the two-dimensional image target recognition result is seamlessly embedded in the three-dimensional point cloud environment, solving the problem of spatial separation and time asynchronization of multi-source information, significantly improving the spatial position estimation accuracy and recognition semantic reliability of obstacles, and compared with the traditional single visual path, the present method has higher environmental modeling integrity and actual adaptability.

[0072] The application adopts a multi-scale attention mechanism guided deep obstacle recognition network design: in view of the problem that the target scale changes greatly and the feature is locally strong in the typical scene of the power transmission line, a multi-scale convolution feature extraction structure and a channel attention fusion mechanism are proposed, the network design supports the extraction and weighted fusion of obstacle features under different resolutions, so that the model considers the global view and local sensitivity, especially improves the detection ability of small targets such as broken conductor, thin branches, air plastic bags and other microstructure obstacles, and has superior fine recognition performance.

[0073] The application adopts a rule planning and reinforcement learning obstacle avoidance dual strategy fusion: the static task driven rule path generation and the dynamic strategy optimized deep reinforcement learning model are structurally fused, the evolution of the flight path planning from 'global static optimization' to 'local dynamic adjustment' is realized. The rule path provides the integrity guarantee of the inspection, the DQN strategy realizes the flexible avoidance of the sudden risk, and the two complement each other in the task execution, so that the behavior robustness and the task completion rate of the system in the complex dynamic environment are significantly improved.

[0074] The application adopts a flight optimization strategy combining energy consumption constraint and trajectory smoothing: in view of the problem that the endurance is limited in the process of unmanned aerial vehicle inspection, the energy consumption model is directly introduced into the obstacle avoidance strategy learning and trajectory reconstruction process, the speed / acceleration penalty term in the reward function is used to constrain the flight aggressiveness, and the smoothing loss function is introduced in the flight path planning to adjust.

[0075] The current technical system has the problems of low obstacle recognition accuracy, poor adaptability to complex environment, poor data fusion ability and poor intelligence of obstacle avoidance strategy, and it is difficult to meet the demand of high reliability, low energy consumption and high efficiency of unmanned aerial vehicle power transmission line inspection. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 It is the flowchart of the application;

[0077] Figure 2 It is the execution flowchart of the deep Q network of the application;

[0078] Figure 3 It is the accuracy comparison diagram of the method of the application and the traditional method. DETAILED DESCRIPTION

[0079] The application is based on multi-source heterogeneous perception data fusion, deep multi-scale visual understanding and reinforcement learning path decision, through constructing a unified three-dimensional semantic map and a double-layer strategy control structure, realizing a high-robustness, high-real-time, high-energy-efficiency autonomous flight obstacle avoidance system, and the flowchart is as shown in Figure 1 .

[0080] Step 1, multi-source perception data acquisition and space-time alignment

[0081] By fusing vision, spatial ranging and pose information, a complete and synchronous perception dataset is constructed. The perception system carried by the UAV includes RGB camera, LiDAR, GPS and IMU, each of which outputs heterogeneous data at different frequencies, which need to be aligned through calibration and timestamp matching.

[0082] In order to map data from different sensors to the same reference coordinate system (usually IMU or GPS world coordinate system), spatial transformation calibration between sensors needs to be completed in sequence.

[0083] Camera and LiDAR extrinsic calibration extracts feature points in camera images and corresponding three-dimensional points in LiDAR point clouds through calibration board or corner reflector, and calculates the transformation matrix from camera to LiDAR by minimizing the re-projection error or point-plane error optimization.

[0084] LiDAR and IMU extrinsic calibration is obtained by using tooling platform or calibration algorithm (such as hand-eye calibration method).

[0085] GPS and IMU coordinate system alignment generally aligns the coordinate origins of the two systems in the initial static state, and the IMU drift is corrected by subsequent filtering.

[0086] The original information output by each sensor in its own coordinate system needs to be mapped to the world coordinate system . For a pixel point in the image , its corresponding spatial point can be converted by the following formula:

[0087]

[0088] Where is the camera intrinsic matrix, is the depth value, which can be provided by LiDAR projection, is the camera-to-radar extrinsic matrix, is the radar-to-GPS / IMU world coordinate system transformation, is the three-dimensional position of the pixel point in the world coordinate system. The acceleration and angular velocity output by the IMU are fused through pre-integration and extended Kalman filtering (EKF) to correct the drift error of the GPS and construct an accurate time sequence flight trajectory.

[0089] Due to different sampling frequencies of each sensor, all data streams need to be synchronized to a unified timeline.

[0090] Unified clock source: the system uses a synchronous clock (such as GPS PPS or internal clock of the main control board) to timestamp all sensors;

[0091] Linear interpolation or IMU pre-integration, for data with small time difference (such as IMU and LiDAR), can align low-frequency data to high-frequency time points through interpolation method;

[0092] Image and point cloud synchronization, taking image frame as the main index, finding the nearest time-stamped point cloud frame and IMU state, and constructing a synchronous triple 〈image, point cloud, pose〉;

[0093] IMU-GPS fusion time series construction, using extended Kalman filter (EKF) to fuse high-speed IMU data and low-speed GPS data, outputting stable and high-frequency position and attitude estimation as the time axis reference.

[0094] Step 2, heterogeneous data preprocessing and feature enhancement

[0095] The main goal of this step is to improve data quality and feature discrimination ability, and to ensure the stability of the input of the subsequent recognition model. For image data, bilateral filtering is used to remove noise while maintaining edge sharpness; in point cloud data, RANSAC plane fitting method is applied to filter out ground interference points, and voxel filter (Voxel Grid) is used to unify the spatial resolution.

[0096] Bilateral filtering to suppress noise and preserve edges:

[0097] Image data often contains interference such as light changes and photosensitive noise, so bilateral filtering (Bilateral Filter) is used for noise reduction. Unlike traditional Gaussian filtering, bilateral filtering preserves image edges while smoothing. The filter considers not only the spatial distance but also the pixel value difference between the pixel and its neighborhood; the mathematical expression is:

[0098]

[0099] where control the influence of spatial distance; control the influence of pixel value difference; is the normalization factor; and represent the pixel position and the pixel position in the field of the current pixel to be filtered; is the filtered pixel value at position x; is the original pixel value at position Based on the noise intensity and edge characteristics of the image itself, reasonable settings for and can preserve image edge contours while smoothing texture areas, improving the robustness of subsequent feature extraction. Specifically, for For the image, it is set to the size level of "small texture / noise"; for the image, it is set to the fluctuation amplitude of the pixel value of the noise. In the actual experiment process, first, the initial setting is made according to the noise intensity, and then the improvement is made according to the edge definition in combination with the above setting principle.

[0100] RANSAC plane fitting removes ground interference:

[0101] In the point cloud data, the ground or platform background may introduce a large number of redundant interference points, which affect the obstacle identification. The RANSAC (Random Sample Consensus) algorithm is used for ground plane modeling and removal, 3 points are randomly sampled, and a plane model is fitted The distance of all points to the plane is calculated; the points with a distance lower than the threshold are regarded as "inliers", and the number of inliers is counted; the model with the most inliers is selected as the best plane after several iterations; the points on the plane (i.e. ground interference points) are deleted. The advantage of this method is that it has strong robustness to outliers and non-structured areas, and is suitable for outdoor irregular ground modeling.

[0102] Voxel filter (Voxel Grid) downsampling

[0103] The original point cloud often has uneven density and high resolution, which not only affects the processing efficiency, but also causes feature overfitting. The voxel filter method is used to downsample the point cloud, which divides the space into fixed-size cubic voxels; the centroid of all points in each voxel or the most central point is calculated as the representative point; the other points in the voxel are discarded, and only the representative point is retained, thereby reducing the number of points and unifying the density.

[0104] The voxel size R is adjusted according to the actual resolution requirement; the result point cloud has a more uniform spatial distribution, and the boundary shape and structure are well preserved. Voxel filtering not only reduces the computational burden, but also provides a homogeneous basis for subsequent feature extraction (such as density estimation and curvature analysis).

[0105] Image feature enhancement processing includes histogram equalization, Gamma correction and local contrast enhancement, which makes the edge and texture details easier to be extracted by convolutional network. The density feature of point cloud data is defined as the number of points in a unit sphere:

[0106]

[0107] where is the point cloud set, is the current point of the cloud set, r is the neighborhood radius, is any one of the neighboring points in the point cloud set with a distance within the neighborhood radius. All sensor timestamps are aligned to a uniform time sequence​ , to ensure the synchronization of multi-source data.

[0108] The final output of the preprocessed data is a triple of synchronized image frames, filtered point clouds, and pose trajectory sequences.

[0109] Step 3: Three-dimensional semantic environment reconstruction and perception fusion model construction

[0110] In this core step, the system constructs a high-precision environment that integrates geometric structure and semantic labels. The two-dimensional target detection results output by the recognition network (such as obstacle bounding boxes) are mapped to the three-dimensional coordinate system through spatial transformation, and are integrated with point cloud structure to construct a semantic occupancy grid map.

[0111] To achieve accurate mapping of two-dimensional target detection results to three-dimensional coordinates, the key pixel points within each detection box region in the camera image need to be processed for depth completion. By projecting the laser radar point cloud onto the image plane, the depth values of the corresponding pixels in the image are obtained, thereby constructing the transformation relationship from pixels to spatial points. For each pixel in the detection box , combined with its depth value , the camera intrinsic matrix can be used to inversely calculate the three-dimensional point in the camera coordinate system , which is then mapped to the position in the unified world coordinate system via the extrinsic matrix . All three-dimensional points belonging to the same target detection box are assigned a unified semantic label and integrated with the geometric structure information in the point cloud, projected into a three-dimensional space grid, and finally a semantic probability distribution occupancy grid map is generated. This map not only retains the geometric boundary features of obstacles, but also embeds semantic category information, providing richer and more interpretable environment perception support for subsequent path planning and decision-making modules.

[0112] Definition of 3D grid map

[0113]

[0114] where xyz represents the spatial grid index (Voxel Index), P is the probability formula symbol, i.e., the i-th grid in the three-dimensional space divided by unit length, is the image semantic score (such as from YOLO or SegNet), is the local density and curvature feature of the point cloud, is the Sigmoid activation function; is a weight hyper-parameter used to balance the contributions of vision and geometry. After the semantic map is constructed, various obstacles (e.g. grass, hanging objects, broken lines) are spatially localized and annotated with attributes, providing a perception basis for path planning and obstacle avoidance.

[0115] Step 4, Rule-based path generation strategy based on topological constraints

[0116] Based on the perception and mapping, the system generates an initial flight path that is obstacle-free and covers the entire inspection area according to the pre-defined power grid topology and task coverage requirements.

[0117] The spatial grid map is not only used to represent the occupancy state and semantic information of each region in the environment, but also serves as the basic data structure for flight path planning. In the path generation phase, the flight area is discretized into a three-dimensional discrete grid map , where each grid cell corresponds to a small cube in the actual space and is associated with attributes such as passability, obstacle probability, semantic type, etc. The path planning algorithm (e.g. A*) runs on this grid map, taking the grid nodes at the start and end points as the start and end nodes, and considering the occupancy probability, obstacle type, flight height, etc. of each cell to search for a path with the minimum cost and continuous feasibility. Grids with high obstacle probability or semantic as high-risk areas will be marked as impassable, guiding the path to automatically avoid potential risk areas and achieve safe and efficient route planning.

[0118] A* (heuristic cost search) algorithm is used for three-dimensional path planning. The entire flight space is represented as a three-dimensional discrete grid map , where each grid cell contains obstacle occupancy probability, semantic category, flight restriction information, etc. Based on this, the path cost function is defined as:

[0119]

[0120] where n is the current search node (i.e. a grid cell); is the cumulative cost (e.g. flight distance, time) from the start point to the current node; is the heuristic estimated cost from the current node to the end point, usually calculated using the Euclidean distance; is the path curvature penalty term, which constrains the continuity and controllability of the path; is the weight coefficient of the curvature term, used to balance the path length and flight feasibility.

[0121] During the search process, the system starts from the grid at the start point and expands the adjacent passable grid nodes in six or twenty-six directions, and stores the visited nodes and their costs in a priority queue. If the obstacle occupancy probability of a certain grid exceeds the safety threshold (e.g. 0.7), it is marked as impassable and the path branch is automatically skipped.

[0122] Use a graph search algorithm (A*) to find the minimum-cost path in a spatial raster graph. The cost function is further defined as:

[0123]

[0124] in From the starting point to The cumulative cost (distance / time). From Heuristic estimation of the destination (Euclidean distance). The curvature penalty function reflects the smoothness of the trajectory. Adjust weights for trajectory controllability. Path constraints include avoiding all grid cells with a probability of [missing information]. Obstacle units and minimum flight altitude requirements Maximum change in rotation angle wait.

[0125] Step 5: Reinforcement learning dynamic obstacle avoidance strategy modeling

[0126] Considering the unpredictability of dynamic obstacles along the path, the system introduces online obstacle avoidance strategy learning based on Deep Q-Network (DQN). The flowchart of DQN is as follows: Figure 2 As shown. The environmental state is defined as follows:

[0127]

[0128] in Current position For velocity vector, The relative positions of nearby obstacles. The location of the target point.

[0129] Action set Includes acceleration / deceleration, left / right turns, and up / down movement.

[0130] The objective is to maximize the expected value function of the strategy.

[0131]

[0132] in, Let t be the obstacle avoidance action. This represents the system state information at time t; The reward after performing the action at time t is obtained from the following reward function; This is the discount factor, which is usually between 0 and 1.

[0133] The reward function is designed as follows:

[0134]

[0135] is the reward function calculation result, wherein is the current-to-nearest obstacle distance, is the current acceleration, is an indication factor of whether close to a target point, is a weight hyperparameter. Through continuous interaction with a simulation environment, a highly adaptive obstacle avoidance action selection strategy for a dynamic scene is generated.

[0136] Step 6, track dynamic reconstruction and energy consumption optimization scheduling

[0137] In order to realize minimum energy consumption flight on the basis of safe obstacle avoidance, an energy consumption sensitive trajectory optimization module is designed. The total energy consumption E is estimated by a dynamics model, and is defined as follows:

[0138]

[0139] wherein , is the initial and final time, is the velocity vector, is the acceleration vector, is the velocity and acceleration energy consumption coefficient. The trajectory smoothing constraint generates a controllable track point sequence through a Bezier curve or spline interpolation , and meets:

[0140] .

[0141] wherein, is the i th actual generated track point after optimization; is the i th reference track point of initial planning; is a position deviation threshold value; is an angle deviation at the i th track point; is a maximum allowed value of the angle deviation.

[0142] While ensuring obstacle avoidance safety, the flight time and energy consumption are shortened as much as possible, and the overall efficiency of the system is optimized.

[0143] The method of the present application is subjected to experiments, and the experimental results are shown in Figure 3 . From Figure 3It can be seen that the algorithm proposed in the present application is significantly better than the traditional algorithm in terms of obstacle detection accuracy. Among all the types of obstacles tested, including trees, utility poles, weeds and buildings, the average accuracy of the algorithm of the present application reaches 0.84, while the traditional algorithm is only 0.775, showing a performance improvement of about 8.4%. From the standard deviation analysis, the two algorithms are 0.061 and 0.056 respectively, indicating that the algorithm of the present application performs more stably on various types of obstacles, without large fluctuations or optimization only for specific targets.

[0144] In specific categories, the recognition accuracy of the algorithm of the present application for weed-type obstacles is 0.92, which is much higher than the 0.85 of the traditional algorithm, showing that the method has stronger recognition ability when dealing with obstacles with complex texture and irregular shape. The detection accuracy of utility poles has relatively small improvement (from 0.72 to 0.78), but still maintains a stable improvement trend, indicating that it also has robustness in regular target detection. In addition, for buildings and trees, which are common structures in power transmission line scenes and have different shapes, the accuracy of the algorithm is 0.81 and 0.85 respectively, which is significantly improved compared with the traditional method (0.75 and 0.78).

Claims

1. A power line inspection unmanned aerial vehicle obstacle intelligent detection and obstacle avoidance method, characterized in that: The method comprises: Step 1, multi-source perception data acquisition, alignment is completed through calibration and timestamp matching; Step 2, heterogeneous data preprocessing and feature enhancement: bilateral filtering is adopted for image data to remove noise while maintaining edge definition; the RANSAC plane fitting method is applied to point cloud data to filter out ground interference points, and voxel filtering is combined to unify the spatial resolution; Step 3, constructing a high-precision environment that fuses geometric structure and semantic labels, mapping the two-dimensional target detection results output by the recognition network to the three-dimensional coordinate system through spatial transformation, and fusing with the point cloud structure to construct a semantic occupancy grid map; Step 4, preliminary route planning according to the preset power grid topology structure and task coverage requirements, generating a flight path without obstacles and covering the entire inspection area; Step 5, dynamic obstacle avoidance strategy of reinforcement learning; Step 6, dynamic reconstruction of flight path and optimization of energy consumption scheduling. 2.The method of claim 1, wherein the method further comprises: The multi-source perception data includes the perception system carried by the unmanned aerial vehicle, including an RGB camera, a laser radar LiDAR, a global positioning system GPS and an inertial measurement unit (IMU), each of which outputs heterogeneous data at different frequencies. 3.The method of claim 2, wherein the method further comprises: The method of alignment through calibration and timestamp matching comprises: Camera and LiDAR external parameter calibration: through a calibration board or a corner reflector, feature points are extracted in the camera image, and corresponding three-dimensional points are extracted in the LiDAR point cloud, and the transformation matrix from the camera to the LiDAR is calculated by minimizing the re-projection error or the point-plane error; LiDAR and IMU external parameter calibration: obtained by using a tool platform or a calibration algorithm; GPS and IMU coordinate system alignment: the coordinate origins of the two are aligned in the initial static state, and the IMU drift is corrected by filtering; The original information output by each sensor in the respective coordinate system is uniformly mapped to the world coordinate system , for a pixel point in the image , for a spatial point is converted by the following formula: ； wherein is a camera intrinsic matrix, is a depth value, is a camera to radar extrinsic matrix, is a radar to GPS / IMU world coordinate system transformation, is a three-dimensional position of a pixel point in the world coordinate, the acceleration and angular velocity output by the IMU are fused through pre-integration and extended Kalman filtering (EKF) to correct the drift error of the GPS and construct an accurate time sequence flight trajectory; The method of synchronizing all data streams to a unified timeline comprises: Using a synchronous clock to timestamp all sensors; Linear interpolation or IMU pre-integration: for data with small time differences, including IMU and LiDAR, the low-frequency data is aligned to the high-frequency time point through interpolation method; Taking image frames as the main index, finding the nearest timestamp point cloud frame and IMU state, and constructing a synchronous triple 〈image, point cloud, attitude〉; Using an extended Kalman filter (EKF) to fuse high-speed IMU data and low-speed GPS data, and outputting stable and high-frequency position and attitude estimation as the time axis reference.

4. The method of claim 1, wherein the method further comprises: The method of adopting bilateral filtering for image data to remove noise while maintaining edge definition comprises: Bilateral filtering preserves image edges while smoothing; the filter considers not only the spatial distance of a pixel, but also the pixel value difference with neighboring pixels; the mathematical expression is: ； wherein is to control the influence of the spatial distance; is to control the influence of the pixel value difference; is a normalization factor; and denote the pixel position and the pixel positions within the domain respectively, which are currently to be filtered; is the filtered pixel value at position x; is the original pixel value at position x. The method of applying the RANSAC plane fitting method to filter out ground interference points and combining voxel filtering to unify the spatial resolution comprises: RANSAC algorithm is used to model and remove the ground plane, three points are randomly sampled to fit a plane model The distance of all points to the plane is calculated; points with distance below a threshold are considered "inliers", the number of inliers is counted; the model with the most inliers is selected as the best plane after several iterations; points on the plane are removed; The voxel filtering method is used to downsample the point cloud, dividing the space into fixed-size cubic voxels; the centroid of all points in each voxel or the most central point is selected as the representative point; the other points in the voxel are discarded, only the representative point is retained, thereby reducing the number of points and unifying the density; The image feature enhancement processing includes histogram equalization, Gamma correction and local contrast enhancement, so that the edge and texture details are easier to be extracted by the convolution network, and the density features of the point cloud data is defined as the number of points in the unit sphere: ； in For point cloud collection, Let r be the current point in the cloud set, and r be the neighborhood radius. It is a point cloud set and All sensor timestamps are aligned to a uniform time series for any neighboring point within the neighborhood radius. This is to ensure the synchronization of data from multiple sources.

5. The method of claim 1, wherein the method further comprises: The construction method of the semantic occupancy grid map comprises: projecting a laser radar point cloud to an image plane to obtain a depth value of a corresponding pixel in the image, thereby constructing a transformation relationship from the pixel to a spatial point , combining the depth value , using a camera intrinsic matrix to inversely calculate a three-dimensional point in a camera coordinate system , and then mapping the three-dimensional point to a position in a unified world coordinate system via an extrinsic matrix, assigning a unified semantic label to a three-dimensional point set belonging to the same target detection frame, fusing geometric structure information in the point cloud, projecting into a three-dimensional space grid, and generating an occupancy grid map with a semantic probability distribution.

6. The method of claim 1, wherein the method further comprises: The method for generating the flight path in step 4 comprises: in a path generation stage, discretizing a flight area into a three-dimensional discrete grid map Each grid cell corresponds to a cube in the actual space, and is attached with attribute information such as whether it is passable, obstacle probability and semantic type; a path planning algorithm A* is run on the grid map, with the grid where the start point and the end point are located as start and end nodes, to search for a path with minimum cost and continuous feasibility; grids with high obstacle probability or semantics as high-risk areas are marked as impassable, thereby guiding the path to automatically avoid potential risk areas, and realizing safe and efficient flight path planning.

7. The method of claim 1, wherein the method further comprises: The path planning algorithm A* represents the entire flight space as a three-dimensional discrete grid map Each grid cell contains obstacle occupancy probability, semantic category and flight restriction information, and defines a path cost function: ； n is the current search node, is the cumulative cost from the start to the current node; is the heuristic estimated cost from the current node to the goal; is the path curvature penalty term, constraining the continuity and controllability of the path; is the weight coefficient of the curvature term, used to balance the path length and flight feasibility; In the search process, from the starting point, along the six or twenty-six directions to expand adjacent passable grid nodes, and the visited nodes and cost into the priority queue, if a grid obstacle occupancy probability exceeds the safety threshold, it is marked as impassable, and the path branch is automatically skipped; finding a path of minimum cost in a spatial raster map using the graph search algorithm A* the cost function is further defined as: ； wherein is the cumulative cost (distance / time) from the start point to , is the heuristic estimate (Euclidean distance) from to the end point, is the curvature penalty function, reflecting the smoothness of the trajectory, is the controllability adjustment weight of the trajectory, the path constraints include avoiding all obstacle cells with probability and satisfying the minimum flight height; maximum corner change amount. 8.The method of claim 1, wherein the method further comprises: determining a distance between the UAV and the obstacle; and determining a direction of the obstacle based on the distance between the UAV and the obstacle. The method of reinforcement learning dynamic obstacle avoidance strategy comprises: An online obstacle avoidance strategy learning based on deep Q network is adopted, and the environment state is defined as: ； is the current position, is the velocity vector, is the relative position of the neighboring obstacle, is the target point position; set of actions including acceleration / deceleration, left / right turn, and up / down float; The objective is to maximize the policy expected value function: ； an obstacle avoidance action for time t; system state information for time t; a reward for performing an action at time t; a discount factor; The reward function is designed as: ； is the reward function calculation result, wherein is the current-to-nearest-obstacle distance, is the current acceleration, is an indication factor of whether close to the target point, is a weight hyperparameter, and a highly adaptive obstacle avoidance action selection strategy for dynamic scenes is generated by continuously interacting with the simulation environment. 9.The method of claim 1, wherein the method further comprises: determining a distance between the UAV and the obstacle; and determining a direction of the obstacle based on the distance between the UAV and the obstacle. The method of flight path dynamic reconstruction and energy consumption optimization scheduling includes: designing energy consumption sensitive trajectory optimization, the total energy consumption E is estimated by the dynamics model, and is defined as follows: ； wherein: , are initial and final times, is the velocity vector, is the acceleration vector, is the velocity and acceleration energy consumption coefficient, trajectory smoothing constraints generate a sequence of waypoints with good controllability through Bezier curves or spline interpolation satisfying: ； is the i-th actual generated waypoint optimized; is the i-th reference waypoint of the initial plan; is a position deviation threshold value; is an angle deviation at the i-th waypoint; is a maximum allowed value for the angle deviation; While ensuring obstacle avoidance safety, shorten the flight time and energy consumption, and achieve overall efficiency optimization.

Images