Unmanned aerial vehicle high-voltage line target point selection and real-time positioning method

Abstract

The application discloses a kind of unmanned plane high-voltage line target point selection and real-time positioning method, including obtaining high-voltage line image by unmanned plane carrying vision depth camera;The high-voltage line image obtained is cut by the depth learning model for power line image segmentation, and the high-voltage line segmentation mask is obtained;Obtain the high-voltage line segmentation image by using power line mask segmentation filtering method to obtain the filtered high-voltage line segmentation mask;Obtain the three-dimensional coordinates of the target point in the initial frame camera coordinate system;Feature points on the original high-voltage line image are continuously extracted in the current frame, and the camera relative motion is solved by using PnP method, and then the position of the high-voltage line target point in the camera coordinate system is updated;According to the relationship between the unmanned aerial vehicle body coordinate system and the camera coordinate system, the target point is converted to the three-dimensional coordinates in the body coordinate system.The application solves the problem that the power line segmentation in the prior art is not accurate, and the high-voltage line feature points with inconspicuous color texture are difficult to track.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically, it relates to a method for selecting and locating target points on high-voltage power lines using unmanned aerial vehicles (UAVs). Background Technology

[0002] Regular inspections of power transmission and distribution lines are crucial for ensuring the safe and stable operation of power transmission. However, manual line inspections often face challenges such as long inspection distances, heavy workloads, slow efficiency, and high risks. Drones, due to their strong environmental adaptability, are increasingly used in power line inspections. However, drone-based line inspections also have limitations, the most significant being limited flight time. A single flight typically lasts no more than 40 minutes, and the operating radius is generally no more than 5 kilometers. Long-distance, long-duration inspections require repeated trips, significantly reducing operational efficiency. Currently, there are three main solutions to the drone flight time problem: 1) using gasoline-powered drones; 2) deploying drone-to-ground charging stations along the drone's flight path, allowing the drone to autonomously land and recharge when its battery is low; 3) allowing the drone to autonomously dock on high-voltage lines along its flight path for wireless charging when its battery is too low. However, the first two methods suffer from high costs and maintenance difficulties, while the third method offers advantages such as low cost, simple maintenance, and greater flexibility. Therefore, drone autonomous docking and charging technology is gradually becoming a hot research topic.

[0003] The prerequisite for drones to dock and charge is the ability to sense and locate high-voltage power lines. Currently, mainstream sensing methods primarily rely on LiDAR and visual sensors. Using depth cameras with depth measurement capabilities can solve problems such as high-voltage line identification and real-time positioning. Vision-based high-voltage line identification can be divided into traditional methods and deep learning-based methods.

[0004] The shortcomings of existing technologies lie in the fact that traditional power line segmentation algorithms use the inherent features of the power lines for matched filtering detection, which easily detects linear structural noise, requiring extensive post-processing to filter out noise and smooth the detection results. To avoid excessive post-processing and improve algorithm speed and detection accuracy, some technologies have begun to use convolutional neural networks (CNNs) to design end-to-end power line detection algorithms. However, the randomness and uncertainty inherent in neural networks introduce obvious "noise" into these algorithms, and they fail to fully utilize the structural characteristics of the power lines themselves, resulting in broken or unevenly segmented power lines. Furthermore, these neural network algorithms have long processing times at the edge of the drone's movement, consume significant resources, and have poor real-time performance. They also fail to fully utilize the edge features of the image, leading to inaccurate segmentation of the power lines and errors in subsequent point selection and localization processes. These errors could ultimately cause safety accidents during drone flight. Additionally, the lack of distinct color and texture makes it difficult to track the feature points of high-voltage power lines. Summary of the Invention

[0005] To address the technical problems of existing technologies failing to fully utilize the structural characteristics of power lines and the edge features of images, resulting in inaccurate segmentation of power lines and unclear color and texture making it difficult to track high-voltage line feature points, this invention provides a method for selecting and real-time locating high-voltage line target points using unmanned aerial vehicles (UAVs).

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for selecting and locating a target point on a high-voltage power line using an unmanned aerial vehicle (UAV), comprising the following steps:

[0008] S1. Obtain images of high-voltage power lines using a drone equipped with a visual depth camera;

[0009] S2. The acquired high-voltage line image is segmented using a deep learning model for power line image segmentation to obtain a high-voltage line segmentation mask;

[0010] S3. Use the power line mask segmentation filtering method to obtain the filtered high-voltage line segmentation mask, so as to obtain the high-voltage line segmentation image;

[0011] S4. Using a high-voltage line image as the initial frame, select the midpoint of the high-voltage line as the high-voltage line positioning target point based on the high-voltage line segmentation image of the initial frame, determine the depth of the target point, and then obtain the three-dimensional coordinates of the target point in the camera coordinate system of the initial frame.

[0012] S5. As the drone moves, continuously acquire high-voltage line images and their high-voltage line segmentation masks, and extract feature points on the original high-voltage line image based on the high-voltage line segmentation mask.

[0013] In the current frame, feature points of the high-voltage line image in the previous frame are tracked, and the PnP method is used to solve the relative motion of the camera, thereby updating the position of the high-voltage line target point in the camera coordinate system.

[0014] S6. Based on the relationship between the UAV body coordinate system and the camera coordinate system, transform the target point to three-dimensional coordinates in the body coordinate system.

[0015] Furthermore, the deep learning model in step S2 is the U-Net network model.

[0016] Furthermore, in the U-Net network model, the input image is expanded from 3D to 32D, and the number of channels in each subsequent layer is reduced by half compared to the original U-Net network model. That is, the number of channels in each layer of the corresponding decoder encoder part is reduced by a factor of 2.

[0017] Furthermore, in the training of the U-Net network model, a combination of DiceLoss and FocalLoss loss functions is used as the loss function Loss, the expression of which is:

[0018] Loss=Dice loss+Focal loss.

[0019] Furthermore, the expression for the loss function DiceLoss is:

[0020]

[0021] In the formula: X is the point set of the segmented prediction map, and Y is the point set of the current label map.

[0022] The point expression for the loss function Focalloss is:

[0023]

[0024] Where: p is the probability that the model predicts the foreground. α is the sample label, β is the adjustment factor, and α is the loss weight of the foreground.

[0025] Furthermore, the power line mask segmentation and filtering method in step S3 includes:

[0026] Extract all wire points in the mask, determine whether the current point is an image edge and whether the current point has the same value as the surrounding points, and record all image edge points and points whose values ​​are different from the surrounding points as edge points, thus obtaining the edge detection image based on the image mask;

[0027] The edge detection image obtained earlier is used to extract contours. The points within the contour with the largest area are taken as wires, and the others are taken as background, resulting in the final wire-filtered mask.

[0028] Furthermore, the detailed steps of step S4 include:

[0029] Based on the high-voltage line segmentation image, several points are uniformly extracted, and the midpoint is selected as the target point p of the high-voltage line. target =[u target ,v target ] T , where P represents the pixel coordinates of the target point in the image, u and v represent the positions of the target point on the u-axis and v-axis in the pixel coordinate system, respectively, and the subscript target represents the target point;

[0030] Expansion calculations are performed on the high-voltage line segmentation mask to enlarge the high-voltage line area within the mask;

[0031] The depth values ​​within the expanded high-voltage line area on the statistical depth map are counted, and outliers that are too large or too small are removed. The mode is then calculated as the depth value s of the high-voltage line target point.

[0032] After calibrating the camera intrinsic parameter matrix K, the two-dimensional pixel coordinates of the target point in the image are transformed into three-dimensional coordinates in the camera coordinate system using the following method.

[0033]

[0034]

[0035]

[0036] Where X, Y, and Z represent the distances of the three-dimensional coordinates along the three axes, the superscript C indicates that the coordinate is in the camera coordinate system, and f x f y c x c y To obtain the parameters within the camera intrinsic parameter matrix from the calibration, we have:

[0037]

[0038] Furthermore, the dilation operation involves convolving image A with kernel B;

[0039] In convolution, the local maximum value of pixels in the region of image A covered by kernel B is calculated, and this local maximum value is assigned to the center position of the region of image A covered by kernel B. The specific mathematical expression is as follows:

[0040]

[0041] Where A represents the original image, A(u+i,v+j) represents the pixel value corresponding to the pixel coordinates (u+i,v+j) in the original image, A` represents the dilated image, A`(u,v) is the pixel value of the dilated image A` at the point (u,v), sizeB represents the size of the convolution kernel B, and i and j are calculated from the size of the convolution kernel B.

[0042] Furthermore, in step S5, the high-voltage line segmentation mask of each frame image is used as a moving window to extract Harris corner points as feature points from the original color image.

[0043] Furthermore, in step S5, the sparse optical flow method is used to track feature points between two frames of images to obtain several sets of matched feature points.

[0044] Furthermore, step S5 also includes using a random sampling consensus method to remove incorrect matches among feature points;

[0045] The relative motion of the camera is solved using the PnP (Perspective-n-Point) method based on the removed feature points, resulting in the relative rotation matrix R and the relative displacement vector t.

[0046] Furthermore, step S5 also includes constructing a BundleAdjustment problem to optimize the motion solved based on the PnP (Perspective-n-Point) method, thereby obtaining the optimized relative rotation matrix R and relative displacement vector t.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] A deep learning network is used to refine and filter the high-voltage lines, resulting in more accurate high-voltage line data. At the same time, the tracking method used to locate target points on the high-voltage lines is more real-time.

[0049] An indirect tracking method is adopted, which uses feature points within the image to calculate relative motion. This avoids directly tracking feature points of high-voltage lines with indistinct color and texture, thus improving tracking efficiency and accuracy.

[0050] In some embodiments, simplifying the network reduces computational load and improves computational efficiency.

[0051] In some embodiments, morphological processing and filtering are also used to obtain more accurate depth information of small objects. Attached Figure Description

[0052] Figure 1 This is an overall flowchart of the UAV high-voltage line target point selection and real-time positioning method in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of a typical U-Net network model in the prior art.

[0054] Figure 3 This is a detailed flowchart of the power line mask segmentation and filtering method in an embodiment of the present invention;

[0055] Figure 4 This is a comparison image of the high-voltage line, the high-voltage line segmentation mask, and the filtered mask in an embodiment of the present invention.

[0056] Figure 5 This is a flowchart illustrating the selection, positioning, and tracking of high-voltage line target points in an embodiment of the present invention.

[0057] Figure 6 This is a comparison image of the high-voltage line segmentation mask before and after expansion in an embodiment of the present invention;

[0058] Figure 7This is a schematic diagram of reprojection error in an embodiment of the present invention;

[0059] Figure 8 This is a comparison chart of tracking and positioning results before and after BA optimization in an embodiment of the present invention. Detailed Implementation

[0060] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0061] like Figure 1 As shown, this embodiment provides a method for selecting and real-time locating target points on high-voltage power lines using a drone, including the following steps:

[0062] S1. Obtain images of high-voltage power lines using a drone equipped with a visual depth camera;

[0063] S2. The acquired high-voltage line image is segmented using a deep learning model for power line image segmentation to obtain a high-voltage line segmentation mask;

[0064] S3. Use the power line mask segmentation filtering method to obtain the filtered high-voltage line segmentation mask, so as to obtain the high-voltage line segmentation image;

[0065] S4. Using a high-voltage line image as the initial frame, select the midpoint of the high-voltage line as the high-voltage line positioning target point based on the high-voltage line segmentation image of the initial frame, determine the depth of the target point, and then obtain the three-dimensional coordinates of the target point in the camera coordinate system of the initial frame.

[0066] S5. As the drone moves, continuously acquire high-voltage line images and their high-voltage line segmentation masks, and extract feature points on the high-voltage line image of that frame based on the high-voltage line segmentation mask.

[0067] In the current frame, feature points of the high-voltage line image in the previous frame are tracked, and the PnP method is used to solve the relative motion of the camera, thereby updating the position of the high-voltage line target point in the camera coordinate system.

[0068] S6. Based on the relationship between the UAV body coordinate system and the camera coordinate system, transform the target point to three-dimensional coordinates in the body coordinate system.

[0069] The drone used in this example also carries an onboard computer and a high-precision satellite navigation and positioning device. The onboard computer is used for image processing and numerical optimization; the high-precision satellite navigation and positioning device is used for position and velocity guidance flight. A depth camera system is installed on the depth camera, which includes a vision processor, an infrared module, and a depth module. The vision processor receives signals from the color camera and processes the acquired color information of the world; the depth module receives depth information from the infrared module, ensuring that the camera can ultimately generate a depth map.

[0070] In step S2, the deep learning model used for power line image segmentation was the U-Net network model, which was adjusted and trained using the training dataset before being applied to the UAV. There is a large discrepancy between the image features extracted by traditional power line segmentation algorithms and the high-level semantics of the image. Fixed parameter models perform well in fixed scenarios but poorly in other scenarios. Power line aerial images, like medical images, have the difficulty of extremely simple structural information of the target to be segmented. The U-Net model originated in the field of medical image segmentation. Its structure is U-shaped and mainly includes two parts: feature extraction and upsampling. Its characteristics are: (1) The image semantics are simple and the structure is fixed, so both high-level semantic information and low-level features are very important; (2) The amount of data is small, so the model should not be too large. Too many parameters can easily lead to overfitting. Therefore, U-Net and its derived algorithms are widely used in power line image segmentation tasks. The classic U-Net model is shown in Figure X.

[0071] In this example, the network size of the U-Net network model was adjusted. For the input layer, U-Net directly expanded the input image from 3D to 64D. At the top layer, U-Net designed the maximum number of channels to be 1024. However, the structure of electric wires is relatively simple, so so many channels are unnecessary. Furthermore, the contribution of shallow features in the electric wire image to electric wire recognition is just as important as that of deep features. To reduce the network's extraction of redundant shallow features, this example only expands the input image from 3D to 32D. Additionally, to reduce the extraction of some useless high-level features, the number of channels in each subsequent layer is halved, meaning the number of channels in each layer of the corresponding decoder and encoder is reduced by a factor of 2. At this point, the maximum number of channels in the entire model is 512.

[0072] In this example, when training the U-Net network model, a combination of DiceLoss and FocalLoss loss functions is used as the loss function Loss, and its expression is:

[0073] Loss=Dice loss+Focal loss;

[0074] The classic U-Net uses the Cross-Entropy Loss Function. During model training, this function examines each pixel individually, and the loss for the entire image is the average of the losses for each pixel. Therefore, the final loss function tends to favor the background, which has a high proportion of pixels, masking the loss for the power lines and making it difficult for the network to learn their features. The Dice Loss function measures the overlap between samples from different classes, tending to uncover foreground regions. It is suitable for imbalanced datasets and is expressed as:

[0075]

[0076] In the formula: X is the point set of the segmented predicted image, and Y is the point set of the current label image. On the other hand, the boundary positions in the foreground and linear noise in the background should be given more attention during training.

[0077] Focalloss adaptively adjusts the proportion of loss value for each pixel, and its expression is:

[0078]

[0079] Where: p is the probability that the model predicts the foreground. α is the sample label, β is the adjustment factor, and α is the loss weight of the foreground. By adjusting α, the contribution of the foreground to the loss value during training can be adjusted.

[0080] The training dataset used in this embodiment is from the TTPLA public dataset. This dataset is used to detect and segment transmission towers (TT) and power lines (PL) in aerial images. It has abundant data, comprehensive scenes, and a large number of labeled files, making it suitable for training the model in this example. The TTPLA dataset used in this case contains a total of 1241 power line images and 1241 labeled files, covering various scenes in urban and rural areas. In addition, during the use of the training dataset, the images in the original training dataset were subjected to distortion, cropping, flipping, and HSV transformation operations to increase the number of samples. Redundant labels on the images were also removed.

[0081] During training, the data was first divided into training, validation, and test sets in an 8:1:1 ratio, with the input image size fixed at 512×512. The optimizer used was Adam, with an initial learning rate of 0.0001. A learning rate descent strategy was employed, with a minimum learning rate of 0.01 of the maximum learning rate, training continued until the validation set loss no longer decreased. The trained model was then integrated into ROS, and the model files were ported to the NX board. Environmental images were captured using a D435i depth camera, and the images were fed into the model for prediction. Finally, the precise filtering effect of the neural network-based electric field segmentation and extraction was observed in RVIZ.

[0082] Considering the relatively open aerial environment during UAV power line inspection missions, the information in the depth map is quite limited, and can be simply divided into foreground and background points. Foreground points generally have a depth distance of less than 100 meters, while background points are generally more than 100 meters away. Therefore, by setting a certain threshold based on distance in the depth map, some background points that might be extracted by the neural network can be filtered out; this is called the depth map filtering method.

[0083] While the depth map-based filtering method can extract some long-range noise, some closer linear noise still significantly affects subsequent power line point extraction and localization. Therefore, step S3 of this example proposes a power line mask segmentation filtering method, the flowchart of which is shown below. Figure 3 As shown, this method mainly consists of two parts: one is contour extraction based on power line masks, and the other is in-contour point filtering. Figure 3 As shown, in the contour extraction part based on the power line mask, all power line points in the mask are first extracted. Then, it is determined whether the current point is an image edge and whether the current point has the same value as the surrounding points. All image edge points and points whose values ​​are different from the surrounding points are recorded as edge points, thus obtaining the edge detection image based on the image mask. In the contour in-contour filtering method, the contour of the previously obtained edge detection image is extracted, and then the points within the contour with the largest area are taken as power lines, and the rest are recorded as background, thus obtaining a very clean filtered mask. Based on this mask, the high-voltage line segmentation image can be segmented from the original image.

[0084] Figure 4 The image, the high-voltage line segmentation mask, and the filtered mask are illustrated from left to right in the middle, as exemplarily shown in the above process.

[0085] Figure 5 The document illustrates the detailed process of two steps in this embodiment: selection and positioning of the high-voltage line target point (step S4) and tracking of the high-voltage line target point (step S5).

[0086] When measuring the depth of power lines at long distances, cameras often produce incomplete or "black hole" depth readings. To address this issue, the high-voltage line target point selection and positioning steps in this embodiment perform morphological processing on the high-voltage line segmentation mask. The high-voltage line area in the mask is expanded using an expansion method. The depth values ​​of each point in the corresponding depth map within this area are statistically analyzed, filtered, and outliers that are too large or too small are removed. The mode of these outliers is then used as the depth value of the high-voltage line target point.

[0087] In this example, step S4 includes the following more detailed steps:

[0088] Using a certain frame of a high-voltage power line image as the initial frame, several points are uniformly extracted from the high-voltage power line segmentation image based on the initial frame, and then the midpoint is calculated as the target point p of the high-voltage power line. target =[u target ,v target ] T , where P represents the pixel coordinates of the target point in the image, u and v represent the positions of the target point on the u-axis and v-axis in the pixel coordinate system, respectively, and the subscript target represents the target point;

[0089] Next, morphological processing is performed on the high-voltage line segmentation mask, that is, expansion calculation is performed on the high-voltage line area in the mask.

[0090] The dilation operation convolves image A with kernel B. During convolution, the local maximum value of pixels in the region of image A covered by kernel B is calculated, and this local maximum value is assigned to the center position of the region of image A covered by kernel B. The specific mathematical expression is as follows:

[0091]

[0092] Where A represents the original image, A(u+i,v+j) represents the pixel value corresponding to the pixel coordinates (u+i,v+j) in the original image, A` represents the dilated image, A`(u,v) is the pixel value of the dilated image A` at the point (u,v), sizeB represents the size of the convolution kernel B, and i and j are calculated from the size of the convolution kernel B.

[0093] Figure 6 The example shows a comparison of the high-voltage line segmentation mask image before and after dilation calculation. The left side is the original image, and the right side is the image after dilation calculation.

[0094] After statistically analyzing the depth values ​​within the expanded high-voltage line region on the depth map and removing outliers that are too large or too small, the mode is calculated as the depth value 's' for the target point of the high-voltage line. Figure 6 The results show that although the noise falsely detected by the high-voltage line is also inflated, calculating the mode of the inflated part as the target point depth value when calculating the depth of the inflated part can effectively filter out the influence of noise.

[0095] The camera intrinsic parameters are calibrated using Zhang's calibration method. The following formula can be used to transform the two-dimensional pixel coordinates of the target point in the image into three-dimensional coordinates in the camera coordinate system.

[0096]

[0097] Where X, Y, and Z represent the distances of the three-dimensional coordinates along the three axes, the superscript C indicates that the coordinates are in the camera coordinate system, K is the camera intrinsic parameter matrix, and f x f y cx c y These are the camera's intrinsic parameters, obtained from calibration. The depth information is obtained from the above statistical calculations; finally, the coordinates of the remaining two dimensions can be deduced as follows:

[0098]

[0099]

[0100] In step S5 of this example, to address the difficulty in tracking power line feature points, instead of directly tracking feature points extracted from high-voltage lines, easy-to-track Harris corner points are extracted from the original color image of each frame as feature points. The sparse optical flow method is used to track the feature points extracted from the previous frame in continuous images, and the RANSAC method is used to remove incorrect matches among the feature points. Then, the PnP method is used to solve the relative motion of the camera for the matched feature points, and a Bundle Adjustment problem is constructed to optimize the motion.

[0101] After initially acquiring the target point of the high-voltage line in one go, as the UAV moves, the position of the target point in the camera coordinate system is updated frame by frame using the calculated relative camera movement parameters, thus completing the target point tracking.

[0102] In this example, the more detailed steps include:

[0103] As the drone moves, the high-voltage line segmentation mask of each frame image is used as a moving window to extract Harris corner points as feature points from the original color image; this technique is well known to those skilled in the art, and further technical details of extracting Harris corner points will not be elaborated here.

[0104] The sparse optical flow method is used to track Harris corner points between consecutive images to obtain several sets of well-matched feature points.

[0105] Optical flow is often used to determine the position of a point in one image within a second image; it's a computational method for recovering motion information from consecutive frames. The sparse optical flow method combines optical flow estimation and global optimization techniques. It accumulates data based on color and spatial optical flow estimation, sparsifies the estimated optical flow, and then performs nonlinear optimization on the sparse optical flow data to determine the position and motion of objects. Because it focuses only on easily trackable points, it has low computational cost, good speed, and also high accuracy and reliability. Therefore, the sparse optical flow method is used to track Harris corner points in each frame of the image, obtaining several sets of well-matched feature points.

[0106] Use the Random Sampling Consensus (RANSAC) algorithm to remove incorrect matches among feature points;

[0107] The RANSAC algorithm iteratively searches for the optimal parameter model in a dataset containing "outsiders." Points that do not conform to the optimal model are defined as "outsiders." For feature points matched between two consecutive image frames, the RANSAC algorithm finds an optimal homography matrix H such that the number of feature points satisfying this matrix is ​​maximized. Points that do not conform to this matrix are considered "outsiders" and discarded. Let p1(uv) and p2(u'v') be a pair of matched feature points between two image frames I1 and I2, and let the homography matrix H satisfy: p2 = Hp1. Let p1 and p2 be homogeneous coordinates, then:

[0108]

[0109] Expanding the equations reveals that a set of matching feature points can construct two constraint equations. It's also easy to prove that point p1, regardless of whether it's mapped by H or aH, transforms into p2. Therefore, when calculating H, let... The homography matrix H has 8 degrees of freedom, meaning it can be calculated using four sets of non-collinear matching feature points. The RANSAC algorithm randomly selects four sets of non-collinear matching feature points, calculates the homography matrix, uses this matrix to test all feature points, and calculates the number of feature points (interior points) satisfying this matrix and the projection error (i.e., the cost function). The cost function is:

[0110]

[0111] Iterate k times to find the set with the largest number of interior points where the projection error is less than a set threshold. This set is the feature point group to be removed from the mismatches. The number of iterations k is not fixed and is expressed as:

[0112]

[0113] Where p is the confidence level, typically 0.995; w is the proportion of "inside points"; and m is the minimum number of samples, which is 4 in this example.

[0114] The relative motion of the camera is solved using the PnP (Perspective-n-Point) method for the matched feature points;

[0115] Perspective-n-Point (PnP) is a method for solving the relative motion of 3D to 2D points. Given n 3D spatial points and their projected positions, the relative camera pose can be estimated. Therefore, combining the feature point set obtained by eliminating mismatches with the depth information provided by the depth camera, and using the transformation relationship described above for converting the 2D pixel coordinates of the target point in the image to 3D coordinates in the camera coordinate system, a set of 3D spatial points and their projections can be constructed. This can be used to solve the relative pose of the camera coordinate system between two consecutive frames, i.e., the relative rotation and translation of the camera coordinate system. The mathematical expression is:

[0116]

[0117] Where s i For depth information, Let R and t be the 3D point coordinates in the camera coordinate system for two consecutive frames, respectively, where R and t are the relative rotation and displacement between the two frames, K is the camera intrinsic parameter matrix, and p is the 3D point coordinates in the camera coordinate system for two consecutive frames. i for The projection.

[0118] Using the EPnP method, four control points are first calculated for all 3D points in space. These control points describe the spatial distribution of these 3D points, and their coordinates are in the camera coordinate system of the previous frame. Then, using 2D points captured by the camera, the coordinates of the control points in the camera coordinate system of the next frame are obtained. Knowing the coordinates of the four control points in the two camera coordinate systems, the relative rotation R and displacement t can be calculated using any 3D-3D matching method.

[0119] Finally, the motion is optimized by constructing a Bundle Adjustment problem.

[0120] To reduce the error, the PnP results need to be optimized. A nonlinear least squares problem for reprojection error is constructed using the Bundle Adjustment (BA) method, with the camera relative pose and spatial points as variables to be optimized. For the aforementioned n 3D spatial points... and its projection p i =[u i ,v i ] T The relative rotation and displacement between the two frames are R and t, respectively, and the Lie group is represented as... The optimization function is expressed in homogeneous form as:

[0121]

[0122] The above error is the difference between the projected position of the 3D point and the observed value, i.e., the projection error. For example... Figure 7As shown, feature point tracking using the sparse optical flow method is used to determine the projections of known spatial points P, where p1 and p2 are the same point P. Initial values ​​of R and t are obtained through PnP, and the projection of P is also determined. There is still a deviation e from the actual p2. This deviation can be reduced by adjusting R and t.

[0123] With the optimized R and t, the coordinates of the high-voltage line target point in the initially obtained camera coordinate system can be determined. Updates are being made, including:

[0124]

[0125] in This indicates the three-dimensional coordinates of the target point in the camera coordinate system of the previous frame.

[0126] In step S6, after each update, the odometry information provided by sensors such as the UAV's RTK is combined to adjust the coordinates of the high-voltage line target point in the camera coordinate system. Transform to the world coordinate system to provide control planning for the UAV.

[0127] Based on the relationship between the UAV's body coordinate system and the camera coordinate system, the target point is transformed into the body coordinate system as follows:

[0128]

[0129] in This represents the three-dimensional coordinates of the target point in the body coordinate system. and These represent the rotation and displacement from the camera coordinate system to the body coordinate system, respectively.

[0130] Similarly, transform the target point from the body coordinate system to the world coordinate system:

[0131]

[0132] in This represents the three-dimensional coordinates of the target point in the world coordinate system. and These represent the rotation and displacement from the body coordinate system to the world coordinate system, respectively, and can be calculated from the UAV's odometer information.

[0133] The tracking of the three-dimensional coordinates of the selected high-voltage line target point, and a comparison of the results before and after optimization. Figure 8As shown in the figure, the red lines in parts a), b), and c) represent the tracking and positioning results of the high-voltage power line target point on the X, Y, and Z axes before optimization (Notoptimization). These results exhibit significant fluctuations, with the largest exceeding 1 meter, making them susceptible to drone movement, RTK accuracy, and depth map errors, resulting in poor robustness. The black lines in parts a), b), and c) correspond to the positioning results obtained after BA optimization (BA optimization). The fluctuation range is approximately ±10cm, demonstrating significantly improved robustness and accuracy. The results in the figure show that the tracking method of this invention can maintain continuous tracking of the high-voltage power line target point relatively well, exhibiting good tracking performance.

[0134] The above provides a detailed description of a method for selecting and real-time locating high-voltage line targets using a drone. The specific embodiments described are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A method for selecting and real-time locating target points on high-voltage power lines using unmanned aerial vehicles (UAVs), characterized in that, Including the following steps: S1. Obtain images of high-voltage power lines using a drone equipped with a visual depth camera; S2. The acquired high-voltage line image is segmented using a deep learning model for power line image segmentation to obtain a high-voltage line segmentation mask; S3. Use the power line mask segmentation filtering method to obtain the filtered high-voltage line segmentation mask, so as to obtain the high-voltage line segmentation image; S4. Using a high-voltage line image as the initial frame, based on the high-voltage line segmentation image of the initial frame, select the midpoint of the high-voltage line as the high-voltage line positioning target point, determine the depth of the target point's location, and then obtain the three-dimensional coordinates of the target point in the camera coordinate system of the initial frame, including: Based on the high-voltage line segmentation image, several points are uniformly extracted, and the midpoint is selected as the target point of the high-voltage line. , where P represents the pixel coordinates of the target point in the image, u and v represent the positions of the target point on the u-axis and v-axis in the pixel coordinate system, respectively, and the subscript target represents the target point; Expansion calculations are performed on the high-voltage line segmentation mask to enlarge the high-voltage line area within the mask; After analyzing the depth values ​​within the expanded high-voltage line area on the depth map and removing outliers that are too large or too small, the mode is calculated as the depth value of the target point on the high-voltage line. ; Calibrate the camera intrinsic parameter matrix The following method is used to transform the two-dimensional pixel coordinates of the target point in the image into three-dimensional coordinates in the camera coordinate system. : Where X, Y, and Z represent the distances of the three-dimensional coordinates along the three axes, and the superscript C indicates that the coordinates are in the camera coordinate system. , , , To obtain the parameters within the camera intrinsic parameter matrix from the calibration, we have: ； S5. As the drone moves, continuously acquire high-voltage line images and their high-voltage line segmentation masks, and extract feature points on the high-voltage line images based on the high-voltage line segmentation masks, including using the high-voltage line segmentation masks of each frame image as a moving window to extract Harris corner points as feature points from the original color image. In the current frame, the feature points of the high-voltage line image in the previous frame are tracked. The sparse optical flow method is used to track the feature points between the two frames to obtain several sets of matched feature points. The PnP method is used to solve the relative motion of the camera, and then the position of the target point of the high-voltage line in the camera coordinate system is updated. This includes using the random sampling consensus method to remove erroneous matches in the feature points; and using the PnP method to solve the relative motion of the camera based on the removed feature points to obtain the relative rotation matrix R and the relative displacement vector t. S6. Based on the relationship between the UAV body coordinate system and the camera coordinate system, transform the target point to three-dimensional coordinates in the body coordinate system.

2. The method according to claim 1, characterized in that, The deep learning model in step S2 is the U-Net network model.

3. The method according to claim 2, characterized in that, In the U-Net network model, the input image is expanded from 3D to 32D. The number of channels in each subsequent layer is reduced by half compared to the original U-Net network model. That is, the number of channels in each layer of the corresponding decoder encoder is reduced by a factor of 2.

4. The method according to claim 2, characterized in that, In training the U-Net network model, a combination of Dice Loss and Focal Loss is used as the loss function. Its expression is: The expression for the loss function Dice Loss is: In the formula: X is the point set of the segmented predicted image, and Y is the point set of the current label image; The point expression for the Focal loss function is: Where: p is the probability that the model predicts the foreground. These are sample labels. As a regulating factor, The loss weight is the foreground.

5. The method according to claim 1, characterized in that, The power line mask segmentation and filtering method in step S3 includes: Extract all wire points in the mask, determine whether the current point is an image edge and whether the current point has the same value as the surrounding points, and record all image edge points and points whose values ​​are different from the surrounding points as edge points, thus obtaining the edge detection image based on the image mask; The edge detection image obtained earlier is used to extract contours. The points within the contour with the largest area are taken as wires, and the others are taken as background, resulting in the final wire-filtered mask.

6. The method according to claim 1, characterized in that, Step S5 also includes constructing a Bundle Adjustment problem to optimize the motion solved based on the PnP method, resulting in the optimized relative rotation matrix R and relative displacement vector t.

Images