Unmanned aerial vehicle electric power inspection route planning method and system based on three-dimensional target detection

Abstract

The invention discloses an unmanned aerial vehicle electric power inspection route planning method and system based on three-dimensional target detection, and the method comprises the steps: collecting laser radar LiDAR data of a to-be-planned scene, carrying out the preprocessing of the data, and obtaining a three-dimensional point cloud with color information; inputting the three-dimensional point cloud into a pre-constructed virtual view generation model based on 2D sputtering, and outputting a virtual view under a side view angle corresponding to the three-dimensional point through the model; outputting the simulated view to a pre-constructed local-global double-path feature fusion target detection network model, and outputting the image positions of the power transmission tower and the component in the virtual view through the model; performing back projection on the image positions of the power transmission tower and the components to obtain spatial three-dimensional points of the power transmission tower and the components, and determining three-dimensional space coordinates of the power transmission tower and the components in the scene to be planned based on the spatial three-dimensional points; and based on a route planning algorithm and the three-dimensional space coordinates of the power transmission tower and the components, determining a planned route of unmanned aerial vehicle power inspection. Therefore, the route planning efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and machine learning, and particularly to a method and system for unmanned aerial vehicle (UAV) power inspection route planning based on three-dimensional object detection. Background Art

[0002] With the rapid progress of UAV remote sensing technology, especially the popular application of high-resolution and lightweight airborne LiDAR devices, power facility inspection based on UAVs has become a research hotspot in grid automation inspection solutions. How to automatically plan UAV inspection routes is the primary problem in UAV power facility inspection. The current UAV route planning methods are mainly divided into two categories. The first category is to perform three-dimensional modeling of the transmission line corridor based on lidar, and plan the UAV route according to certain rules based on the positions of transmission towers and the shooting angle and distance. The other category is to automatically extract transmission towers and components from the collected three-dimensional data during the UAV's inspection task, and then further complete the dynamic route planning according to the detection results. The former highly depends on the three-dimensional point cloud of the transmission line corridor and is not suitable for the dynamically changing field environment. The latter dynamically plans according to the measured data, and the planned route has strong timeliness, but it depends on the detection efficiency and accuracy of the three-dimensional object detection algorithm, and the calculation amount is large, making it difficult to achieve real-time route planning. Therefore, how to optimize the three-dimensional object detection algorithm to achieve rapid and accurate detection of transmission towers and components on the towers is of great significance for the autonomous planning of UAV power inspection routes.

[0003] Existing 3D object detection methods can generally be divided into three categories. The first category is the object detection method based on discrete 3D points, the second category is the object detection method based on 3D voxels, and the third category is the object detection method based on projection. The first two methods are extremely susceptible to the influence of point cloud density, have poor adaptability to sparse point cloud regions, and are prone to missed detections or misdetections. For dense point cloud regions, the detection efficiency is low due to the excessive data volume. Therefore, these two methods lack generality and stability in different scale scenarios. In contrast, for the object detection method based on projection, the 3D data is first transformed into one or more 2D images through a certain projection method, and then the currently mature 2D object detection method is used to quickly detect all objects in the scene. Finally, the 2D object detection results are transformed into 3D object positions according to the back-projection from 2D to 3D. Since this method compresses the 3D space into a 2D image space, it greatly reduces the data volume of the scene to be detected, is less affected by point cloud density and scene size, and has a faster running efficiency compared to the first two methods, and is more suitable for large-scale scenarios such as power transmission line inspection. In the object detection method based on projection, the selection of the projection method and the optimization of the 2D image object detection method are the key to improving the efficiency and accuracy of the entire algorithm. Currently, such research mainly focuses on the 3D point cloud processing in autonomous driving, where the on-vehicle 3D point cloud is projected into a bird's-eye view and conventional image object detection methods, such as the YOLO algorithm, are used to detect vehicles and pedestrians in the scene. However, these methods are not applicable to the UAV power facility inspection scenario. On the one hand, the bird's-eye view will compress the significant features of transmission towers and components in height in the UAV scenario, which is not conducive to subsequent object extraction. On the other hand, the point cloud collected in power inspection has a wide coverage range, and transmission towers and the like account for a very small proportion compared to the entire scene, and it is easy to have missed detection problems when using conventional image object detection methods. Summary of the Invention

[0004] Aiming at the problems of low algorithm efficiency and easy omission of transmission towers in the existing UAV power inspection route planning method, the present invention provides a UAV power inspection route planning method based on 3D object detection, including:

[0005] Collect the LiDAR data of the scene to be planned, preprocess the LiDAR data to obtain 3D point clouds with color information;

[0006] Input the 3D point clouds into a pre-constructed virtual view generation model based on 2D splatting, and output the virtual views corresponding to the side view of the 3D points through the model;

[0007] Output the virtual views to a pre-constructed local-global dual-path feature fusion object detection network model, and output the image positions of the transmission towers and components in the virtual views through the model.

[0008] Back-project the image positions of the transmission tower and components to obtain the three-dimensional spatial points of the transmission tower and components, and based on the three-dimensional spatial points, determine the three-dimensional spatial coordinates of the transmission tower and components in the scene to be planned.

[0009] Based on the route planning algorithm and the three-dimensional spatial coordinates of the transmission tower and components, determine the planned route for the UAV power inspection.

[0010] Further, input the three-dimensional point cloud into a pre-constructed virtual view generation model based on 2D splatting, and output the virtual view from the side view angle corresponding to the three-dimensional points through the model, including: taking each three-dimensional point as a three-dimensional spatial Gaussian distribution, and its mathematical expression is:

[0011]

[0012] where μ is the three-dimensional spatial coordinate of any three-dimensional point in the three-dimensional point cloud, and Σ is the three-dimensional covariance matrix, which is obtained by calculating the average distance of the three nearest neighbor points in the X, Y, and Z directions at this point.

[0013] According to the principal component analysis PCA algorithm, calculate the principal direction of all points in the three-dimensional point cloud in the XY coordinates, and calculate the side view parallel projection model P according to the principal direction and the elevation Z direction.

[0014] Project the three-dimensional Gaussian point G(X) onto the virtual view through the side view projection model P to obtain the two-dimensional Gaussian point g(x), and its mathematical expression is:

[0015]

[0016] where μ′ is the two-dimensional coordinate projected from μ, μ′ = Pμ, and Σ′ is the two-dimensional covariance matrix. The relationship between Σ′ and the three-dimensional covariance is as follows:

[0017] Σ ′ = JPΣP T J T

[0018] where J is the Jacobian matrix of P.

[0019] Sort all the two-dimensional Gaussian points (x) according to the depth information to obtain the virtual view from the side view angle corresponding to the three-dimensional points; the color of the image points on the virtual view is obtained by weighting the color brightness of all the two-dimensional Gaussian points at this point, and the specific formula is as follows:

[0020]

[0021] where, (c i ,α i) is the color and opacity of the two-dimensional Gaussian point i, α′ i represents the opacity of the Gaussian point i at the image point.

[0022] Furthermore, the local-global dual-path feature fusion object detection network model is obtained by improving the YOLO network. Among them,

[0023] The virtual view is divided into 4 sub-blocks, and the feature information of the transmission tower and components is highlighted by local magnification through the visual attention area. The magnified area and the entire virtual view are respectively input into the backbone module in the network through the main path and the branch path;

[0024] The branch input of the magnified area is subjected to feature fusion with the convolutional layer of the global area input at multiple scales.

[0025] Furthermore, based on the route planning algorithm and the three-dimensional spatial coordinates of the transmission tower and components, the planned route of the UAV power inspection is determined, including:

[0026] Taking the minimum value of XY in the transmission tower as the starting observation point of the UAV route, the Euclidean distance is used to determine another transmission tower closest to the current transmission tower as the second observation point of the UAV route, and the line segment between the two points is determined as the flight path of the UAV between the two points;

[0027] Taking the second observation point as the new starting point, repeating the above steps until all the transmission towers are covered;

[0028] According to the three-dimensional point information of the transmission tower and its components under the current observation point, calculate the three-dimensional envelope surface, and automatically calculate the transmission tower data acquisition route according to the flight mode of descending by levels of height and then ascending by levels of height back to the observation point;

[0029] Connect the flight paths of the UAVs between all transmission towers and the data acquisition routes of each transmission tower to obtain the UAV inspection route of the scene to be planned.

[0030] Furthermore, it also includes:

[0031] The flight height of the UAV is set according to the height of the transmission tower.

[0032] The present invention also provides a UAV power inspection route planning system based on three-dimensional object detection, including:

[0033] A three-dimensional point cloud acquisition module, used to collect the LiDAR data of the scene to be planned, preprocess the LiDAR data, and obtain a three-dimensional point cloud with color information;

[0034] A virtual view output module, which is used to input the three-dimensional point cloud into a pre-constructed virtual view generation model based on 2D splatting, and output a virtual view of the three-dimensional points corresponding to the side view angle through the model;

[0035] An image position output module, which is used to input the virtual view into a pre-constructed local-global dual-path feature fusion object detection network model, and output the image positions of transmission towers and components in the virtual view through the model;

[0036] A three-dimensional space coordinate determination module, which is used to obtain the spatial three-dimensional points of the transmission tower and components by back-projection of the image positions of the transmission tower and components, and determine the three-dimensional space coordinates of the transmission tower and components in the scene to be planned based on the spatial three-dimensional points;

[0037] A planned flight path determination module, which is used to determine the planned flight path of the UAV power inspection based on the flight path planning algorithm and the three-dimensional space coordinates of the transmission tower and components.

[0038] Further, the virtual view output module includes:

[0039] A Gaussian distribution sub-module, which is used to regard each three-dimensional point as a three-dimensional space Gaussian distribution, and its mathematical expression is:

[0040]

[0041] where μ is the three-dimensional space coordinate of any three-dimensional point in the three-dimensional point cloud, Σ is the three-dimensional covariance matrix, which is obtained by calculating the average distance of the three nearest neighbor points in the XYZ three directions at this point;

[0042] A projection model P calculation sub-module, which is used to calculate the main direction of all points in the three-dimensional point cloud in the XY coordinates according to the principal component analysis PCA algorithm, and calculate the side view parallel projection model P according to the main direction and the elevation Z direction;

[0043] A two-dimensional Gaussian point acquisition sub-module, which is used to project the three-dimensional Gaussian point G(X) onto the virtual view through the side view projection model P to obtain a two-dimensional Gaussian point g(x), and its mathematical expression is:

[0044]

[0045] where μ′ is the two-dimensional coordinate projected from μ, μ′ = Pμ, Σ′ is the two-dimensional covariance matrix, and the relationship between Σ′ and the three-dimensional covariance is as follows:

[0046] Σ ′ = JPΣP T J T

[0047] Among them, J is the Jacobian matrix of P;

[0048] The virtual view acquisition sub-module is used to sort all two-dimensional Gaussian points (x) according to the depth information to obtain the virtual view of the three-dimensional points corresponding to the side view angle; the color of the image point on the virtual view is obtained by weighting the color brightness of all two-dimensional Gaussian points at this point. The specific formula is as follows:

[0049]

[0050] Among them, (c i ,α i ) is the color and opacity of the two-dimensional Gaussian point i, and α′ i represents the opacity of the Gaussian point i at the image point.

[0051] Furthermore, the image position output module includes:

[0052] The virtual view magnification sub-module is used to divide the virtual view into 4 sub-blocks, locally magnify and highlight the characteristic information of the transmission tower and components through the visual attention area, and input the magnified area and the entire virtual view into the backbone module in the network through the main path and the branch path respectively;

[0053] The feature fusion sub-module is used to perform feature fusion on the branch input of the magnified area and the convolutional layer of the global area input at multiple scales.

[0054] Furthermore, the planned flight route determination module includes:

[0055] The flight path determination sub-module is used to take the minimum value of XY in the transmission tower as the starting observation point of the UAV flight route, determine the nearest transmission tower to the current transmission tower through the Euclidean distance as the second observation point of the UAV flight route, and determine the line segment between the two points as the UAV flight path between the two points;

[0056] The starting point secondary confirmation sub-module is used to take the second observation point as the new starting point and repeat the above steps until all the transmission towers are covered;

[0057] The acquisition route calculation sub-module is used to calculate the three-dimensional envelope surface according to the three-dimensional point information of the transmission tower and its components under the current observation point, and automatically calculate the transmission tower data acquisition route according to the flight mode of descending by levels of height and then ascending by levels of height back to the observation point;

[0058] The inspection flight route acquisition sub-module is used to connect the flight paths of the UAVs between all transmission towers and the acquisition routes of each transmission tower to obtain the UAV inspection flight route of the scene to be planned.

[0059] Furthermore, it also includes:

[0060] The flight altitude setting module is used to set the altitude of the UAV flight according to the height of the transmission tower.

[0061] A method and system for UAV power inspection route planning based on three-dimensional object detection provided by the present invention generate a virtual view by side-view projection of the three-dimensional point cloud of the entire scene, then use a local-global dual-path feature fusion network to detect transmission towers and components in the virtual view, and finally plan the flight route according to the detected transmission towers and components in the entire scene, realizing a double improvement in the efficiency and accuracy of route planning. Brief Description of the Drawings

[0062] Figure 1 It is a schematic flowchart of a method for UAV power inspection route planning based on three-dimensional object detection provided by an embodiment of the present invention;

[0063] Figure 2 It is a flowchart of an algorithm for UAV power inspection route planning based on fast three-dimensional object detection related to an embodiment of the present invention as shown;

[0064] Figure 3 It is a structural diagram of a local-global dual-path feature fusion network related to an embodiment of the present invention;

[0065] Figure 4 It is the extraction result of three-dimensional transmission towers related to an embodiment of the present invention;

[0066] Figure 5 It is the extraction result of three-dimensional transmission towers related to an embodiment of the present invention;

[0067] Figure 6 It is a schematic diagram of the acquisition route of transmission towers related to an embodiment of the present invention;

[0068] Figure 7 It is a structural diagram of a system for UAV power inspection route planning based on three-dimensional object detection according to an embodiment of the present invention. Detailed Embodiments

[0069] Many specific details are set forth in the following description in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0070] Embodiment 1

[0071] The present invention provides a method for UAV power inspection route planning based on three-dimensional object detection, as Figure 1 shown, including the following steps:

[0072] Step S101: Collect the LiDAR data of the scene to be planned, preprocess the LiDAR data, and obtain a three-dimensional point cloud with color information.

[0073] Step S102: Input the three-dimensional point cloud into a pre-constructed virtual view generation model based on 2D splatting, and output the virtual view of the three-dimensional points corresponding to the side view through the model.

[0074] Take each three-dimensional point as a three-dimensional spatial Gaussian distribution, that is, the probability distribution in space satisfies the Gaussian distribution, and its mathematical expression is:

[0075]

[0076] where μ is the three-dimensional spatial coordinate of any three-dimensional point in the three-dimensional point cloud, and Σ is the three-dimensional covariance matrix, which is obtained by calculating the average distance of the three nearest neighbor points in the X, Y, and Z directions at this point;

[0077] According to the principal component analysis PCA algorithm, calculate the principal direction of all points in the three-dimensional point cloud in the XY coordinates, and calculate the side view parallel projection model P according to the principal direction and the elevation Z direction;

[0078] Project the three-dimensional Gaussian point G(X) onto the virtual view through the side view projection model P to obtain the two-dimensional Gaussian point g(x), and its mathematical expression is:

[0079]

[0080] where μ′ is the two-dimensional coordinate projected from μ, μ′ = Pμ, and Σ′ is the two-dimensional covariance matrix. The relationship between Σ′ and the three-dimensional covariance is as follows:

[0081] Σ ′ = JPΣP T J T

[0082] where J is the Jacobian matrix of P;

[0083] Sort all the two-dimensional Gaussian points g(x) according to the depth information to obtain the virtual view of the three-dimensional points corresponding to the side view; the color of the image points on the virtual view is obtained by weighting the color brightness of all the two-dimensional Gaussian points at this point. The specific formula is as follows:

[0084]

[0085] where (c i ,α i ) is the color and opacity (the value is defaulted to 1) of the two-dimensional Gaussian point i, and α′ iRepresents the opacity of the Gaussian point i at the image point.

[0086] Step S103: Input the virtual view into a pre-constructed local-global dual-path feature fusion object detection network model, and output the image positions of transmission towers and components in the virtual view through the model.

[0087] Among them, the local-global dual-path feature fusion object detection network is an improvement based on the YOLO network, specifically including the following two improvement points:

[0088] In the input part: Divide the virtual view into 4 sub-blocks, and then locally magnify through the visual attention area to highlight the feature information of small targets such as transmission towers and components. Input the magnified area and the entire virtual view into the backbone module in the network through the main path and the branch path respectively.

[0089] In the backbone part: The branch input of the locally magnified area and the convolution layer of the global area input perform feature fusion at multiple scales, that is, after connecting the channels through the concat layer, and then passing through a 1×1 convolution layer to recombine the feature channels to update the attention weight for the small-size target area.

[0090] Step S104: Obtain the three-dimensional spatial points of the transmission tower and components by back-projection from the image positions of the transmission tower and components, and determine the three-dimensional spatial coordinates of the transmission tower and components in the to-be-planned scene based on the three-dimensional spatial points.

[0091] Step S105: Determine the planned flight route of the UAV power inspection based on the flight route planning algorithm and the three-dimensional spatial coordinates of the transmission tower and components.

[0092] The flight route planning algorithm specifically includes the following steps:

[0093] S1: Use the minimum value of XY in the transmission tower as the starting observation point of the UAV flight route, and then find the nearest transmission tower to this transmission tower through the Euclidean distance as the next observation point of the UAV flight route. The line segment between the two points is the flight path of the UAV between the two points. Then use this observation point as the new starting point and repeat this step until all transmission towers are covered. In addition, the flight height of the UAV can be set according to the height of the transmission tower, that is, add a certain distance to the height of the transmission tower as the flight safety distance.

[0094] S2: For the observation point, calculate the three-dimensional outer envelope surface according to the three-dimensional point information of the transmission tower and its components under this observation point, and automatically calculate the data acquisition route of the transmission tower according to the flight mode of descending by levels and then ascending by levels back to the observation point.

[0095] S3: Combine the flight route connecting all transmission towers described in S1 and the data collection route for each transmission tower described in S2 to finally obtain the UAV inspection route for the entire area.

[0096] Embodiment 2

[0097] The technical solution of the present invention will be described in detail below in conjunction with the drawings and embodiments.

[0098] The UAV power inspection route planning method based on fast 3D object detection provides an efficient and feasible method for the route planning of UAVs for power inspection. The implementation manner of the present invention will be elaborated in detail below with examples.

[0099] The algorithm flow of the UAV power inspection route planning based on fast 3D object detection is as Figure 2 shown. First, the LiDAR data is preprocessed through format conversion, coordinate transformation, etc. to be transformed into 3D point cloud data; then, a virtual view generation algorithm based on 2Dsplatting is used to generate a 2D virtual view in the side view perspective; on this basis, an object detection network based on local-global dual-path feature fusion is used to detect the positions of transmission towers and their components on the 2D virtual view; then, according to the projection relationship corresponding to the virtual view, the areas where the transmission towers and components are located on the 2D virtual view are back-projected onto the 3D point cloud to obtain the spatial 3D points of the transmission towers and components; finally, according to the spatial position distribution of the transmission towers, the route planning algorithm is called to complete the power inspection route planning diagram.

[0100] Step 1: Preprocess the input LiDAR data to complete the conversion from LiDAR data to spatial 3D point cloud. The preprocessing process includes data format conversion, coordinate system conversion, etc. Extract the coordinates and color information of all 3D points from the original LiDAR data, and convert the coordinates of all 3D points to a unified spatial coordinate system through a coordinate conversion algorithm, and finally save it in a common format of 3D point cloud.

[0101] Through the original data reading interface function corresponding to the LiDAR acquisition device, read out the number of 3D points recorded in the original LiDAR data, and then sequentially read out the 3D coordinates (X, Y, Z) of each point from the data, as well as the color information of this point, namely (R, G, B). Finally, transform the coordinates of all 3D points to the same coordinate system according to the unified coordinate transformation. This coordinate system usually selects the geodetic coordinate system or a relative spatial coordinate system proportional to the actual poles and towers.

[0102] Step 2: Generation of virtual views based on 2D splatting technology. Currently, conventional virtual views are divided into bird's-eye views and side views. The bird's-eye view provides a perspective from top to bottom, that is, the generated view is an orthographic projection view on the XY plane, which can directly display the overall situation of transmission towers and the environment. However, since the tower features are mainly distributed in the elevation direction, the projection coverage on the XY plane is small and the features are not significant, which is likely to cause the problem that the targets are difficult to distinguish after projection. Therefore, the present invention adopts a side view projection model that is more conducive to detecting towers, that is, by setting a parallel projection matrix, the point cloud is projected onto the vertical plane formed by the flight direction and the elevation direction.

[0103] When projecting the three-dimensional point cloud onto the two-dimensional virtual view, it is necessary to set the spatial size of each three-dimensional point, otherwise the original spatial scale and occlusion relationship of the three-dimensional point cloud will be lost. However, if the size is set too small, the situation where the distant background penetrates the nearby objects will occur, and the occlusion relationship in the point cloud cannot be reflected. If the size is set too large, the situation where three-dimensional points overlap with each other will occur in the dense area of the point cloud. To solve this problem, the 2D splatting technology is adopted to automatically set the three-dimensional point size according to the local density of the point cloud.

[0104] Specifically, each three-dimensional point can be regarded as a three-dimensional Gaussian distribution, that is, the probability distribution in space satisfies the Gaussian distribution, and its mathematical expression is as follows:

[0105]

[0106] where μ is the three-dimensional spatial coordinate of a certain three-dimensional point in the three-dimensional point cloud, and Σ is the three-dimensional covariance matrix, which is obtained by calculating the average distance of the three nearest neighbor points in the XYZ three directions at this point. When the three-dimensional Gaussian point G(X) is projected onto the virtual view through the camera model P corresponding to the side view, a two-dimensional Gaussian point g(x) is obtained, and its mathematical expression is as follows:

[0107]

[0108] where μ′ is the two-dimensional coordinate obtained by projecting μ, that is, μ′ = Pμ, and Σ′ is the two-dimensional covariance matrix, and its relationship with the three-dimensional covariance is as follows:

[0109] Σ ′ = JPΣP T J T

[0110] Among them, J is the Jacobian matrix of P. After projecting the three-dimensional Gaussian points onto the two-dimensional plane through the camera model P, the virtual view from this perspective is then rendered based on the geometric and color attributes after projection: First, the projected two-dimensional Gaussian points are sorted according to the depth information. The color of the pixel on the virtual image is obtained by weighting the colors at this point by all two-dimensional Gaussian points. The specific formula is as follows:

[0111]

[0112] where (c i , α i ) are the color and opacity of the two-dimensional Gaussian point i (the value is defaulted to 1), and α′ i represents the opacity of the Gaussian point i at the pixel.

[0113] Step 3: Object detection based on local-global dual-path feature fusion. Aiming at the problems of small proportion of tower pole size and high detection difficulty in the virtual view, the feature extraction structure of the YOLO network is optimized. Combining multi-scale feature fusion according to the mechanism of visual focus attention, a tower pole and component detection network with local-global dual-path feature fusion is designed to improve the accuracy of object detection. The overall network structure is as Figure 3 shown.

[0114] The whole network is divided into three parts. The input part realizes the simulation of visual focus amplification of the image. The backbone part realizes the feature extraction of the local branch and the global branch and the fusion of feature maps at different scales. The head part realizes the inference and prediction of targets such as tower poles.

[0115] The network divides the input image into sub-blocks at the input end. Considering the calculation speed and accuracy, the image is divided into 4 sub-blocks. Subsequently, the local visual attention area of the image division is locally amplified to enrich the feature information of small targets. Then, the amplified area is input through the branch to extract the feature information and combine it with the features globally extracted from the virtual view for detection. The information extracted globally is mainly used to detect the background, while the information extracted from the local branch features is used to detect the transmission tower target. The design of significantly enhancing the local part in the model is an extended operation of the visual attention mechanism, which amplifies the overall and detailed information of the target object of interest and enhances its spatial position information at the same time, making the model more sensitive to small transmission tower targets. In addition, the feature extraction module fuses the local and global feature information located in different branches at multiple scales, which can more effectively obtain accurate position information and high-level semantic information and increase the spatial correlation. Different from the existing attention mechanism modules, the dual-path feature fusion network amplifies the focus area at the data input end and expands the spatio-temporal information of the target by means of dual-branch multi-scale feature information fusion, thereby changing the weight of the feature map.

[0116] The branches of the local region are input through the convolutional layers of the Backbone and are feature - fused with the convolutional layers of the global region input at multiple scales. That is, after connecting the channels through the Concat (channel connection) layer, a 1×1 convolutional layer is used to recombine the feature channels to update the attention weight for the small - size target region and remove redundant information. The feature extraction architecture of the branch input is similar to that of the global region input. The feature map of the branch local input is interpolated and complemented by combining the global feature map size and the sub - block coordinate information at the input. Subsequently, it passes through the Concat layer and the convolutional module to be fused with the global feature map. Since the output of the Concat layer realizes the connection of the local and global input channel data, the algorithm places a 1×1 convolutional layer after the Concat layer to integrate the feature map information of the target and the background, and there is a SiLU activation function layer after each convolution. In the feature extraction stage of each branch, a 3×3 convolutional module with a stride of 2 is used to downsample the feature map, and then it is passed to the Concat layer together with other inputs to form feature fusion at multiple scales. The finally obtained backbone feature map has the largest receptive field, and the semantic information and spatial information in it are the richest.

[0117] Step 4: According to the strict geometric relationship between the three - dimensional point cloud and the generated virtual view, through the parallel projection matrix in Step 2, the two - dimensional position of the transmission tower is back - projected into the three - dimensional point cloud to obtain its three - dimensional spatial position. Aiming at the problem that only partial side three - dimensional points of the tower can be obtained after back - projection, a three - dimensional region growing algorithm based on seed points is adopted. By using the three - dimensional points that have been detected as belonging to the transmission tower as seed points, and taking the average Euclidean distance and the local point cloud density in the two directions of plane and elevation as the similarity discrimination criteria, according to the region growth, it is successively judged whether the neighborhood points near the seed points meet the similarity criteria. If they meet, the neighborhood points belong to the transmission tower and are used as new seed points, otherwise the points are skipped. This region growth step is continuously executed until no new seed points are added and then it stops. Thus, the entire point cloud of the transmission tower is completely extracted, as Figure 4 and 5 shown. As can be seen from Figure 4 and 5 , although there are complex terrain undulations and ground object distributions near the transmission tower, all three transmission towers in the scene can be detected on the virtual view. The algorithm of this patent projects the two - dimensional detection results into the three - dimensional scene according to the virtual view camera parameters to obtain the accurate position and point cloud information of each transmission tower in the three - dimensional scene, and the results are as Figure 5 shown. All the three - dimensional points belonging to the transmission tower in the figure are highlighted in red, and the details of all three towers are shown in the local enlarged views. From Figure 5It can be seen that the three-dimensional point cloud of the transmission tower can be well extracted by using the algorithm of this patent. However, it can also be seen that a small amount of point clouds of the trees and bare ground near the transmission tower will also be classified in.

[0118] Step Five: Plan the UAV flight route according to the three-dimensional spatial positions of all transmission towers. The entire flight route planning includes two steps. The first step is responsible for planning the global flight route for connecting all transmission towers, that is, taking the minimum value of XY in the transmission towers as the starting observation point of the UAV flight route, and then finding the other transmission tower closest to this transmission tower through the Euclidean distance as the next observation point of the UAV flight route. The line segment between the two points is the flight path of the UAV between the two points. Then, taking this observation point as the new starting point, repeat this step until all transmission towers are covered. In addition, the flight altitude of the UAV can be set according to the height of the transmission tower, that is, adding a certain distance to the height of the transmission tower as the flight safety distance; The second step is to take the observation point above each transmission tower as the starting point and generate the data collection route of this transmission tower, that is, calculate the three-dimensional envelope surface according to the three-dimensional point information of the transmission tower and its components under this observation point, and automatically calculate the data collection route of the transmission tower according to the flight mode of descending in height step by step and then ascending in height step by step back to the observation point, as Figure 6 shown. Combining the flight routes generated by the above two steps, the UAV inspection route for the entire area is finally obtained.

[0119] Embodiment 3

[0120] Based on the same inventive concept, the present invention also provides a UAV power inspection route planning system based on three-dimensional object detection, as Figure 7 shown, including:

[0121] A three-dimensional point cloud acquisition module 710, configured to collect LiDAR data of the scene to be planned, preprocess the LiDAR data, and obtain a three-dimensional point cloud with color information;

[0122] A virtual view output module 720, configured to input the three-dimensional point cloud into a pre-constructed virtual view generation model based on 2Dsplatting, and output the virtual view corresponding to the side view angle of the three-dimensional point through the model;

[0123] An image position output module 730, configured to input the virtual view into a pre-constructed local-global dual-path feature fusion object detection network model, and output the image positions of the transmission tower and its components in the virtual view through the model;

[0124] A three-dimensional space coordinate determination module 740, configured to back-project the image positions of the transmission tower and its components to obtain the three-dimensional spatial points of the transmission tower and its components, and determine the three-dimensional spatial coordinates of the transmission tower and its components in the scene to be planned based on the three-dimensional spatial points;

[0125] The planned route determination module 750 is configured to determine the planned route for the UAV power inspection based on the route planning algorithm and the three-dimensional spatial coordinates of the transmission towers and components.

[0126] Furthermore, the virtual view output module includes:

[0127] The Gaussian distribution sub-module is configured to take each three-dimensional point as a three-dimensional spatial Gaussian distribution, and its mathematical expression is:

[0128]

[0129] where μ is the three-dimensional spatial coordinate of any three-dimensional point in the three-dimensional point cloud, and Σ is the three-dimensional covariance matrix, which is obtained by calculating the average distance of the three nearest neighbor points in the X, Y, and Z directions at this point;

[0130] The projection model P calculation sub-module is configured to calculate the main direction of all points in the three-dimensional point cloud in the XY coordinates according to the principal component analysis (PCA) algorithm, and calculate the parallel projection model P of the side view according to the main direction and the elevation Z direction;

[0131] The two-dimensional Gaussian point obtaining sub-module is configured to project the three-dimensional Gaussian point G(X) onto the virtual view through the side view projection model P to obtain the two-dimensional Gaussian point g(x), and its mathematical expression is:

[0132]

[0133] where μ′ is the two-dimensional coordinate projected from μ, μ′ = Pμ, Σ′ is the two-dimensional covariance matrix, and the relationship between Σ′ and the three-dimensional covariance is as follows:

[0134] Σ ′ = JPΣP T J T

[0135] where J is the Jacobian matrix of P;

[0136] The virtual view obtaining sub-module is configured to sort all the two-dimensional Gaussian points (x) according to the depth information to obtain the virtual view in the side view angle corresponding to the three-dimensional points; the color of the image points on the virtual view is obtained by weighting the color brightness of all the two-dimensional Gaussian points at this point, and the specific formula is as follows:

[0137]

[0138] where (c i ,α i ) is the color and opacity of the two-dimensional Gaussian point i, and α′ i represents the opacity of the Gaussian point i at the image point.

[0139] Further, the image position output module includes:

[0140] The virtual view magnification sub-module is used to divide the virtual view into 4 sub-blocks, locally magnify the power transmission tower and component feature information through the visual attention area, and input the magnified area and the entire virtual view into the backbone module in the network through the main path and the branch path respectively;

[0141] The feature fusion sub-module is used to perform feature fusion on the branch input of the magnified area and the convolution layer input of the global area at multiple scales.

[0142] Further, the planned flight route determination module includes:

[0143] The flight path determination sub-module is used to take the minimum XY value in the power transmission tower as the starting observation point of the UAV flight route, determine the other power transmission tower closest to the current power transmission tower as the second observation point of the UAV flight route through the Euclidean distance, and determine the line segment between the two points as the UAV flight path between the two points;

[0144] The starting point secondary confirmation sub-module is used to take the second observation point as the new starting point, and repeat the above steps until all the power transmission towers are covered;

[0145] The acquisition route calculation sub-module is used to calculate the three-dimensional envelope surface according to the three-dimensional point information of the power transmission tower and its components under the current observation point, and automatically calculate the power transmission tower data acquisition route according to the flight mode of descending in height step by step and then ascending in height step by step back to the observation point;

[0146] The inspection flight route acquisition sub-module is used to connect the flight paths of the UAVs between all power transmission towers and the data acquisition routes of each power transmission tower to obtain the UAV inspection flight route for the scene to be planned.

[0147] Further, it further includes:

[0148] The flight altitude setting module is used to set the altitude of the UAV flight according to the height of the power transmission tower.

[0149] A UAV power inspection flight route planning method and system provided by the present invention generate a virtual view through side view projection of the three-dimensional point cloud of the entire scene, then use a local-global dual-path feature fusion network to detect power transmission towers and components in the virtual view, and finally plan the flight route according to the power transmission towers and components detected in the entire scene, achieving a double improvement in flight route planning efficiency and accuracy.

[0150] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0151] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0152] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that modifications or equivalent replacements can still be made to the specific implementation manners of the present invention. Any modification or equivalent replacement without departing from the spirit and scope of the present invention shall be covered by the scope of the claims of the present invention.

Claims

1. A method for planning a UAV power inspection route based on three-dimensional target detection, characterized in that: include: Collecting LiDAR data of the scene to be planned, preprocessing the LiDAR data, and obtaining a three-dimensional point cloud with color information; Input the three-dimensional point cloud into a pre-built virtual view generation model based on 2D splatting, and output a virtual view corresponding to the three-dimensional point under a side view angle through the model; Outputting the virtual view to a pre-built local-global dual-path feature fusion target detection network model, and outputting the image positions of the transmission towers and components in the virtual view through the model; The image positions of the transmission tower and the components are back-projected to obtain the spatial three-dimensional points of the transmission tower and the components, and based on the spatial three-dimensional points, the three-dimensional spatial coordinates of the transmission tower and the components in the scene to be planned are determined; Based on the route planning algorithm and the three-dimensional spatial coordinates of transmission towers and components, the planned route for drone power inspection is determined.

2. The method according to claim 1, characterized in that The 3D point cloud is input into a pre-built virtual view generation model based on 2D splatting, and the virtual view corresponding to the 3D point under the side view angle is output through the model, including: each 3D point is regarded as a 3D space Gaussian distribution, and its mathematical expression is: Among them, μ is the 3D spatial coordinate of any 3D point in the 3D point cloud, Σ is the 3D covariance matrix, which is obtained by calculating the average distance of the three nearest neighboring points of the point in the three directions of XYZ; According to the principal component analysis PCA algorithm, the main direction of all points in the three-dimensional point cloud in the XY coordinates is calculated, and the side view parallel projection model P is calculated according to the main direction and the elevation Z direction; The three-dimensional Gaussian point G(X) is projected onto the virtual view through the side view projection model P to obtain the two-dimensional Gaussian point g(x), whose mathematical expression is: Among them, μ′ is the two-dimensional coordinate obtained by the projection of μ, μ′=Pμ, Σ′ is the two-dimensional covariance matrix, and the relationship between Σ′ and the three-dimensional covariance is as follows: Σ ′ =JPΣP T J T Where J is the Jacobian matrix of P; All two-dimensional Gaussian points (x) are sorted according to the depth information to obtain the virtual view under the side view angle corresponding to the three-dimensional point; the color of the image point on the virtual view is obtained according to the weighted color brightness of all two-dimensional Gaussian points at the point, and the specific formula is as follows: Among them, (c i ,α i ) is the color and opacity of the two-dimensional Gaussian point i, α′ i Represents the opacity of Gaussian point i at the image point.

3. The method according to claim 1, characterized in that The local-global dual-path feature fusion target detection network model is obtained based on the improvement of the YOLO network, where: The virtual view is divided into four sub-blocks, and the characteristic information of the transmission tower and components is highlighted by locally zooming in the visual attention area, and the zoomed-in area and the entire virtual view are respectively input into the backbone module in the network through the main road and the branch road; The branch input of the enlarged area is passed through the convolution layer and the convolution layer of the global area input to perform feature fusion at multiple scales.

4. The method according to claim 1, characterized in that Based on the route planning algorithm and the three-dimensional spatial coordinates of the transmission towers and components, the planned route for the UAV power inspection is determined, including: The minimum XY value of the transmission tower is used as the starting observation point of the drone route. The other transmission tower closest to the current transmission tower is determined by Euclidean distance as the second observation point of the drone route. The line segment between the two points is determined as the flight path of the drone between the two points. The second observation point is used as a new starting point, and the above steps are repeated until all the transmission towers are covered; The 3D outer envelope surface is calculated based on the 3D point information of the transmission tower and its components at the current observation point, and the data collection route of the transmission tower is automatically calculated in a flight mode of descending step by step and then ascending step by step back to the observation point; Connect the flight paths of drones between all transmission towers and the data collection routes of each transmission tower to obtain the drone inspection routes for the planned scenarios.

5. The method according to claim 1, characterized in that Also includes: The altitude at which the drone flies is set according to the height of the transmission tower.

6. A UAV power inspection route planning system based on three-dimensional target detection, characterized in that: include: A three-dimensional point cloud acquisition module is used to collect LiDAR data of the scene to be planned, pre-process the LiDAR data, and obtain a three-dimensional point cloud with color information; A virtual view output module, used for inputting the three-dimensional point cloud into a pre-built virtual view generation model based on 2D splatting, and outputting a virtual view corresponding to the three-dimensional point under a side view angle through the model; An image position output module, used for outputting the virtual view to a pre-built local-global dual-path feature fusion target detection network model, and outputting the image positions of the transmission towers and components in the virtual view through the model; A three-dimensional space coordinate determination module is used to obtain the spatial three-dimensional points of the transmission tower and the components by back-projecting the image positions of the transmission tower and the components, and determine the three-dimensional space coordinates of the transmission tower and the components in the scene to be planned based on the spatial three-dimensional points; The planned route determination module is used to determine the planned route for UAV power inspection based on the route planning algorithm and the three-dimensional spatial coordinates of the transmission towers and components.

7. The system according to claim 6, characterized in that Virtual view output module, including: The Gaussian distribution submodule is used to treat each 3D point as a 3D space Gaussian distribution, and its mathematical expression is: Among them, μ is the 3D spatial coordinate of any 3D point in the 3D point cloud, Σ is the 3D covariance matrix, which is obtained by calculating the average distance of the three nearest neighboring points of the point in the three directions of XYZ; The projection model P calculation submodule is used to calculate the main direction of all points in the three-dimensional point cloud in the XY coordinates according to the principal component analysis PCA algorithm, and calculate the side view parallel projection model P according to the main direction and the elevation Z direction; The two-dimensional Gaussian point acquisition submodule is used to project the three-dimensional Gaussian point G(X) onto the virtual view through the side view projection model P to obtain the two-dimensional Gaussian point g(x), and its mathematical expression is: Among them, μ′ is the two-dimensional coordinate obtained by the projection of μ, μ′=Pμ, Σ′ is the two-dimensional covariance matrix, and the relationship between Σ′ and the three-dimensional covariance is as follows: Σ′=JPΣP T J T Where J is the Jacobian matrix of P; The virtual view acquisition submodule is used to sort all the two-dimensional Gaussian points (x) according to the depth information to obtain the virtual view under the side view angle corresponding to the three-dimensional point; the color of the image point on the virtual view is obtained according to the weighted color brightness of all the two-dimensional Gaussian points at the point, and the specific formula is as follows: Among them, (c i ,α i ) is the color and opacity of the two-dimensional Gaussian point i, α′ i Represents the opacity of Gaussian point i at the image point.

8. The system according to claim 6, characterized in that Image position output module, including: A virtual view magnification submodule is used to divide the virtual view into four sub-blocks, locally magnify the visual attention area to highlight the characteristic information of the transmission tower and components, and input the magnified area and the entire virtual view into the backbone module in the network through the main road and the branch road respectively; The feature fusion submodule is used to perform feature fusion at multiple scales on the branch input of the enlarged area through the convolution layer and the convolution layer of the global area input.

9. The system according to claim 6, characterized in that Planning route determination module, including: The flight path determination submodule is used to use the minimum XY value of the transmission tower as the starting observation point of the drone route, determine another transmission tower closest to the current transmission tower through Euclidean distance as the second observation point of the drone route, and determine the line segment between the two points as the flight path of the drone between the two points; A starting point secondary confirmation submodule is used to take the second observation point as a new starting point and repeat the above steps until all the transmission towers are covered; The collection route calculation submodule is used to calculate the three-dimensional outer envelope surface according to the three-dimensional point information of the transmission tower and its components under the current observation point, and automatically calculate the transmission tower data collection route according to the flight mode of descending step by step and then ascending step by step back to the observation point; The inspection route acquisition submodule is used to connect the flight paths of drones between all transmission towers and the data collection routes of each transmission tower to obtain the drone inspection route for the scenario to be planned.

10. The system according to claim 6, characterized in that Also includes: The flight altitude setting module is used to set the flying altitude of the drone according to the height of the transmission tower.

Images