Path planning and defect identification method and device for unmanned aerial vehicle inspection

Abstract

The present invention discloses a method and device for path planning and defect identification in the process of unmanned aerial vehicle inspection, the method comprising: collecting oblique photography image data of the inspection area; generating point cloud data of the inspection area according to the oblique photography image data; reconstructing the point cloud data based on the average curvature smoothing and triangulation gridding method to obtain the real scene reconstruction result of the inspection area; gridding the real scene reconstruction result, dividing the area according to the divided grid, performing path planning based on the area division result, and optimizing the planned path to obtain the optimal inspection path; controlling the unmanned aerial vehicle to inspect according to the optimal inspection path, collecting target image data during the inspection process, and inputting the target image data into a pre-established defect detection model for defect identification. The method can effectively improve the inspection efficiency of unmanned aerial vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method and device for path planning and defect identification in an unmanned aerial vehicle inspection process. Background Art

[0002] Driven by science and technology, software and hardware technologies have gradually been improved, making intelligent equipment gradually mature. Compared with manual inspection operations, drone inspection technology developed based on intelligent technology has gradually occupied an important position in the inspection field.

[0003] As the speed of economic development increases exponentially, the landforms and buildings also increase exponentially with the development of the economy. Therefore, if the inspection process follows the original inspection path, the inspection path will be blocked. To address this problem, the industry has developed a variety of path planning solutions. For example, patent document CN112327920B discloses a method and device for autonomous obstacle avoidance inspection path planning for unmanned aerial vehicles, wherein the unmanned aerial vehicle inspection path planning method includes: establishing a three-dimensional model of the inspection area, the three-dimensional model including point cloud data of equipment in the inspection area; determining a three-dimensional inspection safety area based on the three-dimensional model and the no-fly policy; determining a first inspection target task point set based on the inspection plan and the three-dimensional model; and establishing a first inspection path based on the three-dimensional inspection safety area, the first inspection target task point set and the unmanned aerial vehicle information. For another example, patent document CN109358650B discloses a patrol path planning method, device, drone and computer-readable storage medium, the method including receiving a patrol target position set sent by a remote control terminal, selecting any patrol target position from the patrol target position set and determining a set of optional patrol positions corresponding to the patrol target position according to a first preset navigation obstacle avoidance condition, calculating the patrol coverage rate corresponding to each optional patrol position in the optional patrol position set, obtaining a patrol coverage rate set corresponding to the patrol target position, selecting the optional patrol position corresponding to the maximum value in the patrol coverage rate set as the patrol position corresponding to the patrol target position, calculating the patrol positions corresponding to all patrol target positions in the patrol target position set, planning a patrol path set according to a second preset navigation obstacle avoidance condition and all the above-mentioned patrol positions, determining an initial patrol path from the patrol path set according to the preset conditions and patrol navigating along the initial patrol path.

[0004] Although the above path planning scheme can achieve obstacle avoidance during the drone inspection process, the planned path is not the optimal path, and the planning speed is slow and the response time is long, which makes the drone inspection efficiency low. Summary of the invention

[0005] The present invention provides a method and device for path planning and defect identification in a drone inspection process, which can effectively improve the efficiency of drone inspection.

[0006] A path planning and defect identification method for unmanned aerial vehicle inspection process, comprising:

[0007] Collect oblique photography image data of the inspection area;

[0008] Generate point cloud data of the inspection area according to the oblique photography image data;

[0009] Reconstructing the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area;

[0010] Divide the real scene reconstruction result into grids, divide the regions according to the divided grids, plan the path based on the region division results, and optimize the planned path to obtain the optimal inspection path;

[0011] The drone is controlled to perform inspection along the optimal inspection path, target image data is collected during the inspection, and the target image data is input into a pre-established defect detection model for defect identification.

[0012] Furthermore, the defect detection model preprocesses the target image data, performs branching and fusion operations on the preprocessed target image data, and introduces an attention mechanism to obtain shallow target image features of different sizes and different numbers of channels; performs deep extraction on the shallow target image features, introduces an attention mechanism to obtain deep target image features, fuses the shallow target image features with the deep target image features to obtain fused target image features, performs defect recognition based on the fused target image features, and obtains a defect recognition result.

[0013] Furthermore, the point cloud data is reconstructed based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area, including:

[0014] Performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data;

[0015] Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh;

[0016] The low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

[0017] Further, the point cloud data is smoothed by average curvature, and the point cloud data is converted from a high-frequency scale space to a low-frequency scale space to obtain low-frequency point cloud data, including:

[0018] Selecting a data point in the point cloud data, obtaining a maximum curvature and a minimum curvature of the selected data point, and calculating an average curvature according to the maximum curvature and the minimum curvature;

[0019] According to the average curvature, the selected data points are moved and smoothed in the direction of the normal vector at a proportional speed to obtain low-frequency point cloud data and a spatial transformation relationship of the data points;

[0020] Converting the low-frequency reconstruction grid from the low-frequency scale space to the high-frequency scale space includes:

[0021] The low-frequency reconstruction grid is mapped according to the data point space conversion relationship to achieve conversion from low-frequency scale space to high-frequency scale space.

[0022] Further, performing triangulation meshing on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh includes:

[0023] Selecting an initial triangular mesh seed from the low-frequency point cloud data to form an initial triangular mesh;

[0024] Based on an edge of the initial triangular mesh, point cloud data satisfying a distance threshold is selected within a preset range to construct a first candidate point set;

[0025] Selecting unmeshed data points or data points located on the boundary from the first candidate point set as a second candidate point set for a new triangle;

[0026] Selecting data points satisfying the triangulation criterion from the second candidate point set as vertices of a new triangle to form a new triangle;

[0027] Add new triangles to the current triangle mesh;

[0028] After completing the triangulation of all data points in the low-frequency point cloud data, the low-frequency reconstructed mesh is obtained.

[0029] Furthermore, the initial triangle mesh seed is selected, including:

[0030] Selecting a random point from the low-frequency point cloud data;

[0031] An octree query method is used to search for data points whose distances from the random points are less than a preset threshold from the remaining low-frequency point cloud data to form an initial triangle candidate point set;

[0032] When the initial triangle candidate point set is not empty, two points satisfying a preset condition are selected from the initial triangle candidate point set and the random point to form an initial triangle mesh seed;

[0033] Among them, the preset conditions include: the two selected points and the random point are not on the same straight line; the sphere formed by the two selected points and the random point does not contain any point in the initial triangle candidate point set; any point in the initial triangle candidate point set is on the same side of the sphere formed by the two selected points and the random point.

[0034] Furthermore, the real scene reconstruction result is grid-divided, and regions are divided according to the divided grids, including:

[0035] Step A, dividing the real scene reconstruction result into grids of the same size, and taking each grid as a node;

[0036] Step B: take the grid where the path starting point is located as the root node to form the current sub-area, and take the grids adjacent to the path starting point as child nodes;

[0037] Step C: Determine whether there are nodes of the same type in the child nodes around the current root node. If there are nodes of the same type, jump to step E. If there are nodes of different types, jump to step D. The node types include blank nodes and non-blank nodes.

[0038] Step D: Use the child nodes of different types adjacent to the current root node as new root nodes, generate new sub-regions, increase the number of sub-regions by 1, and jump to step E;

[0039] Step E: Determine whether the number of nodes in the current area has reached a preset upper limit. If so, jump to step F; if not, jump to step G.

[0040] Step F, generate a new sub-area, increase the number of sub-areas by 1, and jump to step H;

[0041] Step G, adding child nodes of the same type to the current sub-region;

[0042] Step H: Determine whether all nodes are divided into sub-areas. If so, end the process; otherwise, jump to step C.

[0043] Furthermore, path planning is performed based on the area division results, and the planned path is optimized to obtain the optimal inspection path, including:

[0044] Each grid is regarded as a node, and the shortest path Dijkstra algorithm is used to plan the paths for different sub-areas to obtain the inspection paths between grids.

[0045] The second node of the inspection path is used as the starting reference point, and the connection line between the reference point and each subsequent node is searched to see whether there is an intersection with the obstacle. If there is no intersection, the node between the reference point and the connection node is deleted and the next node is connected. If there is an intersection, the third node is used as the reference point to continue searching until the end point of the inspection path becomes the reference point, and the deleted inspection path is obtained.

[0046] The deleted inspection path is optimized twice using the Bezier curve optimization method to obtain the optimal inspection path.

[0047] Furthermore, the oblique photography image data is image data synchronously collected by the drone from multiple different viewing angles.

[0048] A path planning and defect identification device for unmanned aerial vehicle inspection process, comprising:

[0049] An acquisition module, used to acquire oblique photography image data of the inspection area;

[0050] A point cloud generation module, used to generate point cloud data of the inspection area according to the oblique photography image data;

[0051] A point cloud reconstruction module reconstructs the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real-scene reconstruction result of the inspection area;

[0052] A path planning module is used to divide the real scene reconstruction result into grids, divide the regions according to the divided grids, plan the path based on the region division results, and optimize the planned path to obtain the optimal inspection path;

[0053] The detection module is used to control the UAV to perform inspections along the optimal inspection path, collect target image data during the inspection, and input the target image data into a pre-established defect detection model for defect identification.

[0054] Furthermore, the defect detection model preprocesses the target image data, performs branching and fusion operations on the preprocessed target image data, and introduces an attention mechanism to obtain shallow target image features of different sizes and different numbers of channels; performs deep extraction on the shallow target image features, introduces an attention mechanism to obtain deep target image features, fuses the shallow target image features with the deep target image features to obtain fused target image features, performs defect recognition based on the fused target image features, and obtains a defect recognition result.

[0055] Furthermore, the point cloud reconstruction module reconstructs the point cloud data based on average curvature smoothing and triangular meshing to obtain a real scene reconstruction result of the inspection area, including:

[0056] Performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data;

[0057] Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh;

[0058] The low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

[0059] Furthermore, the point cloud reconstruction module performs average curvature smoothing processing on the point cloud data, converts the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtains low-frequency point cloud data, including:

[0060] Selecting a data point in the point cloud data, obtaining a maximum curvature and a minimum curvature of the selected data point, and calculating an average curvature according to the maximum curvature and the minimum curvature;

[0061] According to the average curvature, the selected data points are moved and smoothed in the direction of the normal vector at a proportional speed to obtain low-frequency point cloud data and a spatial transformation relationship of the data points;

[0062] Converting the low-frequency reconstruction grid from the low-frequency scale space to the high-frequency scale space includes:

[0063] The low-frequency reconstruction grid is mapped according to the data point space conversion relationship to achieve conversion from low-frequency scale space to high-frequency scale space.

[0064] Furthermore, the point cloud reconstruction module performs triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh, including:

[0065] Selecting an initial triangular mesh seed from the low-frequency point cloud data to form an initial triangular mesh;

[0066] Based on an edge of the initial triangular mesh, point cloud data satisfying a distance threshold is selected within a preset range to construct a first candidate point set;

[0067] Selecting unmeshed data points or data points located on the boundary from the first candidate point set as a second candidate point set for a new triangle;

[0068] Selecting data points satisfying the triangulation criterion from the second candidate point set as vertices of a new triangle to form a new triangle;

[0069] Add new triangles to the current triangle mesh;

[0070] After completing the triangulation of all data points in the low-frequency point cloud data, the low-frequency reconstructed mesh is obtained.

[0071] Furthermore, the point cloud reconstruction module selects an initial triangle mesh seed, including:

[0072] Selecting a random point from the low-frequency point cloud data;

[0073] An octree query method is used to search for data points whose distances from the random points are less than a preset threshold from the remaining low-frequency point cloud data to form an initial triangle candidate point set;

[0074] When the initial triangle candidate point set is not empty, two points satisfying a preset condition are selected from the initial triangle candidate point set and the random point to form an initial triangle mesh seed;

[0075] Among them, the preset conditions include: the two selected points and the random point are not on the same straight line; the sphere formed by the two selected points and the random point does not contain any point in the initial triangle candidate point set; any point in the initial triangle candidate point set is on the same side of the sphere formed by the two selected points and the random point.

[0076] Furthermore, the path planning module divides the real scene reconstruction result into grids, and divides the regions according to the divided grids, including:

[0077] Step A, dividing the real scene reconstruction result into grids of the same size, and taking each grid as a node;

[0078] Step B: take the grid where the path starting point is located as the root node to form the current sub-area, and take the grids adjacent to the path starting point as child nodes;

[0079] Step C: Determine whether there are nodes of the same type in the child nodes around the current root node. If there are nodes of the same type, jump to step E. If there are nodes of different types, jump to step D. The node types include blank nodes and non-blank nodes.

[0080] Step D: Use the child nodes of different types adjacent to the current root node as new root nodes, generate new sub-regions, increase the number of sub-regions by 1, and jump to step E;

[0081] Step E: Determine whether the number of nodes in the current area has reached a preset upper limit. If so, jump to step F; if not, jump to step G.

[0082] Step F, generate a new sub-area, increase the number of sub-areas by 1, and jump to step H;

[0083] Step G, adding child nodes of the same type to the current sub-region;

[0084] Step H: Determine whether all nodes are divided into sub-areas. If so, end the process; otherwise, jump to step C.

[0085] Furthermore, the path planning module performs path planning based on the area division result, and optimizes the planned path to obtain the optimal inspection path, including:

[0086] Each grid is regarded as a node, and the shortest path Dijkstra algorithm is used to plan the paths for different sub-areas to obtain the inspection paths between grids.

[0087] The second node of the inspection path is used as the starting reference point, and the connection line between the reference point and each subsequent node is searched to see whether there is an intersection with the obstacle. If there is no intersection, the node between the reference point and the connection node is deleted and the next node is connected. If there is an intersection, the third node is used as the reference point to continue searching until the end point of the inspection path becomes the reference point, and the deleted inspection path is obtained.

[0088] The deleted inspection path is optimized twice using the Bezier curve optimization method to obtain the optimal inspection path.

[0089] Furthermore, the oblique photography image data is image data synchronously collected by the drone from multiple different viewing angles.

[0090] A computer-readable storage medium stores computer instructions, which are read by a processor and execute the above method.

[0091] The path planning and defect identification method and device for drone inspection provided by the present invention have at least the following beneficial effects:

[0092] (1) The working environment of the UAV is obtained through reconstruction and analysis of the working environment, and the optimal inspection path is generated according to the working environment to improve the inspection efficiency of the UAV. At the same time, image acquisition is performed based on the inspection path to complete the detection and identification analysis of equipment defects on the transmission line;

[0093] (2) In the process of performing 3D reconstruction, the frequency conversion of point cloud data and the region growing method are used to realize triangular mesh reconstruction of the real scene, thereby avoiding the problem of holes in the process of using triangular mesh to reconstruct the real scene;

[0094] (3) Based on the results of real-scene reconstruction, the optimal inspection path is generated by combining global and local analysis, which has fast planning speed and short response time, further improving the inspection efficiency of drones;

[0095] (4) By introducing the attention mechanism, the image feature extraction process can focus more on the defective part. At the same time, by combining channel information and spatial position information, the feature extraction has stronger anti-interference ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A flowchart of an embodiment of a method for path planning and defect identification during an unmanned aerial vehicle inspection process provided by the present invention.

[0097] Figure 2 A flowchart of an embodiment of point cloud data reconstruction in a path planning and defect identification method for an unmanned aerial vehicle inspection process provided by the present invention.

[0098] Figure 3 A flowchart of an embodiment of average curvature smoothing processing of point cloud data in the path planning and defect identification method for unmanned aerial vehicle inspection provided by the present invention.

[0099] Figure 4 A flowchart of an embodiment of performing triangulated meshing on low-frequency point cloud data in the path planning and defect identification method for unmanned aerial vehicle inspection provided by the present invention.

[0100] Figure 5 A flowchart of an embodiment of path planning in a method for path planning and defect identification in a drone inspection process provided by the present invention.

[0101] Figure 6 A schematic structural diagram of an embodiment of a path planning and defect identification device for unmanned aerial vehicle inspection provided by the present invention. DETAILED DESCRIPTION

[0102] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0103] refer to Figure 1 In some embodiments, a path planning and defect identification method for a drone inspection process is provided, including:

[0104] S1, collecting oblique photography image data of the inspection area;

[0105] S2. generating point cloud data of the inspection area according to the oblique photography image data;

[0106] S3, reconstructing the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area;

[0107] S4, dividing the real scene reconstruction result into grids, dividing regions according to the divided grids, performing path planning based on the region division results, and optimizing the planned path to obtain an optimal inspection path;

[0108] S5. Control the UAV to perform inspection along the optimal inspection path, collect target image data during the inspection, and input the target image data into a pre-established defect detection model for defect identification.

[0109] Furthermore, in step S1, the oblique photography image data is image data synchronously collected by the drone from multiple different viewing angles.

[0110] Oblique photography technology is a high-tech technology developed in the international surveying and mapping field in recent years. It overturns the previous limitation that orthophotos can only be taken from a vertical angle. By carrying multiple sensors on the same flying platform, it simultaneously collects images from five different angles, including one vertical, four oblique angles, etc., introducing users into a real and intuitive world that conforms to human vision.

[0111] According to the 3D reconstruction requirements of the current inspection area, the drone's aerial photography route is planned, and then the drone collects real-scene image data according to the planned aerial photography route. Optionally, there are a total of five sensors directly below and around the drone, and all-round data collection is achieved through the sensors directly below and around the drone to obtain oblique photography image data.

[0112] Furthermore, in step S2, based on the oblique photography image data, the images at different angles are converted into three-dimensional point cloud data using a technique such as triangulation to generate point cloud data of the inspection area.

[0113] Further, refer to Figure 2 In step S3, the point cloud data is reconstructed based on average curvature smoothing and triangular meshing to obtain a real scene reconstruction result of the inspection area, including:

[0114] S31, performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data;

[0115] S32, performing triangulation meshing on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh;

[0116] S33: Convert the low-frequency reconstruction grid from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

[0117] Among them, reference Figure 3In step S31, the point cloud data is smoothed by average curvature to convert the point cloud data from a high-frequency scale space to a low-frequency scale space to obtain low-frequency point cloud data, including:

[0118] S311, selecting a data point in the point cloud data, obtaining the maximum curvature and the minimum curvature of the selected data point, and calculating the average curvature according to the maximum curvature and the minimum curvature;

[0119] S312, moving and smoothing the selected data points in the direction of the normal vector at a proportional speed according to the average curvature to obtain low-frequency point cloud data and obtain a spatial conversion relationship of the data points.

[0120] Specifically, in the process of performing the mean curvature movement, the moving radius r is first preset, and the domain point set N of the selected point p is obtained within the range of the moving radius r. r (p); then traverse the data points in the domain point set and preset the domain point number threshold n; then, judge the size of the number in the domain point set and the number threshold, when the number in the set is less than the number threshold, move the data point p, obtain point p' and update the normal vector of the moved point p'; finally, preset the number of iterations, and perform the mean curvature move within the preset number of iterations, and obtain the conversion from high-frequency to low-frequency scale space after completing the mean curvature move at each point.

[0121] Specifically, the obtained point cloud data is smoothed by moving the average curvature to ensure that the detailed features are not ignored, wherein the average curvature represents the average value of any two mutually perpendicular orthogonal curvatures at any point on the spatial surface. In this embodiment, any data point p in the point cloud data is obtained, and the data point p is located on a continuous and smooth surface S. The normal vector corresponding to the data point p is n(p). L is the tangent plane perpendicular to the data point p. The tangent vector t(p) passing through the data point p is selected on the tangent plane L. A plane Q is formed by the normal vector n(p) and the tangent vector t(p). The plane Q intersects with the surface S to obtain a curve l. The curvature of the curve l at the data point p is k. Since there are countless tangent vectors passing through the data point p on the surface S, there are countless corresponding curves. The maximum curvature k of these curves at the data point p is read. max and the minimum curvature k min , and the mean curvature is obtained by the expression of the mean curvature shift, where the expression of the mean curvature is:

[0122]

[0123] Where H(p) is the average curvature of data point p, k max is the maximum curvature at data point p, k min is the minimum curvature at data point p.

[0124] Further, in step S312, according to the obtained average curvature, the selected data point p is smoothed in the direction of the normal vector of the data point at a speed proportional to the average curvature at the selected point, wherein the corresponding smoothing expression is:

[0125]

[0126] Where H(p) represents the mean curvature at data point p, and n(p) represents the normal vector of data point p.

[0127] Further, refer to Figure 4 In step S32, triangulating the low-frequency point cloud data to obtain a low-frequency reconstructed mesh includes:

[0128] S321, selecting an initial triangular mesh from the low-frequency point cloud data to form an initial triangular mesh;

[0129] S322, based on an edge of the initial triangular mesh, selecting point cloud data that meets a distance threshold within a preset range to construct a first candidate point set;

[0130] S323, selecting data points that are not gridded or data points located on the boundary from the first candidate point set as a second candidate point set for a new triangle;

[0131] S324, selecting data points that meet the triangulation criterion from the second candidate point set as vertices of a new triangle to form a new triangle;

[0132] S325, adding the new triangle to the current triangle mesh;

[0133] S326: After completing the triangulation of all data points in the low-frequency point cloud data, obtain the low-frequency reconstructed mesh.

[0134] Specifically, in step S321, further, selecting an initial triangle mesh seed includes:

[0135] Selecting a random point from the low-frequency point cloud data;

[0136] An octree query method is used to search for data points whose distances from the random points are less than a preset threshold from the remaining low-frequency point cloud data to form an initial triangle candidate point set;

[0137] When the initial triangle candidate point set is not empty, two points satisfying a preset condition are selected from the initial triangle candidate point set and the random point to form an initial triangle mesh seed;

[0138] Among them, the preset conditions include: the two selected points and the random point are not on the same straight line; the sphere formed by the two selected points and the random point does not contain any point in the initial triangle candidate point set; any point in the initial triangle candidate point set is on the same side of the sphere formed by the two selected points and the random point.

[0139] Specifically, if the initial triangle candidate point set is empty, a random point is reselected and the above steps are performed.

[0140] In step S324, the triangulation criteria are met, including:

[0141] a. Empty circle feature: The Delaunay triangulation is unique. There will be no other points within the circumscribed circle of any triangle in the Delaunay triangulation.

[0142] b. Maximizing the minimum angle property: Among the possible triangulations formed by the scattered point set, the minimum angle of the triangle formed by the Delaunay triangulation is the largest. In this sense, the Delaunay triangulation is the "closest to regularization" triangulation. Specifically, it means that after the diagonals of the convex quadrilateral formed by two adjacent triangles are exchanged, the minimum angle of the six internal angles will no longer increase.

[0143] Further, in step S33, the low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space, including:

[0144] The low-frequency reconstruction grid is mapped according to the data point space conversion relationship to achieve conversion from low-frequency scale space to high-frequency scale space.

[0145] After the conversion from high frequency to low frequency, the conversion relationship between the data points can be obtained. According to the conversion relationship, the data points can be mapped back to achieve the conversion from low frequency to high frequency.

[0146] Three-dimensional reconstruction is achieved through the collected image data. As one of the environmental perception technologies, three-dimensional reconstruction technology has gradually occupied a place in urban reconstruction, virtual reality, behavior analysis and other fields. Therefore, in the process of performing real-scene reconstruction, three-dimensional point cloud technology is often used to achieve three-dimensional real-scene reconstruction. However, small defects on the surface of the target object are often difficult to reproduce, which leads to the reconstruction effect not being able to meet actual needs.

[0147] In order to solve the problems existing in the prior art, this embodiment realizes triangular mesh real scene reconstruction by frequency conversion of point cloud data and region growing method during the three-dimensional reconstruction process, thereby improving the defect of holes appearing in the process of real scene reconstruction using triangular mesh.

[0148] Furthermore, in step S4, the real scene reconstruction result is grid-divided, and regions are divided according to the divided grids, including:

[0149] Step A, dividing the real scene reconstruction result into grids of the same size, and taking each grid as a node;

[0150] Step B: take the grid where the path starting point is located as the root node to form the current sub-area, and take the grids adjacent to the path starting point as child nodes;

[0151] Step C: Determine whether there are nodes of the same type in the child nodes around the current root node. If there are nodes of the same type, jump to step E. If there are nodes of different types, jump to step D. The node types include blank nodes and non-blank nodes.

[0152] Step D: Use the child nodes of different types adjacent to the current root node as new root nodes, generate new sub-regions, increase the number of sub-regions by 1, and jump to step E;

[0153] Step E: Determine whether the number of nodes in the current area has reached a preset upper limit. If so, jump to step F; if not, jump to step G.

[0154] Step F, generate a new sub-area, increase the number of sub-areas by 1, and jump to step H;

[0155] Step G, adding child nodes of the same type to the current sub-region;

[0156] Step H: Determine whether all nodes are divided into sub-areas. If so, end the process; otherwise, jump to step C.

[0157] Specifically, the blank nodes are the traversable areas for the UAV, and the non-empty nodes are the obstacle areas where the UAV cannot traverse. The traversable areas for the UAV are obtained by dividing the areas, and the subsequent path planning is based on the traversable areas.

[0158] A topological structure is used to divide the real-scene reconstruction results of the area to be analyzed into different sub-areas. In each sub-area, only the starting point, end point and obstacles of the inspection path in the area are considered. At the same time, a heuristic function is constructed for optimization to ensure efficiency while improving accuracy.

[0159] Further, refer to Figure 5 In step S4, path planning is performed based on the area division result, and the planned path is optimized to obtain the optimal inspection path, including:

[0160] S41, taking each grid as a node, using the shortest path Dijkstra algorithm to plan paths for different sub-areas, and obtaining inspection paths between grids;

[0161] S42, taking the second node of the inspection path as the starting reference point, searching whether the line connecting the reference point and each subsequent node has an intersection with the obstacle, if no intersection exists, deleting the node between the reference point and the connecting node and continuing to connect the next node; if an intersection exists, taking the third node as the reference point and continuing to search until the end point of the inspection path becomes the reference point, and obtaining the deleted inspection path;

[0162] S43, performing secondary optimization on the deleted inspection path by using Bezier curve optimization method to obtain the optimal inspection path.

[0163] Specifically, in step S43, the deleted inspection path is optimized twice using the Bezier curve optimization method, and the shape of the curve is defined by the control points. The curve smoothing optimization expression is:

[0164]

[0165] Where P i represents the i-th control point; B i,n represents the nth-degree Bernstein polynomial.

[0166]

[0167] The inspection path optimized by removing redundant nodes and smoothing curves can effectively shorten the length of the inspection path and the turning points of the path.

[0168] Furthermore, since the target image data collected by the drone uses image compression technology during the transmission process, it often causes information loss and artifacts, making the defect detection result less accurate. Therefore, in step S5, after collecting the target image data during the inspection process, it also includes: performing an image super-resolution operation on the target image data using a preset image super-resolution model, and the image super-resolution model includes a preliminary extraction module, a residual connection module, and a reconstruction module;

[0169] Furthermore, in step S5, the defect detection model preprocesses the target image data, performs branching and fusion operations on the preprocessed target image data, and introduces an attention mechanism to obtain shallow target image features of different sizes and different numbers of channels; performs deep extraction on the shallow target image features, introduces an attention mechanism to obtain deep target image features, fuses the shallow target image features with the deep target image features to obtain fused target image features, performs defect recognition based on the fused target image features, and obtains a defect recognition result.

[0170] Specifically, the defect detection model preprocesses the target image data, including adjusting the size, to obtain an image size that meets the requirements for defect detection.

[0171] Furthermore, the shallow target image features are deeply extracted to obtain deep target image features including color, texture, edge, and corner.

[0172] Among them, in the process of performing defect detection analysis, since the target is a smaller part, it is easily affected by background factors, which leads to low efficiency in extracting effective features in the process of image feature extraction. For this reason, this embodiment further proposes to introduce an attention mechanism in the defect detection model to improve the extraction of key information.

[0173] The attention mechanism introduced in this embodiment is embedded in the stage of shallow feature extraction, and is used in the last layer of the deep feature extraction module. In the process of data processing, the spatial information is weighted and fused on the channel, and the information of the two dimensions of direction and position is obtained simultaneously in the process of feature extraction. Specifically, the received input features are fused and encoded with the corresponding coordinate information, and then the weights corresponding to different positions are obtained by the attention mechanism to obtain the weighted image feature representation.

[0174] The introduction of the attention mechanism allows the image feature extraction process to focus more on the defective part. At the same time, the combination of channel information and spatial position information makes the feature extraction more resistant to interference.

[0175] The path planning method for drone inspection provided in the above embodiment has at least the following beneficial effects:

[0176] (1) The working environment of the UAV is obtained through reconstruction and analysis of the working environment, and the optimal inspection path is generated according to the working environment to improve the inspection efficiency of the UAV. At the same time, image acquisition is performed based on the inspection path to complete the detection and identification analysis of equipment defects on the transmission line;

[0177] (2) In the process of performing 3D reconstruction, the frequency conversion of point cloud data and the region growing method are used to realize triangular mesh reconstruction of the real scene, thereby avoiding the problem of holes in the process of using triangular mesh to reconstruct the real scene;

[0178] (3) Based on the results of real-scene reconstruction, the optimal inspection path is generated by combining global and local analysis, which has fast planning speed and short response time, further improving the inspection efficiency of drones;

[0179] (4) The introduction of the attention mechanism makes the image feature extraction process more focused on the defect part. At the same time, the combination of channel information and spatial position information makes the feature extraction more resistant to interference.

[0180] refer to Figure 6 In some embodiments, a path planning and defect identification device for drone inspection is provided, including:

[0181] The acquisition module 201 is used to acquire oblique photography image data of the inspection area;

[0182] A point cloud generation module 202, used to generate point cloud data of the inspection area according to the oblique photography image data;

[0183] The point cloud reconstruction module 203 reconstructs the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area;

[0184] The path planning module 204 is used to divide the real scene reconstruction result into grids, divide the regions according to the divided grids, plan the path based on the region division results, and optimize the planned path to obtain the optimal inspection path;

[0185] The detection module 205 is used to control the UAV to perform inspections along the optimal inspection path, collect target image data during the inspection process, and input the target image data into a pre-established defect detection model for defect identification.

[0186] Furthermore, the oblique photography image data is image data synchronously collected by the drone from multiple different viewing angles.

[0187] Furthermore, the point cloud reconstruction module 203 reconstructs the point cloud data based on average curvature smoothing and triangular meshing to obtain a real scene reconstruction result of the inspection area, including:

[0188] Performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data;

[0189] Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh;

[0190] The low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

[0191] Furthermore, the point cloud reconstruction module 203 performs average curvature smoothing processing on the point cloud data, converts the point cloud data from the high-frequency scale space to the low-frequency scale space, and obtains low-frequency point cloud data, including:

[0192] Selecting a data point in the point cloud data, obtaining a maximum curvature and a minimum curvature of the selected data point, and calculating an average curvature according to the maximum curvature and the minimum curvature;

[0193] According to the average curvature, the selected data points are moved and smoothed in the direction of the normal vector at a proportional speed to obtain low-frequency point cloud data and a spatial transformation relationship of the data points;

[0194] The point cloud reconstruction module converts the low-frequency reconstructed grid from the low-frequency scale space to the high-frequency scale space, including:

[0195] The low-frequency reconstruction grid is mapped according to the data point space conversion relationship to achieve conversion from low-frequency scale space to high-frequency scale space.

[0196] Furthermore, the point cloud reconstruction module 203 performs triangulation meshing on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh, including:

[0197] Selecting an initial triangular mesh seed from the low-frequency point cloud data to form an initial triangular mesh;

[0198] Based on an edge of the initial triangular mesh, point cloud data satisfying a distance threshold is selected within a preset range to construct a first candidate point set;

[0199] Selecting unmeshed data points or data points located on the boundary from the first candidate point set as a second candidate point set for a new triangle;

[0200] Selecting data points satisfying the triangulation criterion from the second candidate point set as vertices of a new triangle to form a new triangle;

[0201] Add new triangles to the current triangle mesh;

[0202] After completing the triangulation of all data points in the low-frequency point cloud data, the low-frequency reconstructed mesh is obtained.

[0203] Furthermore, the point cloud reconstruction module 203 selects an initial triangle mesh seed, including:

[0204] Selecting a random point from the low-frequency point cloud data;

[0205] An octree query method is used to search for data points whose distances from the random points are less than a preset threshold from the remaining low-frequency point cloud data to form an initial triangle candidate point set;

[0206] When the initial triangle candidate point set is not empty, two points satisfying a preset condition are selected from the initial triangle candidate point set and the random point to form an initial triangle mesh seed;

[0207] Among them, the preset conditions include: the two selected points and the random point are not on the same straight line; the sphere formed by the two selected points and the random point does not contain any point in the initial triangle candidate point set; any point in the initial triangle candidate point set is on the same side of the sphere formed by the two selected points and the random point.

[0208] Furthermore, the path planning module 204 divides the real scene reconstruction result into grids, and divides the regions according to the divided grids, including:

[0209] Step A, dividing the real scene reconstruction result into grids of the same size, and taking each grid as a node;

[0210] Step B: take the grid where the path starting point is located as the root node to form the current sub-area, and take the grids adjacent to the path starting point as child nodes;

[0211] Step C: Determine whether there are nodes of the same type in the child nodes around the current root node. If there are nodes of the same type, jump to step E. If there are nodes of different types, jump to step D. The node types include blank nodes and non-blank nodes.

[0212] Step D: Use the child nodes of different types adjacent to the current root node as new root nodes, generate new sub-regions, increase the number of sub-regions by 1, and jump to step E;

[0213] Step E: Determine whether the number of nodes in the current area has reached a preset upper limit. If so, jump to step F; if not, jump to step G.

[0214] Step F, generate a new sub-area, increase the number of sub-areas by 1, and jump to step H;

[0215] Step G, adding child nodes of the same type to the current sub-region;

[0216] Step H: Determine whether all nodes are divided into sub-areas. If so, end the process; otherwise, jump to step C.

[0217] Furthermore, the path planning module 204 performs path planning based on the area division result, and optimizes the planned path to obtain the optimal inspection path, including:

[0218] Each grid is regarded as a node, and the shortest path Dijkstra algorithm is used to plan the paths for different sub-areas to obtain the inspection paths between grids.

[0219] The second node of the inspection path is used as the starting reference point, and the connection line between the reference point and each subsequent node is searched to see whether there is an intersection with the obstacle. If there is no intersection, the node between the reference point and the connection node is deleted and the next node is connected. If there is an intersection, the third node is used as the reference point to continue searching until the end point of the inspection path becomes the reference point, and the deleted inspection path is obtained.

[0220] The deleted inspection path is optimized twice using the Bezier curve optimization method to obtain the optimal inspection path.

[0221] Furthermore, the detection module 205 preprocesses the target image data based on the defect detection model, performs branching and fusion operations on the preprocessed target image data, and introduces an attention mechanism to obtain shallow target image features of different sizes and different numbers of channels; performs deep extraction on the shallow target image features, and introduces an attention mechanism to obtain deep target image features, fuses the shallow target image features with the deep target image features to obtain fused target image features, performs defect recognition based on the fused target image features, and obtains a defect recognition result.

[0222] Specifically, the defect detection model preprocesses the target image data, including adjusting the size, to obtain an image size that meets the requirements for defect detection.

[0223] Furthermore, the shallow target image features are deeply extracted to obtain deep target image features including color, texture, edge, and corner.

[0224] Among them, in the process of performing defect detection analysis, since the target is a smaller part, it is easily affected by background factors, which leads to low efficiency in extracting effective features in the process of image feature extraction. For this reason, this embodiment further proposes to introduce an attention mechanism in the defect detection model to improve the extraction of key information.

[0225] Please refer to the method embodiment for the specific principle, which will not be described here in detail.

[0226] In some embodiments, a computer-readable storage medium is also provided, on which computer instructions are stored, and the computer instructions are read by a processor to execute the above method.

[0227] The path planning and defect identification device for drone inspection provided in the above embodiment has at least the following beneficial effects:

[0228] (1) The working environment of the UAV is obtained through reconstruction and analysis of the working environment, and the optimal inspection path is generated according to the working environment to improve the inspection efficiency of the UAV. At the same time, image acquisition is performed based on the inspection path to complete the detection and identification analysis of equipment defects on the transmission line;

[0229] (2) In the process of performing 3D reconstruction, the frequency conversion of point cloud data and the region growing method are used to realize triangular mesh reconstruction of the real scene, thereby avoiding the problem of holes in the process of using triangular mesh to reconstruct the real scene;

[0230] (3) Based on the results of real-scene reconstruction, the optimal inspection path is generated by combining global and local analysis, which has fast planning speed and short response time, further improving the inspection efficiency of drones;

[0231] (4) The introduction of the attention mechanism makes the image feature extraction process more focused on the defect part. At the same time, the combination of channel information and spatial position information makes the feature extraction more resistant to interference.

[0232] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A path planning and defect identification method for unmanned aerial vehicle inspection, characterized in that: include: Collect oblique photography image data of the inspection area; Generate point cloud data of the inspection area according to the oblique photography image data; Reconstructing the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area; Divide the real scene reconstruction result into grids, divide the regions according to the divided grids, plan the path based on the region division results, and optimize the planned path to obtain the optimal inspection path; Controlling the UAV to perform inspection along the optimal inspection path, collecting target image data during the inspection, and inputting the target image data into a pre-established defect detection model for defect identification; The point cloud data is reconstructed based on average curvature smoothing and triangulation meshing to obtain a real scene reconstruction result of the inspection area, including: Performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data; Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh; The low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

2. The method according to claim 1, characterized in that The defect detection model preprocesses the target image data, performs branching and fusion operations on the preprocessed target image data, and introduces an attention mechanism to obtain shallow target image features of different sizes and different numbers of channels; performs deep extraction on the shallow target image features, introduces an attention mechanism to obtain deep target image features, fuses the shallow target image features with the deep target image features to obtain fused target image features, performs defect recognition based on the fused target image features, and obtains a defect recognition result.

3. The method according to claim 1, characterized in that The point cloud data is subjected to average curvature smoothing processing, and the point cloud data is converted from a high-frequency scale space to a low-frequency scale space to obtain low-frequency point cloud data, including: Selecting a data point in the point cloud data, obtaining a maximum curvature and a minimum curvature of the selected data point, and calculating an average curvature according to the maximum curvature and the minimum curvature; According to the average curvature, the selected data points are moved and smoothed in the direction of the normal vector at a proportional speed to obtain low-frequency point cloud data and a spatial transformation relationship of the data points; Converting the low-frequency reconstruction grid from the low-frequency scale space to the high-frequency scale space includes: The low-frequency reconstruction grid is mapped according to the data point space conversion relationship to achieve conversion from low-frequency scale space to high-frequency scale space.

4. The method according to claim 2, characterized in that: Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh includes: Selecting an initial triangular mesh seed from the low-frequency point cloud data to form an initial triangular mesh; Based on an edge of the initial triangular mesh, point cloud data satisfying a distance threshold is selected within a preset range to construct a first candidate point set; Selecting unmeshed data points or data points located on the boundary from the first candidate point set as a second candidate point set for a new triangle; Selecting data points satisfying the triangulation criterion from the second candidate point set as vertices of a new triangle to form a new triangle; Add new triangles to the current triangle mesh; After completing the triangulation of all data points in the low-frequency point cloud data, the low-frequency reconstructed mesh is obtained.

5. The method according to claim 4, characterized in that Select the initial triangle mesh seed, including: Selecting a random point from the low-frequency point cloud data; An octree query method is used to search for data points whose distances from the random points are less than a preset threshold from the remaining low-frequency point cloud data to form an initial triangle candidate point set; When the initial triangle candidate point set is not empty, two points satisfying a preset condition are selected from the initial triangle candidate point set and the random point to form an initial triangle mesh seed; Among them, the preset conditions include: the two selected points and the random point are not on the same straight line; the sphere formed by the two selected points and the random point does not contain any point in the initial triangle candidate point set; any point in the initial triangle candidate point set is on the same side of the sphere formed by the two selected points and the random point.

6. The method according to claim 1, characterized in that The real scene reconstruction result is grid-divided, and regions are divided according to the divided grids, including: Step A, dividing the real scene reconstruction result into grids of the same size, and taking each grid as a node; Step B: take the grid where the path starting point is located as the root node to form the current sub-area, and take the grids adjacent to the path starting point as child nodes; Step C: Determine whether there are nodes of the same type in the child nodes around the current root node. If there are nodes of the same type, jump to step E. If there are nodes of different types, jump to step D. The node types include blank nodes and non-blank nodes. Step D: Use the child nodes of different types adjacent to the current root node as new root nodes, generate new sub-regions, increase the number of sub-regions by 1, and jump to step E; Step E: Determine whether the number of nodes in the current area has reached a preset upper limit. If so, jump to step F; if not, jump to step G. Step F, generate a new sub-area, increase the number of sub-areas by 1, and jump to step H; Step G, adding child nodes of the same type to the current sub-region; Step H: Determine whether all nodes are divided into sub-areas. If so, end the process; otherwise, jump to step C.

7. The method according to claim 6, characterized in that Path planning is performed based on the area division results, and the planned path is optimized to obtain the optimal inspection path, including: Each grid is regarded as a node, and the shortest path Dijkstra algorithm is used to plan the paths for different sub-areas to obtain the inspection paths between grids. The second node of the inspection path is used as the starting reference point, and the connection line between the reference point and each subsequent node is searched to see whether there is an intersection with the obstacle. If there is no intersection, the node between the reference point and the connection node is deleted and the next node is connected. If there is an intersection, the third node is used as the reference point to continue searching until the end point of the inspection path becomes the reference point, and the deleted inspection path is obtained. The deleted inspection path is optimized twice using the Bezier curve optimization method to obtain the optimal inspection path.

8. A path planning and defect identification device for drone inspection process applied to the method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire oblique photography image data of the inspection area; A point cloud generation module, used to generate point cloud data of the inspection area according to the oblique photography image data; A point cloud reconstruction module reconstructs the point cloud data based on average curvature smoothing and triangulation meshing to obtain a real-scene reconstruction result of the inspection area; A path planning module is used to divide the real scene reconstruction result into grids, divide the regions according to the divided grids, plan the path based on the region division results, and optimize the planned path to obtain the optimal inspection path; A detection module is used to control the UAV to perform inspections along the optimal inspection path, collect target image data during the inspection process, and input the target image data into a pre-established defect detection model for defect identification; The point cloud reconstruction module reconstructs the point cloud data based on average curvature smoothing and triangular meshing to obtain a real scene reconstruction result of the inspection area, including: Performing average curvature smoothing processing on the point cloud data, converting the point cloud data from a high-frequency scale space to a low-frequency scale space, and obtaining low-frequency point cloud data; Performing triangulation on the low-frequency point cloud data to obtain a low-frequency reconstructed mesh; The low-frequency reconstruction grid is converted from the low-frequency scale space to the high-frequency scale space to obtain the real scene reconstruction result.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: The computer instructions are read by a processor and execute the method according to any one of claims 1-7.

Images