Unmanned aerial vehicle approaching image acquisition method based on incremental reconstruction

Abstract

The invention discloses an unmanned aerial vehicle approaching image acquisition method based on incremental reconstruction. The method comprises the following steps: performing oblique photography or multi-view image acquisition on a target structure by using an unmanned aerial vehicle, and reconstructing the target structure into a low-precision point cloud; dispersing the three-dimensional grid model of the target structure into point clouds, and matching the point clouds with low-precision point clouds to serve as target point clouds; acquiring a flight space and an image acquisition candidate point set; establishing a camera main view model at each image acquisition candidate point; target grid visual information under each camera model is calculated; dividing the visible target grids into I-class grids and II-class grids, and obtaining an effective viewpoint set; obtaining a working viewpoint from the effective viewpoint set; and generating an unmanned aerial vehicle flight path in the flight space based on the working viewpoint coordinates. According to the method, the unmanned aerial vehicle flies close to the target structure through the viewpoint preprocessing method, and the image pixel precision is improved; static obstacle avoidance of the unmanned aerial vehicle is realized through a viewpoint preprocessing method; and through a viewpoint screening method, the three-dimensional reconstruction precision is ensured.

Description

Technical Field

[0001] The present invention relates to the fields of information technology and building safety management, and in particular to a method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction. Background Art

[0002] Thanks to the maturity of point cloud processing algorithms, 3D point cloud data has become one of the main data sources for digital dimension detection and virtual pre-assembly of building and bridge structures. Combined with 3D point data with high coverage of the structure surface and efficient data processing algorithms, point cloud technology can obtain more comprehensive structural geometric information and extract structural geometric features more conveniently than traditional measurement methods such as total stations, rulers, and rulers.

[0003] Land-based 3D laser scanners have a large scanning range, and the point accuracy is usually maintained at around ±2mm, which can meet the point cloud accuracy requirements of most large steel structures. However, it is necessary to set up a measuring station. At the top of some large steel structures, there is often no station condition, and the bottom station cannot collect data on the top of the structure, which leads to the loss of important point cloud data at the top, such as the upper chord of the truss structure, thus affecting the accurate extraction of geometric features. Oblique photography and multi-view reconstruction based on drones can reconstruct 3D point clouds by collecting images of the top of the structure, but due to the complex construction site environment, the flight altitude often needs to exceed the highest point on site, resulting in insufficient pixel accuracy, and the point cloud reconstruction accuracy cannot meet the requirements of structural size detection and pre-assembly.

[0004] Therefore, in order to solve the problems of incomplete acquisition and insufficient accuracy of point cloud data in the digital size detection and pre-assembly process of large steel structures, it is urgent to develop a UAV close-range image acquisition method based on incremental reconstruction. Summary of the invention

[0005] The purpose of the present invention is to provide a method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction, comprising the following steps:

[0006] 1) Using a drone to collect oblique photography images and / or multi-view images of the target structure, and reconstructing the collected images into low-precision image point clouds;

[0007] 2) Obtain a three-dimensional mesh model of the target structure and discretize it into a point cloud of the target structure;

[0008] 3) Match the target structure point cloud with the low-precision image point cloud, and record the coordinate transformation matrix M during the matching process. The matched target structure point cloud is used as the target point cloud, and the low-precision point cloud with a distance greater than dmin from the target point cloud is used as the obstacle point cloud;

[0009] 4) Based on the target point cloud and obstacle point cloud, construct the target grid and obstacle grid respectively;

[0010] 5) Obtain the flight space and image acquisition candidate point set;

[0011] 6) Establishing the camera main view model at each image acquisition candidate point;

[0012] 7) Create a light projection scene based on the target grid, obstacle grid and camera model, and calculate the visible information of the target grid under each camera model, including the index of each visible target grid and the distance between each visible target grid and the candidate point of the drone image acquisition;

[0013] 8) Based on the visible information of the target grid, the visible target grid under each camera model is divided into type I grid and type II grid, and the candidate points for drone image acquisition with the number of type I grids greater than N are selected as the valid viewpoint set;

[0014] 9) Obtain working viewpoints from the valid viewpoint set;

[0015] 10) Generate the UAV flight path in the flight space based on the working viewpoint coordinates;

[0016] 11) Based on the flight path of the UAV and the camera external parameters of the working viewpoint, calculate the flight parameters of the UAV at the working viewpoint and generate a flight file.

[0017] Further, in step 4), the target point cloud is encapsulated into a target grid, or the three-dimensional grid of the target structure transformed by the coordinate transformation matrix M is encapsulated into a target grid.

[0018] Further, in step 5), the step of obtaining the flight space and the image acquisition candidate point set includes:

[0019] 5.1) Create a semi-directed bounding box of the target mesh and expand the bounding box to the side and upwards. The expansion size is d = (P / λ) 2 pf eq / S 35 , the side length of the expanded bounding box is taken as an integer multiple of the drone hovering accuracy; P is the expected point cloud position tolerance, λ∈[5,10] is the conversion factor between pixel accuracy and point cloud tolerance; p is pixel accuracy, f eq is the camera’s 35mm equivalent focal length, S 35 It is 35mm film size;

[0020] 5.2) According to the second rule, the expansion bounding box is divided into voxels with a spacing of k times the hovering accuracy of the drone, and the coordinates of the center point of each voxel in the voxel are calculated as the point set A; k is a natural number not less than 1;

[0021] 5.3) Divide point set A. If the voxel represented by a point in point set A contains the target point cloud, the point is divided into point set B. If the voxel contains the obstacle point cloud, the point is divided into point set C. If the voxel does not contain the target point cloud and the obstacle point cloud, the point is divided into point set D.

[0022] 5.4) According to the third rule, remove from point set D the points whose distance to the closest points in point set B and point set C is less than the safety distance d. safe points, and obtain the point set E;

[0023] 5.5) Remove the points directly below point set B and point set C from point set E to obtain point set F;

[0024] 5.6) Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the flight space of the UAV;

[0025] 5.7) Remove points whose distance to the nearest point of the target point cloud is less than d from the point set G to obtain the point set H;

[0026] 5.8) Take each point in the point set H as the starting point, point to the nearest point in the target point cloud, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition.

[0027] Furthermore, the height direction of the target grid semi-directed bounding box in step 5.1) is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY plane.

[0028] Further, the safety distance d in step 5.4) safe , not less than the sum of the drone’s hovering accuracy, the estimated accuracy of the low-precision point cloud, and the drone’s minimum obstacle avoidance distance.

[0029] Further, in step 6), when establishing the camera main view model at each image acquisition candidate point, the camera external parameter P c is a matrix of 4 rows and 4 columns; the camera external parameter P c The first 3 rows and columns are the orthogonal rotation matrix R = [η 1 ,η 2 ,η 3 ], the last column of the first three rows is the offset column vector T, and the fourth row is [0,0,0,1];

[0030] η 1 ,η 2 ,η 3 are all unit row vectors, η 3 is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η 1 At the same time, perpendicular to the Z axis and η 3, η 2 At the same time perpendicular to η 1 and η 3 ; The offset column vector T is obtained by multiplying R by the coordinates of the candidate image acquisition points and taking the negative value;

[0031] The camera intrinsic parameters are obtained through manufacturer instructions, calibration of calibration plates, or during low-precision point cloud reconstruction.

[0032] Further, in step 8), the type I grid is a grid whose distance between the visible target grid and the candidate point for drone image acquisition is within a threshold value d; the type II grid is a grid whose distance between the visible target grid and the candidate point for drone image acquisition is within a threshold value d=(P / λ) 2 pf eq / S 35 Outside the grid.

[0033] Further, in step 9), the step of obtaining a working viewpoint from the valid viewpoint set includes:

[0034] 9.1) Initialize the grid coverage to 0 and initialize the observed grid set to an empty set;

[0035] 9.2) Calculate the grid visibility table, which contains the grid index and the number of times each grid is observed by valid viewpoints as a type I grid and a type II grid. and And go to step 9.3); i is the grid number; if the number If it is less than 3, the grid is considered to be not visible enough. Before executing step 9.3), and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible;

[0036] 9.3) Calculate the weight of effective viewpoints j is the viewpoint number; a is the visibility identifier of the type I grid, which is 1 when the grid i is a type I grid and can be seen by the valid viewpoint j, otherwise it is 0; b is the visibility identifier of the type II grid, which is 1 when the grid i is a type II grid and can be seen by the valid viewpoint j, otherwise it is 0; γ is the weight correction parameter of the type II grid, which is a real number less than 1 and greater than 0;

[0037] 9.4) Take the viewpoint with the largest weight as the new working viewpoint, add it to the working viewpoint set, and obtain the visual grid of the new working viewpoint;

[0038] 9.5) Add the Class I grid index in the newly added working viewpoint visible grid to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0039] 9.6) Update effective viewpoint weights c is the newly added Class I grid visibility identifier. When grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, c is 1, otherwise c is 0;

[0040] 9.7) From the valid viewpoints that are not in the working viewpoint set, calculate the number of target grids that observe the same target grid as each working viewpoint in the working viewpoint set, as the number of common view grids;

[0041] Calculate the number of newly added type I grids compared to the observed grid set as the number of newly added type I grids;

[0042] If the ratio of the number of common view grids to the total number of valid viewpoint observation grids not in the working viewpoint set, and to the total number of corresponding working viewpoint observation grids, exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set; otherwise, the valid viewpoint and its nearest point in the target point cloud, the working viewpoint with the largest number of common view grids with the valid viewpoint, and its nearest point in the target point cloud are added to the oblique viewpoint information cache set;

[0043] 9.8) When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step 9.1), otherwise select the valid viewpoint with the largest weight from the newly added working viewpoint cache set as the newly added working viewpoint and add it to the working viewpoint set;

[0044] 9.9) Calculate the ratio of the number of newly added working viewpoint visible type I grids compared to the observed grid set to the total number of target grids as the new addition rate;

[0045] 9.10) Add the newly added working viewpoint visible Class I grid index to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0046] When the number of viewpoints in the working viewpoint set is less than N max , when the addition rate is less than 0.001 and the grid coverage rate is less than f, return to step 9.6);

[0047] 9.11) According to the visible Class I grids of each viewpoint in the working viewpoint set, calculate the number of observations of each target grid, and select the target grids with 1 and 2 grid observations as the grids to be supplemented;

[0048] 9.12) Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented from large to small, traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose observation number is not 0 according to the Class I grid index observed by the valid viewpoint;

[0049] 9.13) Repeat step 9.12) until the target grid observation times do not exist 1 time and 2 times, or there are no valid viewpoints that meet the conditions of the previous step, and output the working viewpoint set;

[0050] Further, in step 9.8), the step of adding a valid oblique viewpoint to the valid viewpoint set includes:

[0051] Sort the valid viewpoints in the squint viewpoint information cache set from large to small according to the number of common view grids;

[0052] Traverse the sorted oblique viewpoint cache set, select the valid viewpoint whose nearest point in the target point cloud and the nearest point in the target point cloud of the working viewpoint are not the same point, as the newly added valid viewpoint and stop traversal;

[0053] A camera squint model is established for the newly added valid viewpoints, the target mesh visual information is calculated and added to the valid viewpoint set.

[0054] Further, in step 10), the step of generating a flight path of the drone in the flight space based on the working viewpoint coordinates includes:

[0055] 10.1) Input the coordinates of the UAV flight space and working viewpoint;

[0056] 10.2) Obtain the maximum Z coordinate from the working viewpoint coordinates;

[0057] 10.3) According to the coordinates after ZY screen projection, the layer closest to the maximum Z coordinate is used as the transfer layer;

[0058] 10.4) Project all working viewpoints to the transport layer, and use the projected coordinates as working control points;

[0059] 10.5) Carry out work control point path planning;

[0060] 10.6) Plan the path according to the work control points. After the UAV flies to each work control point in turn, it vertically takes off and lands to each work viewpoint corresponding to the control point to collect images, and then returns and goes to the next work control point.

[0061] The technical effect of the present invention is undoubted, and the beneficial effects of the present invention are as follows:

[0062] 1) Through the viewpoint preprocessing method, the UAV can fly close to the target structure, improving the image pixel accuracy

[0063] 2) Through the viewpoint preprocessing method, the target grid and obstacle grid are distinguished, and static obstacle avoidance of the UAV is achieved.

[0064] 3) Through the viewpoint screening method, the use of type II grids ensures the reconstructibility between images, and the use of type I grids ensures the accuracy of 3D reconstruction

[0065] 4) The UAV flight path generated by the present invention can automatically perform close-up image acquisition without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 For structural diagram

[0067] Figure 2 Flowchart of the method

[0068] Figure 3 The low-precision point cloud described in the present invention

[0069] Figure 4 The target point cloud and obstacle point cloud after matching described in the present invention are

[0070] Figure 5 The target grid, semi-directed bounding box and expanded bounding box of the present invention are

[0071] Figure 6 The flight space of the present invention

[0072] Figure 7 The effective viewpoint set of the present invention

[0073] Figure 8 The working viewpoint of the present invention

[0074] Fig. 9 The flight path of the drone of the present invention

[0075] In the figure: 1. Target structure; 2. Target point cloud; 3. Obstacle point cloud; 4. Target grid; 5. Semi-directed bounding box; 6. Expanded bounding box; 7. Flight space; 8. Valid viewpoint set; 9. Working viewpoint; 10. Obstacle grid; 11. Working control point; 12. UAV flight path. DETAILED DESCRIPTION

[0076] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0077] Embodiment 1:

[0078] See also Figures 1 to 9 , a method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction, comprising the following steps:

[0079] 1) Using a drone to collect oblique photography images and / or multi-view images of the target structure, and reconstructing the collected images into low-precision image point clouds;

[0080] 2) Obtain a three-dimensional mesh model of the target structure and discretize it into a point cloud of the target structure;

[0081] 3) Match the target structure point cloud with the low-precision image point cloud, and record the coordinate transformation matrix M during the matching process. The matched target structure point cloud is used as the target point cloud, and the low-precision point cloud with a distance greater than dmin = 10 cm from the target point cloud is used as the obstacle point cloud;

[0082] 4) Based on the target point cloud and obstacle point cloud, construct the target grid and obstacle grid respectively;

[0083] 5) Obtain the flight space and image acquisition candidate point set;

[0084] 6) Establishing the camera main view model at each image acquisition candidate point;

[0085] 7) Create a light projection scene based on the target grid, obstacle grid and camera model, and calculate the visible information of the target grid under each camera model, including the index of each visible target grid and the distance between each visible target grid and the candidate point of the drone image acquisition;

[0086] 8) Based on the visible information of the target grid, the visible target grid under each camera model is divided into type I grid and type II grid, and the candidate points for drone image acquisition with the number of type I grids greater than N are selected as the valid viewpoint set;

[0087] 9) Obtain working viewpoints from the valid viewpoint set;

[0088] 10) Generate the UAV flight path in the flight space based on the working viewpoint coordinates;

[0089] 11) Based on the flight path of the UAV and the camera external parameters of the working viewpoint, calculate the flight parameters of the UAV at the working viewpoint and generate a flight file.

[0090] In step 4), the target point cloud is encapsulated into a target grid, or the three-dimensional grid of the target structure transformed by the coordinate transformation matrix M is encapsulated into a target grid.

[0091] In step 5), the steps of obtaining the flight space and the image acquisition candidate point set include:

[0092] 5.1) Create a semi-directed bounding box of the target mesh and expand the bounding box to the side and upwards. The expansion size is d = (P / λ) 2 pf eq / S 35, the side length of the expanded bounding box is taken as an integer multiple of the drone hovering accuracy; P is the expected point cloud position tolerance, λ∈[5,10] is the conversion factor between pixel accuracy and point cloud tolerance; p is pixel accuracy, f eq is the camera’s 35mm equivalent focal length, S 35 It is 35mm film size;

[0093] 5.2) According to the second rule, the expansion bounding box is divided into voxels with a spacing of k times the hovering accuracy of the drone, and the coordinates of the center point of each voxel in the voxel are calculated as the point set A; k is a natural number not less than 1;

[0094] 5.3) Divide point set A. If the voxel represented by a point in point set A contains the target point cloud, the point is divided into point set B. If the voxel contains the obstacle point cloud, the point is divided into point set C. If the voxel does not contain the target point cloud and the obstacle point cloud, the point is divided into point set D.

[0095] 5.4) According to the third rule, remove from point set D the points whose distance to the closest points in point set B and point set C is less than the safety distance d. safe points, and obtain the point set E;

[0096] 5.5) Remove the points directly below point set B and point set C from point set E to obtain point set F;

[0097] 5.6) Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the flight space of the UAV;

[0098] 5.7) Remove points whose distance to the nearest point of the target point cloud is less than d from the point set G to obtain the point set H;

[0099] 5.8) Take each point in the point set H as the starting point, point to the nearest point in the target point cloud, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition.

[0100] The height direction of the target grid semi-directed bounding box in step 5.1) is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY plane.

[0101] The safety distance d mentioned in step 5.4) safe , not less than the sum of the drone’s hovering accuracy, the estimated accuracy of the low-precision point cloud, and the drone’s minimum obstacle avoidance distance.

[0102] In step 6), when establishing the camera main view model at each image acquisition candidate point, the camera external parameter P c is a matrix of 4 rows and 4 columns; the camera external parameter P c The first 3 rows and 3 columns in the matrix are the orthogonal rotation matrix R = [η1 ,η 2 ,η 3 ], the last column of the first three rows is the offset column vector T, and the fourth row is [0,0,0,1];

[0103] η 1 ,η 2 ,η 3 are all unit row vectors, η 3 is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η 1 At the same time, perpendicular to the Z axis and η 3 , η 2 At the same time perpendicular to η 1 and η 3 ; The offset column vector T is obtained by multiplying R by the coordinates of the candidate image acquisition points and taking the negative value;

[0104] The camera intrinsic parameters are obtained through manufacturer instructions, calibration of calibration plates, or during low-precision point cloud reconstruction.

[0105] In step 8), the type I grid is a grid whose distance between the visible target grid and the candidate point of the drone image acquisition is within the threshold d; the type II grid is a grid whose distance between the visible target grid and the candidate point of the drone image acquisition is within the threshold d = (P / λ) 2 pf eq / S 35 Outside the grid.

[0106] In step 9), the step of obtaining the working viewpoint from the valid viewpoint set includes:

[0107] 9.1) Initialize the grid coverage to 0 and initialize the observed grid set to an empty set;

[0108] 9.2) Calculate the grid visibility table, which contains the grid index and the number of times each grid is observed by valid viewpoints as a type I grid and a type II grid. and And go to step 9.3); i is the grid number; if the number If it is less than 3, the grid is considered to be not visible enough. Before executing step 9.3), and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible;

[0109] 9.3) Calculate the weight of effective viewpoints j is the viewpoint number; a is the visibility identifier of the type I grid, which is 1 when the grid i is a type I grid and can be seen by the valid viewpoint j, otherwise it is 0; b is the visibility identifier of the type II grid, which is 1 when the grid i is a type II grid and can be seen by the valid viewpoint j, otherwise it is 0; γ is the weight correction parameter of the type II grid, which is a real number less than 1 and greater than 0;

[0110] 9.4) Take the viewpoint with the largest weight as the new working viewpoint, add it to the working viewpoint set, and obtain the visual grid of the new working viewpoint;

[0111] 9.5) Add the Class I grid index in the newly added working viewpoint visible grid to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0112] 9.6) Update effective viewpoint weights c is the newly added Class I grid visibility identifier. When grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, c is 1, otherwise c is 0;

[0113] 9.7) From the valid viewpoints that are not in the working viewpoint set, calculate the number of target grids that observe the same target grid as each working viewpoint in the working viewpoint set, as the number of common view grids;

[0114] Calculate the number of newly added type I grids compared to the observed grid set as the number of newly added type I grids;

[0115] If the ratio of the number of common view grids to the total number of valid viewpoint observation grids not in the working viewpoint set, and to the total number of corresponding working viewpoint observation grids, exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set; otherwise, the valid viewpoint and its nearest point in the target point cloud, the working viewpoint with the largest number of common view grids with the valid viewpoint, and its nearest point in the target point cloud are added to the oblique viewpoint information cache set;

[0116] 9.8) When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step 9.1), otherwise select the valid viewpoint with the largest weight from the newly added working viewpoint cache set as the newly added working viewpoint and add it to the working viewpoint set;

[0117] 9.9) Calculate the ratio of the number of newly added working viewpoint visible type I grids compared to the observed grid set to the total number of target grids as the new addition rate;

[0118] 9.10) Add the newly added working viewpoint visible Class I grid index to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0119] When the number of viewpoints in the working viewpoint set is less than N max , when the addition rate is less than 0.001 and the grid coverage rate is less than f, return to step 9.6);

[0120] 9.11) According to the visible Class I grids of each viewpoint in the working viewpoint set, calculate the number of observations of each target grid, and select the target grids with 1 and 2 grid observations as the grids to be supplemented;

[0121] 9.12) Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented from large to small, traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose observation number is not 0 according to the Class I grid index observed by the valid viewpoint;

[0122] 9.13) Repeat step 9.12) until the target grid observation times do not exist 1 time and 2 times, or there are no valid viewpoints that meet the conditions of the previous step, and output the working viewpoint set;

[0123] In step 9.8), the step of adding a valid oblique viewpoint to the valid viewpoint set includes:

[0124] Sort the valid viewpoints in the squint viewpoint information cache set from large to small according to the number of common view grids;

[0125] Traverse the sorted oblique viewpoint cache set, select the valid viewpoint whose nearest point in the target point cloud and the nearest point in the target point cloud of the working viewpoint are not the same point, as the newly added valid viewpoint and stop traversal;

[0126] A camera squint model is established for the newly added valid viewpoints, the target mesh visual information is calculated and added to the valid viewpoint set.

[0127] In step 10), the step of generating the flight path of the drone in the flight space based on the working viewpoint coordinates includes:

[0128] 10.1) Input the coordinates of the UAV flight space and working viewpoint;

[0129] 10.2) Obtain the maximum Z coordinate from the working viewpoint coordinates;

[0130] 10.3) According to the coordinates after ZY screen projection, the layer closest to the maximum Z coordinate is used as the transfer layer;

[0131] 10.4) Project all working viewpoints to the transport layer, and use the projected coordinates as working control points;

[0132] 10.5) Carry out work control point path planning;

[0133] 10.6) Plan the path according to the work control points. After the UAV flies to each work control point in turn, it vertically takes off and lands to each work viewpoint corresponding to the control point to collect images, and then returns and goes to the next work control point.

[0134] Embodiment 2:

[0135] A method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction includes the following steps:

[0136] 1) Using a drone to collect oblique photography images and / or multi-view images of the target structure, and reconstructing the collected images into low-precision image point clouds;

[0137] 2) Obtain a three-dimensional mesh model of the target structure and discretize it into a point cloud of the target structure;

[0138] 3) Match the target structure point cloud with the low-precision image point cloud, and record the coordinate transformation matrix M during the matching process. The matched target structure point cloud is used as the target point cloud, and the low-precision point cloud with a distance greater than dmin = 10 cm from the target point cloud is used as the obstacle point cloud;

[0139] 4) Based on the target point cloud and obstacle point cloud, construct the target grid and obstacle grid respectively;

[0140] 5) Obtain the flight space and image acquisition candidate point set;

[0141] 6) Establishing the camera main view model at each image acquisition candidate point;

[0142] 7) Create a light projection scene based on the target grid, obstacle grid and camera model, and calculate the visible information of the target grid under each camera model, including the index of each visible target grid and the distance between each visible target grid and the candidate point of the drone image acquisition;

[0143] 8) Based on the visible information of the target grid, the visible target grid under each camera model is divided into type I grid and type II grid, and the candidate points for drone image acquisition with the number of type I grids greater than N are selected as the valid viewpoint set;

[0144] 9) Obtain working viewpoints from the valid viewpoint set;

[0145] 10) Generate the UAV flight path in the flight space based on the working viewpoint coordinates;

[0146] 11) Based on the flight path of the UAV and the camera external parameters of the working viewpoint, calculate the flight parameters of the UAV at the working viewpoint and generate a flight file.

[0147] Embodiment 3:

[0148] A method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction, the technical content of which is the same as that of Example 2. Furthermore, in step 4), the target point cloud is encapsulated into a target grid, or the three-dimensional grid of the target structure transformed by the coordinate transformation matrix M is encapsulated into a target grid.

[0149] Embodiment 4:

[0150] A method for acquiring close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of Embodiments 2-3, and further, in step 5), the step of acquiring a flight space and a candidate point set for image acquisition includes:

[0151] 5.1) Create a semi-directed bounding box of the target mesh and expand the bounding box to the side and upwards. The expansion size is d = (P / λ) 2 pf eq / S 35 , the side length of the expanded bounding box is taken as an integer multiple of the drone hovering accuracy; P is the expected point cloud position tolerance, λ∈[5,10] is the conversion factor between pixel accuracy and point cloud tolerance; p is pixel accuracy, f eq is the camera’s 35mm equivalent focal length, S 35 It is 35mm film size;

[0152] 5.2) According to the second rule, the expansion bounding box is divided into voxels with a spacing of k times the hovering accuracy of the drone, and the coordinates of the center point of each voxel in the voxel are calculated as the point set A; k is a natural number not less than 1;

[0153] 5.3) Divide point set A. If the voxel represented by a point in point set A contains the target point cloud, the point is divided into point set B. If the voxel contains the obstacle point cloud, the point is divided into point set C. If the voxel does not contain the target point cloud and the obstacle point cloud, the point is divided into point set D.

[0154] 5.4) According to the third rule, remove from point set D the points whose distance to the closest points in point set B and point set C is less than the safety distance d. safe points, and obtain the point set E;

[0155] 5.5) Remove the points directly below point set B and point set C from point set E to obtain point set F;

[0156] 5.6) Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the flight space of the UAV;

[0157] 5.7) Remove points whose distance to the nearest point of the target point cloud is less than d from the point set G to obtain the point set H;

[0158] 5.8) Take each point in the point set H as the starting point, point to the nearest point in the target point cloud, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition.

[0159] Embodiment 5:

[0160] A method for close-up image acquisition of an unmanned aerial vehicle based on incremental reconstruction, the technical content of which is the same as any one of Examples 2-4, and further, the height direction of the target grid semi-directed bounding box in step 5.1) is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY plane.

[0161] Embodiment 6:

[0162] A method for collecting close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of Embodiments 2-5, and further, the safety distance d in step 5.4) safe , not less than the sum of the drone’s hovering accuracy, the estimated accuracy of the low-precision point cloud, and the drone’s minimum obstacle avoidance distance.

[0163] Embodiment 7:

[0164] A method for acquiring close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of Embodiments 2-6, and further, in step 6), when establishing a camera main view model at each image acquisition candidate point, the camera external reference P c is a matrix of 4 rows and 4 columns; the camera external parameter P c The first 3 rows and 3 columns in the matrix are the orthogonal rotation matrix R = [η 1 ,η 2 ,η 3 ], the last column of the first three rows is the offset column vector T, and the fourth row is [0,0,0,1];

[0165] η 1 ,η 2 ,η 3 are all unit row vectors, η 3 is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η 1 At the same time, perpendicular to the Z axis and η 3 , η 2 At the same time perpendicular to η 1 and η 3 ; The offset column vector T is obtained by multiplying R by the coordinates of the candidate image acquisition points and taking the negative value;

[0166] The camera intrinsic parameters are obtained through manufacturer instructions, calibration of calibration plates, or during low-precision point cloud reconstruction.

[0167] Embodiment 8:

[0168] A method for close-up image acquisition of a drone based on incremental reconstruction, the technical content of which is the same as any one of Examples 2-7, and further, in step 8), the Class I grid is a grid whose distance from the visible target grid to the candidate point for drone image acquisition is within a threshold value d; the Class II grid is a grid whose distance from the visible target grid to the candidate point for drone image acquisition is outside the threshold value d.

[0169] Embodiment 9:

[0170] A method for acquiring close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of Embodiments 2-8, and further, in step 9), the step of acquiring a working viewpoint from a valid viewpoint set comprises:

[0171] 9.1) Initialize the grid coverage to 0 and initialize the observed grid set to an empty set;

[0172] 9.2) Calculate the grid visibility table, which contains the grid index and the number of times each grid is observed by valid viewpoints as a type I grid and a type II grid. and And go to step 9.3); i is the grid number; if the number If it is less than 3, the grid is considered to be not visible enough. Before executing step 9.3), and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible;

[0173] 9.3) Calculate the weight of effective viewpoints j is the viewpoint number; a is the visibility identifier of the type I grid, which is 1 when the grid i is a type I grid and can be seen by the valid viewpoint j, otherwise it is 0; b is the visibility identifier of the type II grid, which is 1 when the grid i is a type II grid and can be seen by the valid viewpoint j, otherwise it is 0; γ is the weight correction parameter of the type II grid, which is a real number less than 1 and greater than 0;

[0174] 9.4) Take the viewpoint with the largest weight as the new working viewpoint, add it to the working viewpoint set, and obtain the visual grid of the new working viewpoint;

[0175] 9.5) Add the Class I grid index in the newly added working viewpoint visible grid to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0176] 9.6) Update effective viewpoint weights c is the newly added Class I grid visibility identifier. When grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, c is 1, otherwise c is 0;

[0177] 9.7) From the valid viewpoints that are not in the working viewpoint set, calculate the number of target grids that observe the same target grid as each working viewpoint in the working viewpoint set, as the number of common view grids;

[0178] Calculate the number of newly added type I grids compared to the observed grid set as the number of newly added type I grids;

[0179] If the ratio of the number of common view grids to the total number of valid viewpoint observation grids not in the working viewpoint set, and to the total number of corresponding working viewpoint observation grids, exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set; otherwise, the valid viewpoint and its nearest point in the target point cloud, the working viewpoint with the largest number of common view grids with the valid viewpoint, and its nearest point in the target point cloud are added to the oblique viewpoint information cache set;

[0180] 9.8) When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step 9.1), otherwise select the valid viewpoint with the largest weight from the newly added working viewpoint cache set as the newly added working viewpoint and add it to the working viewpoint set;

[0181] 9.9) Calculate the ratio of the number of newly added working viewpoint visible type I grids compared to the observed grid set to the total number of target grids as the new addition rate;

[0182] 9.10) Add the newly added working viewpoint visible Class I grid index to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids;

[0183] When the number of viewpoints in the working viewpoint set is less than N max , when the addition rate is less than 0.001 and the grid coverage rate is less than f, return to step 9.6);

[0184] 9.11) According to the visible Class I grids of each viewpoint in the working viewpoint set, calculate the number of observations of each target grid, and select the target grids with 1 and 2 grid observations as the grids to be supplemented;

[0185] 9.12) Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented from large to small, traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose observation number is not 0 according to the Class I grid index observed by the valid viewpoint;

[0186] 9.13) Repeat step 9.12) until the target grid observation times do not exist 1 time and 2 times, or there are no valid viewpoints that meet the conditions of the previous step, and output the working viewpoint set; Embodiment 10:

[0187] A method for collecting close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of Embodiments 2-9, and further, in step 9.8), the step of adding a valid oblique viewpoint to the valid viewpoint set comprises:

[0188] Sort the valid viewpoints in the squint viewpoint information cache set from large to small according to the number of common view grids;

[0189] Traverse the sorted oblique viewpoint cache set, select the valid viewpoint whose nearest point in the target point cloud and the nearest point in the target point cloud of the working viewpoint are not the same point, as the newly added valid viewpoint and stop traversal;

[0190] A camera squint model is established for the newly added valid viewpoints, the target mesh visual information is calculated and added to the valid viewpoint set.

[0191] Embodiment 11:

[0192] A method for collecting close-up images of a drone based on incremental reconstruction, the technical content of which is the same as any one of embodiments 2-10, further, in step 10), the step of generating a flight path of the drone in the flight space based on the working viewpoint coordinates includes:

[0193] 10.1) Input the coordinates of the UAV flight space and working viewpoint;

[0194] 10.2) Obtain the maximum Z coordinate from the working viewpoint coordinates;

[0195] 10.3) According to the coordinates after ZY screen projection, the layer closest to the maximum Z coordinate is used as the transfer layer;

[0196] 10.4) Project all working viewpoints to the transport layer, and use the projected coordinates as working control points;

[0197] 10.5) Carry out work control point path planning;

[0198] 10.6) Plan the path according to the work control points. After the UAV flies to each work control point in turn, it vertically takes off and lands to each work viewpoint corresponding to the control point to collect images, and then returns and goes to the next work control point.

[0199] Embodiment 12:

[0200] A method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction, the steps are as follows:

[0201] 1) Use drones to perform oblique photography or multi-view image acquisition of the target structure and reconstruct it into a low-precision point cloud;

[0202] 2) Discretize the target structure 3D grid model into a point cloud, match it with the low-precision point cloud and record the coordinate transformation matrix M during the matching process. The matched target structure point cloud is used as the target point cloud, and the low-precision point cloud with a distance greater than 10 cm from the target point cloud is used as the obstacle point cloud;

[0203] 3) Encapsulating the target point cloud into a target grid, or using the target structure three-dimensional grid transformed by the coordinate transformation matrix M as the target grid, and encapsulating the obstacle point cloud into an obstacle grid;

[0204] 4) Obtain the flight space and image acquisition candidate point set;

[0205] 5) Establishing the camera main view model at each image acquisition candidate point;

[0206] 6) Create a light projection scene based on the target grid, obstacle grid and camera model, and calculate the visible information of the target grid under each camera model, including the index of each visible target grid and the distance between each visible target grid and the candidate point of the drone image acquisition;

[0207] 7) According to the visible information of the target grid, the visible target grid under each camera model is divided into Class I grid and Class II grid, and the candidate points for drone image acquisition with the number of Class I grids greater than N are selected as the valid viewpoint set;

[0208] 8) Obtain working viewpoints from the valid viewpoint set;

[0209] 9) Generate the UAV flight path in the flight space based on the working viewpoint coordinates;

[0210] 10) According to the flight path of the UAV and the camera external parameters of the working viewpoint, calculate the flight parameters of the UAV at the working viewpoint, and generate the flight file according to the selected UAV flight file format.

[0211] Obtain the flight space and image acquisition candidate point set. The specific steps include:

[0212] 4.1) According to the first rule, a semi-directed bounding box of the target grid is established, and the bounding box is expanded to the side and upward by d. After the expansion, the side length of the bounding box is taken as an integer multiple of the drone's hovering accuracy;

[0213] 4.2) According to the second rule, the expansion bounding box is divided into voxels with a spacing of k times the hovering accuracy of the drone, and the coordinates of the center point of each voxel in the voxel are calculated as the point set A;

[0214] 4.3) Divide the point set A into point set B when the voxels represented by the points in point set A contain the target point cloud, divide them into point set C when they contain the obstacle point cloud, and divide them into point set D when they do not contain the point cloud;

[0215] 4.4) According to the third rule, remove from point set D the points whose distance to the closest points in point set B and point set C is less than the safety distance d. safe points, and obtain the point set E;

[0216] 4.5) Remove the points directly below point set B and point set C from point set E to obtain point set F;

[0217] 4.6) Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the flight space of the UAV;

[0218] 4.7) Remove points whose distance to the nearest point of the target point cloud is less than d from the point set G to obtain the point set H;

[0219] 4.8) Take each point in the point set H as the starting point, point to its nearest point in the target point cloud, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition;

[0220] The first rule includes: the height direction of the semi-directed bounding box is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY plane; the viewpoint preprocessing method, expansion d = (P / λ) 2 pf eq / S 35 In the formula, P is the expected point cloud position tolerance, λ is the conversion factor between pixel accuracy and point cloud tolerance, which takes values ​​in the interval [5,10], p is pixel accuracy, and f eq is the camera’s 35mm equivalent focal length, S 35 For 35mm film size, take 36mmx24mm

[0221] The second rule includes: the k-fold drone hovering accuracy, where k is a natural number not less than 1;

[0222] The third rule includes: the safety distance d safe , not less than the sum of the drone hovering accuracy, the estimated accuracy of the low-precision point cloud described in step 1, and the minimum obstacle avoidance distance of the drone

[0223] The camera main view model is established at each image acquisition candidate point, and the camera external parameter P c A matrix with 4 rows and 4 columns: P c The first 3 rows and columns are the orthogonal rotation matrix R = [η 1 ,η 2 ,η 3 ],η 1 ,η 2 ,η 3 are all unit row vectors, η 3is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η 1 At the same time, perpendicular to the Z axis and η 3 , η 2 At the same time perpendicular to η 1 and η 3 ;P c The last column T of the first three rows is the offset column vector, which is obtained by multiplying R by the coordinates of the candidate points of image acquisition and taking the negative value; P c The fourth line is [0,0,0,1]; the camera intrinsic parameters are obtained through manufacturer instructions, calibration plate calibration or when reconstructing low-precision point clouds.

[0224] According to the visual information of the target grid, the visible target grids under each camera model are divided into Class I grids and Class II grids, and the candidate points for drone image acquisition with the number of Class I grids greater than N are selected as the valid viewpoint set. The Class I grids refer to the grids whose distances from the visible target grids to the candidate points for drone image acquisition are within the threshold d; the Class II grids refer to the grids whose distances from the visible target grids to the candidate points for drone image acquisition are outside the threshold d.

[0225] Acquire the working viewpoint from the valid viewpoint set. The specific steps include:

[0226] a) Initialize the grid coverage to 0 and initialize the observed grid set to an empty set

[0227] b) Calculate the grid visibility table, which contains the grid index and the number of times each grid is observed by a valid viewpoint as a type I grid and a type II grid. and i is the grid number, when When it is less than 3, the grid is considered to be not visible enough. and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible.

[0228] c) Calculate the weight of effective viewpoints Where j is the viewpoint number; a is the visibility identifier of the type I grid, which is 1 when the grid i is visible to the valid viewpoint j as a type I grid, otherwise it is 0; b is the visibility identifier of the type II grid, which is 1 when the grid i is visible to the valid viewpoint j as a type II grid, otherwise it is 0; γ is the weight correction parameter of the type II grid, which is a real number less than 1 and greater than 0.

[0229] d) Take the viewpoint with the largest weight as the new working viewpoint, add it to the working viewpoint set, and obtain the visual grid of the new working viewpoint.

[0230] e) Add the Class I grid index in the visible grid of the newly added working viewpoint to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids.

[0231] f) Update effective viewpoint weights c is a newly added Class I grid visibility identifier. When grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, c is 1, otherwise c is 0.

[0232] g) Sort the valid viewpoints by their weights from large to small, and calculate the number of target grids that are observed by the currently traversed valid viewpoint and each working viewpoint in the working viewpoint set in turn, as the number of common view grids; calculate the number of Class I grids that can be added compared to the observed grid set, as the number of newly added Class I grids. When the ratio of the number of common view grids to the total number of grids observed by the currently traversed valid viewpoints and to the total number of grids observed by the corresponding working viewpoints exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set and the traversal is stopped. Otherwise, the valid viewpoint and its nearest point in the target point cloud, and the working viewpoint with the largest number of common view grids with the valid viewpoint and its nearest point in the target point cloud are added to the oblique viewpoint information cache set.

[0233] h) When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step a), otherwise the valid viewpoint in the newly added working viewpoint cache set is added to the working viewpoint set as a new working viewpoint.

[0234] i) Calculate the ratio of the number of newly added working viewpoint visible Class I grids compared to the observed grid set to the total number of target grids as the new addition rate.

[0235] j) Add the newly added working viewpoint visible Class I grid index to the observed grid set, and update the grid coverage according to the number of grid indexes in the observed network set and the total number of target grids.

[0236] k) When the number of viewpoints in the working viewpoint set is less than N max , when the addition rate is less than 0.001 and the grid coverage rate is less than f, repeat steps f) to j).

[0237] l) According to the visible Class I grids at each viewpoint in the working viewpoint set, calculate the number of observations of each target grid, and select the target grids with 1 and 2 grid observations as the grids to be supplemented.

[0238] m) Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented, and traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, then stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose observation times are not 0 according to the Class I grid index observed by the valid viewpoint

[0239] n) Repeat step m) until the target grid observation times do not exist 1 time and 2 times, or there are no valid viewpoints that meet the conditions of step m), and output the working viewpoint set.

[0240] The fourth rule includes: sorting the valid viewpoints in the squint viewpoint information cache set from large to small according to the number of common view grids; traversing the sorted squint viewpoint cache set, selecting the valid viewpoint whose closest point in the target point cloud and the closest point in the target point cloud of the working viewpoint are not the same as the new valid viewpoint and stopping the traversal; establishing a camera squint model for the new valid viewpoint, calculating the target grid visual information and adding it to the valid viewpoint set. The camera squint model is similar to the camera main view model, except that η 3 The unit direction vector pointing to the nearest point of the working viewpoint in the target point cloud for the newly added valid viewpoint.

[0241] Generate the UAV flight path in the flight space based on the working viewpoint coordinates. The specific steps include:

[0242] Input the coordinates of the drone’s flight space and working viewpoint;

[0243] Get the maximum Z coordinate from the working viewpoint coordinates;

[0244] According to the ZY screen projection coordinates, cut the layer closest to the maximum Z coordinate as the transfer layer;

[0245] Project all working viewpoints to the transport layer, and use the projected coordinates as working control points;

[0246] Carry out work control point path planning;

[0247] The path is planned according to the work control points. After the UAV flies to each work control point in turn, it takes off and lands vertically to each work viewpoint corresponding to the control point to collect images, and then returns to the next work control point.

[0248] Embodiment 13:

[0249] A method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction, comprising the following steps:

[0250] 101: Use drones to collect multi-view images of the structure and reconstruct them into low-precision point clouds;

[0251] 102: discretize the three-dimensional grid model of the target structure 1 in the structure into a point cloud, match it with the low-precision point cloud and record the coordinate transformation matrix M in the matching process, the matched target structure point cloud is used as the target point cloud 2, and the low-precision point cloud with a distance greater than 10 cm from the target point cloud is used as the obstacle point cloud 3;

[0252] 103: encapsulate the target point cloud into a target grid 4, and encapsulate the obstacle point cloud 3 into an obstacle grid 10;

[0253] 104: Acquire the flight space 7 and the image acquisition candidate point set;

[0254] 105: Establishing a camera main view model at each image acquisition candidate point;

[0255] 106: Create a light projection scene according to the target mesh, obstacle mesh and camera model, and calculate the visible information of the target mesh under each camera model;

[0256] 107: According to the visible information of the target grid, the grids whose distances between the visible target grid and the candidate points for UAV image acquisition are within the threshold γ are classified as Class I grids, and the grids whose distances between the visible target grid and the candidate points for UAV image acquisition are outside the threshold γ are classified as Class II grids, and the candidate points for UAV image acquisition whose number of Class I grids is greater than N are selected as the valid viewpoint set 8;

[0257] 108: Get working viewpoint 9;

[0258] 109: Generate UAV flight path;

[0259] 110: Calculate the flight parameters of the UAV at the working viewpoint according to the flight path of the UAV and the camera external parameters of the working viewpoint, and generate the flight file according to the selected UAV flight file format.

[0260] In the present application, the viewpoint preprocessing method is used to enable the UAV to fly close to the target structure, thereby improving the image pixel accuracy; the viewpoint preprocessing method is used to distinguish between the target grid and the obstacle grid, thereby realizing the static obstacle avoidance of the UAV; the viewpoint screening method is used to ensure the reconstructability between images by using Class II grids, and to ensure the three-dimensional reconstruction accuracy by using Class I grids; the UAV flight path generated by the present invention can automatically perform close image acquisition work without human intervention.

[0261] In this embodiment, the structure is a steel truss, and the target structure 1 is a top-level member of the truss. In other embodiments, the structure and the target structure are determined according to task requirements.

[0262] In this embodiment, the elevation angle of the drone is limited to 30°, the safe obstacle avoidance distance is 0.5m, and the hovering accuracy is 0.2m. In other embodiments, the technical parameters of the drone vary according to the model;

[0263] In this embodiment, multi-view image acquisition of the structure is performed manually, and in other embodiments, oblique photography may be used to automatically acquire images of the structure;

[0264] Step 104: adopting a candidate point generation method to obtain a flight space and an image acquisition candidate point set, and obtaining a flight space and an image acquisition candidate point set, specifically including steps 1041 to 1048:

[0265] Step 1041 : Create a semi-directed bounding box 5 and an expanded bounding box 6 of the target mesh 4 .

[0266] Specifically, the height direction of the semi-directed bounding box 5 is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY screen. The semi-directed bounding box is expanded sideways and upward by d to form an expanded bounding box 6.

[0267] It should be noted that d is given by the formula d=(P / λ) 2 pf eq / S 35 Determine, where P is the desired point cloud position tolerance, in this embodiment, it is 6 mm; λ can be verified by the embodiment, usually 5 to 10; f eq S is the 35mm equivalent focal length of the camera on the drone; 35 The film size is 35mm, and in this embodiment, it is 36mmx24mm.

[0268] It should be noted that the side length of the expanded bounding box is an integer multiple of the drone's hovering accuracy.

[0269] Step 1042: Divide the expanded bounding box into voxels with a spacing of k times the hovering accuracy of the drone, and calculate the coordinates of the center points of the voxels as point set A.

[0270] In this embodiment, the value of k is 2. In other embodiments, the value of k may be an integer not less than 2.

[0271] Step 1043: Divide point set A into point set B, point set C and point set D.

[0272] Specifically, when the voxels represented by the points in point set A contain target point clouds, they are divided into point set B; when they contain obstacle point clouds, they are divided into point set C; when they do not contain point clouds, they are divided into point set D.

[0273] Step 1044: Remove from point set D the points whose distances to the closest points in point set B and point set C are less than the safety distance d. safepoints, and obtain the point set E.

[0274] Specifically, the safety distance d safe , not less than the sum of the drone's hovering accuracy, the estimated accuracy of the low-precision point cloud described in step 1, and the drone's minimum obstacle avoidance distance.

[0275] Step 1045: Remove the points directly below point set B and point set C from point set E to obtain point set F.

[0276] Step 1046: Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the UAV flight space 7.

[0277] Step 1047: Remove from the point set G the points whose distance to the target point cloud is less than the pixel accuracy threshold d. pre points, and obtain the point set H.

[0278] Step 1048: Take each point in the point set H as the starting point, point to its nearest point in the target point cloud 2, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition.

[0279] It should be noted that if the drone wings appear in the field of view when the line of sight elevation angle is close to the maximum elevation angle of the drone gimbal, the maximum elevation angle of the drone gimbal should be reduced.

[0280] In step 105: in establishing the camera primary view model at each image acquisition candidate point, the camera external parameter P c is a matrix of 4 rows and 4 columns, P c The first 3 rows and 3 columns in the matrix are the orthogonal rotation matrix R = [η 1 ,η 2 ,η 3 ],η1,η 2 ,η 3 are all unit row vectors, η 3 is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η 1 At the same time, perpendicular to the Z axis and η 3 , η 2 At the same time perpendicular to η 1 and η 3 ;P c The last column T of the first three rows is the offset column vector, which is obtained by multiplying R by the coordinates of the candidate points of image acquisition and taking the negative value; P c The fourth line is [0,0,0,1]; the camera intrinsic parameters are obtained during low-precision point cloud reconstruction;

[0281] In this embodiment, the camera intrinsic parameters are obtained during low-precision point cloud reconstruction. In other embodiments, the camera intrinsic parameters can also be obtained through manufacturer instructions or calibration with a calibration plate.

[0282] Among them, in step 106: according to the target grid 4, the obstacle grid 10 and the camera model, a light projection scene is created, and the visual information of the target grid 4 under each camera model is calculated. The visual information includes the visible target grid index and the distance between each visible target grid and the candidate point of the drone image acquisition.

[0283] In this embodiment, the Open3d open source library is used to create the light projection scene. In other embodiments, other methods may also be used to create the light projection scene.

[0284] On the basis of the above embodiment, in this embodiment, the working viewpoint 9 is obtained, and the specific steps include step 10801 to step 10814:

[0285] Step 10801: Initialize the grid coverage to 0 and initialize the observed grid set to an empty set.

[0286] Step 10802: Calculate the grid visibility table.

[0287] Specifically, the visible count table contains the grid index and the number of times each grid is observed by a valid viewpoint as a type I grid and a type II grid. and i is the grid number, when When it is less than 3, the grid is considered to be not visible enough. and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible.

[0288] Step 10803: Calculate the weight W of the effective viewpoint j .

[0289] Specifically, according to the formula:

[0290] Determine, where j is the viewpoint number; a is the Class I mesh visibility identifier, which is 1 when mesh i is visible to valid viewpoint j as a Class I mesh, otherwise it is 0; b is the Class II mesh visibility identifier, which is 1 when mesh i is visible to valid viewpoint j as a Class II mesh, otherwise it is 0; γ is the Class II mesh weight correction parameter, which is a real number less than 1 and greater than 0.

[0291] Step 10804: Take the viewpoint with the largest weight as the newly added working viewpoint, add it to the working viewpoint 9 set, and obtain the visual grid of the newly added working viewpoint.

[0292] Step 10805: add the Class I grid index in the newly added working viewpoint visible grid to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids.

[0293] Step 10806: Update the effective viewpoint weight W j .

[0294] Specifically, according to the formula:

[0295] Determine, where c is the newly added Class I grid visibility identifier, and c is 1 when grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, otherwise c is 0.

[0296] Step 10807: Add valid viewpoints that meet the overlap rate requirements to the newly added working viewpoint cache set, and add valid viewpoints that do not meet the overlap rate requirements and their nearest points in the target point cloud to the oblique viewpoint information cache set.

[0297] Specifically, the valid viewpoints are sorted from large to small according to the weight of the valid viewpoints, and the number of the same target grids observed by the current traversed valid viewpoint and each working viewpoint in the working viewpoint set is calculated in turn as the number of common view grids; the number of Class I grids that can be added compared to the observed grid set is calculated as the number of newly added Class I grids. When the ratio of the number of common view grids to the total number of grids observed by the currently traversed valid viewpoints and to the total number of grids observed by the corresponding working viewpoints exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set and the traversal is stopped. Otherwise, the valid viewpoint and its nearest point in the target point cloud, and the working viewpoint with the largest number of common view grids with the valid viewpoint and its nearest point in the target point cloud are added to the oblique viewpoint information cache set.

[0298] It should be noted that, in this embodiment, r is 0.6, and in other embodiments, r may also be a value less than 1.

[0299] Step 10808: When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step 10801, otherwise select the valid viewpoint with the largest weight from the newly added working viewpoint cache set as the new working viewpoint and add it to the working viewpoint set.

[0300] Specifically, the effective squint viewpoint is determined according to steps 108081 to 108083:

[0301] Step 108081: Sort the valid viewpoints in the squint viewpoint information cache set by the number of common view grids from large to small

[0302] Step 108082: Traverse the sorted oblique viewpoint cache set, select the valid viewpoint whose closest point in the target point cloud and the closest point in the target point cloud of the working viewpoint are not the same point, as the newly added valid viewpoint and stop traversal

[0303] Step 108083: Establish a camera oblique model for the newly added valid viewpoint, calculate the target grid visual information and add it to the valid viewpoint set. The camera oblique model is similar to the camera main view model described in step 105, except that η 3 The unit direction vector of the nearest point of the newly added effective viewpoint to the effective viewpoint described in step 10807 in the target point cloud.

[0304] Step 10809: Calculate the ratio of the number of newly added working viewpoint visible Class I grids compared to the observed grid set to the total number of target grids as the new addition rate.

[0305] Step 10810 adds the newly added working viewpoint visible Class I grid index to the observed grid set, and updates the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids.

[0306] Step 10811: When the number of viewpoints in the working viewpoint set is less than N max , when the new addition rate is less than the threshold and the grid coverage rate is less than f, repeat steps 10806 to 10810.

[0307] In this embodiment, N max The value is 500, the new rate threshold value is 0.001, and the value of f is 0.8. In other embodiments, N max The threshold value of the new addition rate can be determined according to the computing power of the computer, and f is greater than 0 and not greater than 1.

[0308] Step 10812: Calculate the number of observations of each target grid based on the visible Class I grids at each viewpoint in the working viewpoint set, and select target grids with 1 and 2 grid observations as grids to be supplemented.

[0309] Step 10813: Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented, and traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, then stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose number of observations is not 0 according to the Class I grid index observed by the valid viewpoint.

[0310] Step 10814: Repeat step 10813 until the target grid observation times do not exist 1 time and 2 times, or there are no valid viewpoints that meet the conditions of step 10813, and output 9 sets of working viewpoints.

[0311] On the basis of the above embodiment, in this embodiment, the UAV flight path 12 is generated, and the specific steps include step 1091 to step 1096:

[0312] Step 1091: Input the coordinates of the drone flight space 7 and the working viewpoint 9

[0313] Step 1092: Get the maximum Z coordinate from the work viewpoint 9 coordinates

[0314] Step 1093: Layer the flight space according to the Z coordinate value, select a layer that is not less than the maximum Z coordinate and contains the coordinates of all voxels after projection onto the XY screen, and cut the layer closest to the maximum Z coordinate as the transfer layer.

[0315] Step 1094: Project all working viewpoints to the transport layer, and use the projected coordinates as working control points 11;

[0316] Step 1095: Perform path planning for the work control point 11;

[0317] In this embodiment, an adaptive large neighborhood search algorithm (ALNS) is used to plan the working control point path. In other embodiments, other traveling salesman problem algorithms may also be used to plan the control point path.

[0318] Step 1096: Plan a path according to the working control point 11. After the UAV flies to each working control point in turn, it vertically takes off and lands to each working viewpoint corresponding to the control point to collect images, then returns and goes to the next working control point until all working viewpoints are covered. The path flown by the UAV during the whole process is the UAV flight path 12.

[0319] On the basis of the above embodiment, in this embodiment, the flight parameters include the longitude, latitude, altitude, heading angle and pitch angle of the UAV at the working viewpoint.

Claims

1. A method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction, characterized in that: The following steps are involved: 1) Using drones to collect oblique photographic images and / or multi-view images of target structures, and reconstructing the collected images into low-precision image point clouds. 2) Obtain a three-dimensional mesh model of the target structure and discretize it into a point cloud of the target structure; 3) Match the target structure point cloud with the low-precision image point cloud, and record the coordinate transformation matrix M during the matching process. The matched target structure point cloud is used as the target point cloud, and the low-precision point cloud with a distance greater than dmin from the target point cloud is used as the obstacle point cloud; 4) Based on the target point cloud and obstacle point cloud, construct the target grid and obstacle grid respectively; 5) Obtain the flight space and image acquisition candidate point set; 6) Establishing the camera main view model at each image acquisition candidate point; 7) Create a light projection scene based on the target grid, obstacle grid and camera model, and calculate the visible information of the target grid under each camera model, including the index of each visible target grid and the distance between each visible target grid and the candidate point of the drone image acquisition; 8) Based on the visible information of the target grid, the visible target grid under each camera model is divided into type I grid and type II grid, and the candidate points for drone image acquisition with the number of type I grids greater than N are selected as the valid viewpoint set; 9) Obtain working viewpoints from the valid viewpoint set; 10) Generate the UAV flight path in the flight space based on the working viewpoint coordinates; 11) Based on the flight path of the UAV and the camera external parameters of the working viewpoint, calculate the flight parameters of the UAV at the working viewpoint and generate a flight file.

2. The method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 4), the target point cloud is encapsulated into a target grid, or the three-dimensional grid of the target structure transformed by the coordinate transformation matrix M is encapsulated into a target grid.

3. The method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 5), the steps of obtaining the flight space and the image acquisition candidate point set include: 5.1) Create a semi-directed bounding box of the target mesh and expand the bounding box to the side and upwards. The expansion size is d = (P / λ) 2 pf eq / S 35 , the side length of the expanded bounding box is taken as an integer multiple of the drone hovering accuracy; P is the expected point cloud position tolerance, λ∈[5,10] is the conversion factor between pixel accuracy and point cloud tolerance; p is pixel accuracy, f eq is the camera’s 35mm equivalent focal length, S 35 It is 35mm film size; 5.2) According to the second rule, the expansion bounding box is divided into voxels with a spacing of k times the hovering accuracy of the drone, and the coordinates of the center point of each voxel in the voxel are calculated as the point set A; k is a natural number not less than 1; 5.3) Divide point set A. If the voxel represented by a point in point set A contains the target point cloud, the point is divided into point set B. If the voxel contains the obstacle point cloud, the point is divided into point set C. If the voxel does not contain the target point cloud and the obstacle point cloud, the point is divided into point set D. 5.4) According to the third rule, remove from point set D the points whose distance to the closest points in point set B and point set C is less than the safety distance d. safe points, and obtain the point set E; 5.5) Remove the points directly below point set B and point set C from point set E to obtain point set F; 5.6) Select the point cluster with the largest number from the point set F as the point set G. The voxel space represented by the point set G is the flight space of the UAV; 5.7) Remove points whose distance to the nearest point of the target point cloud is less than d from the point set G to obtain the point set H; 5.8) Take each point in the point set H as the starting point, point to the nearest point in the target point cloud, construct the line of sight and calculate the line of sight elevation angle, remove the points whose line of sight elevation angle exceeds the maximum elevation angle of the drone gimbal from the point set H, and obtain the candidate point set for drone image acquisition.

4. The method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 3 is characterized in that: The height direction of the target grid semi-directed bounding box in step 5.1) is aligned with the positive direction of the Z axis, and the sides of the bounding box are perpendicular to the XY plane.

5. The method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 3 is characterized in that: The safety distance d mentioned in step 5.4) safe , not less than the sum of the drone’s hovering accuracy, the estimated accuracy of the low-precision point cloud, and the drone’s minimum obstacle avoidance distance.

6. The method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 6), when establishing the camera main view model at each image acquisition candidate point, the camera external parameter P c is a matrix of 4 rows and 4 columns; the camera external parameter P c The first three rows and the first three columns are the orthogonal rotation matrix R = [η1, η2, η3], the last column of the first three rows is the offset column vector T, and the fourth row is [0, 0, 0, 1]; η1, η2, η3 are all unit row vectors, η3 is the unit direction vector from the point in the image acquisition candidate point set to its nearest point in the target point cloud, η1 is perpendicular to both the Z axis and η3, and η2 is perpendicular to both η1 and η3; the offset column vector T is obtained by taking the negative value of the dot multiplication of R and the coordinates of the image acquisition candidate point; The camera intrinsic parameters are obtained through manufacturer instructions, calibration of calibration plates, or during low-precision point cloud reconstruction.

7. The method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 8), the type I grid is a grid whose distance from the visible target grid to the candidate point for drone image acquisition is within a threshold value d; the type II grid is a grid whose distance from the visible target grid to the candidate point for drone image acquisition is outside the threshold value d.

8. The method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 9), the step of obtaining the working viewpoint from the valid viewpoint set includes: 9.1) Initialize the grid coverage to 0 and initialize the observed grid set to an empty set; 9.2) Calculate the grid visibility table, which contains the grid index and the number of times each grid is observed by valid viewpoints as a type I grid and a type II grid. and And go to step 9.3); i is the grid number; if the number If it is less than 3, the grid is considered to be not visible enough. Before executing step 9.3), and Assign a value of 0 and modify the visual information of the grid in all valid viewpoints to be invisible; 9.3) Calculate the weight of effective viewpoints j is the viewpoint number; a is the visibility identifier of the type I grid, which is 1 when the grid i is a type I grid and can be seen by the valid viewpoint j, otherwise it is 0; b is the visibility identifier of the type II grid, which is 1 when the grid i is a type II grid and can be seen by the valid viewpoint j, otherwise it is 0; γ is the weight correction parameter of the type II grid, which is a real number less than 1 and greater than 0; 9.4) Take the viewpoint with the largest weight as the new working viewpoint, add it to the working viewpoint set, and obtain the visual grid of the new working viewpoint; 9.5) Add the Class I grid index in the newly added working viewpoint visible grid to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids; 9.6) Update effective viewpoint weights c is the newly added Class I grid visibility identifier. When grid i is a Class I grid that can be seen by valid viewpoint j and is not in the observed grid set, c is 1, otherwise c is 0; 9.7) From the valid viewpoints that are not in the working viewpoint set, calculate the number of target grids that observe the same target grid as each working viewpoint in the working viewpoint set, as the number of common view grids; Calculate the number of newly added type I grids compared to the observed grid set as the number of newly added type I grids; If the ratio of the number of common view grids to the total number of valid viewpoint observation grids not in the working viewpoint set, and to the total number of corresponding working viewpoint observation grids, exceeds the overlap rate r, the valid viewpoint is added to the newly added working viewpoint cache set; otherwise, the valid viewpoint and its nearest point in the target point cloud, the working viewpoint with the largest number of common view grids with the valid viewpoint, and its nearest point in the target point cloud are added to the oblique viewpoint information cache set; 9.8) When the working viewpoint cache set is an empty set, after adding a valid oblique viewpoint to the valid viewpoint set, return to step 9.1), otherwise select the valid viewpoint with the largest weight from the newly added working viewpoint cache set as the newly added working viewpoint and add it to the working viewpoint set; 9.9) Calculate the ratio of the number of newly added working viewpoint visible type I grids compared to the observed grid set to the total number of target grids as the new addition rate; 9.10) Add the newly added working viewpoint visible Class I grid index to the observed grid set, and update the grid coverage rate according to the number of grid indexes in the observed network set and the total number of target grids; When the number of viewpoints in the working viewpoint set is less than N max , when the addition rate is less than 0.001 and the grid coverage rate is less than f, return to step 9.6); 9.11) According to the visible Class I grids of each viewpoint in the working viewpoint set, calculate the number of observations of each target grid, and select the target grids with 1 and 2 grid observations as the grids to be supplemented; 9.12) Sort the valid viewpoints that are not in the working viewpoint set by the number of overlaps between the Class I grid and the grid to be supplemented from large to small, traverse the sorted valid viewpoints until the overlap rate of the traversed valid viewpoints exceeds the overlap rate r, stop traversing, add the valid viewpoint to the working viewpoint set, and update each target grid whose observation number is not 0 according to the Class I grid index observed by the valid viewpoint; 9.13) Repeat step 9.12) until the target grid is observed no more than 1 time or 2 times, or there is no valid viewpoint that meets the conditions of the previous step, and output the working viewpoint set.

9. The method for collecting close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 8, characterized in that: In step 9.8), the step of adding a valid oblique viewpoint to the valid viewpoint set includes: Sort the valid viewpoints in the squint viewpoint information cache set from large to small according to the number of common view grids; Traverse the sorted oblique viewpoint cache set, select the valid viewpoint whose nearest point in the target point cloud and the nearest point in the target point cloud of the working viewpoint are not the same point, as the newly added valid viewpoint and stop traversal; A camera squint model is established for the newly added valid viewpoints, the target mesh visual information is calculated and added to the valid viewpoint set.

10. The method for acquiring close-up images of unmanned aerial vehicles based on incremental reconstruction according to claim 1, characterized in that: In step 10), the step of generating the flight path of the drone in the flight space based on the working viewpoint coordinates includes: 10.1) Input the coordinates of the UAV flight space and working viewpoint; 10.2) Obtain the maximum Z coordinate from the working viewpoint coordinates; 10.3) According to the coordinates after ZY screen projection, the layer closest to the maximum Z coordinate is used as the transfer layer; 10.4) Project all working viewpoints to the transport layer, and use the projected coordinates as working control points; 10.5) Carry out work control point path planning; 10.6) Plan the path according to the work control points. After the UAV flies to each work control point in turn, it vertically takes off and lands to each work viewpoint corresponding to the control point to collect images, and then returns and goes to the next work control point.

Images